Issue 5092: weird memory usage in multiprocessing module (original) (raw)

Issue5092

Created on 2009-01-29 00:13 by Orlowski, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (12)
msg80731 - (view)	Author: Jerzy (Orlowski)	Date: 2009-01-29 00:13
Hi I am using the multiprocessing mudule and I found a very weird thing. It seems that that the result of one fragment of the code depends on the fragment of the code that is after it, which should not happen. My script looks like this import time import multiprocessing import sys def f(): sys.stderr.write(str(len(l))+"\n") print len(l) #del l while(True): time.sleep(1) l=[] for i in range(2*1000*1000): l.append(str(i)) process = multiprocessing.Process(target=f) process.start() while(True): time.sleep(1) And its output is as expected: 2000000 2000000 but when I uncoment the 'del l' line I get: File "/home/jerzyo/programs/python2.6/Python-2.6.1/Lib/multiprocessing/process.py", line 231, in _bootstrap self.run() File "/home/jerzyo/programs/python2.6/Python-2.6.1/Lib/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "bin/momory.py", line 6, in f sys.stderr.write(str(len(l))+"\n") UnboundLocalError: local variable 'l' referenced before assignment How is that? The line that deletes l is after the printing line. How python interpreter knows that l will be deleted. This is a very anomalus behaviour and should never happen. By the way. Is there any way to free some parts of memory in child process. Suppose I wand to create 100 child processes that do not use the l list. How can I avoid making 100 copies of l in child processes. That is my firs post and won't come here very often, so please answer also to my email (if it is not automaic). I am running python 2.6.1 on ubuntu 8.04 32bit. jerzy
msg80738 - (view)	Author: David W. Lambert (LambertDW)	Date: 2009-01-29 01:43
The del statement makes the variable local, as alluded to by http://docs.python.org/dev/3.0/reference/simple_stmts.html#the-del- statement The manual is clearer about assignments, which are local unless declared global or nonlocal. For other question, me thinks you need to write slightly cleverer code that passes only required data. I have experience with multiprocessing.Pool().map which lets me control the arguments I pass to functions.
msg80739 - (view)	Author: David W. Lambert (LambertDW)	Date: 2009-01-29 01:50
My second answer is irrelevant. Function receives the global data as well as the arguments.
msg80741 - (view)	Author: Martin v. Löwis (loewis) *	Date: 2009-01-29 02:29
As David says, this is not a bug. del l indicates that there is a local variable to be deleled, but when the del statement is executed, there is no local variable. The error message is confusing in this case: there actually is no later assignment to l (in the function at all). Typically, when you have an unbound local, it is because of a later assignment, such as def foo(): a = l + 1 l = 2 In this specific example, there is no later assignment - yet it is still an unbound local. So that you get the exception is not a bug. I was going to suggest that the error message could be better, but I can't think of any other error message that is better and still correct, hence closing it as won't fix.
msg80756 - (view)	Author: Jerzy (Orlowski)	Date: 2009-01-29 10:53
I still do not understand what is going on when python executed thic code. I have a local variable l in my parent process. When I create a child process, program makes first makes a copy of memory. Than what? I am sure that l still exists in child process because 1. It can be printed 2. It has still a lot of memory allocated for it You say that l does not exist as a local variable in child process. Is it global? How can I dealocate it in child process? Jerzy Martin v. Löwis pisze: > Martin v. Löwis <martin@v.loewis.de> added the comment: > > As David says, this is not a bug. del l indicates that there is a local > variable to be deleled, but when the del statement is executed, there is > no local variable. The error message is confusing in this case: there > actually is no later assignment to l (in the function at all). > Typically, when you have an unbound local, it is because of a later > assignment, such as > > def foo(): > a = l + 1 > l = 2 > > In this specific example, there is no later assignment - yet it is still > an unbound local. > > So that you get the exception is not a bug. > > I was going to suggest that the error message could be better, but I > can't think of any other error message that is better and still correct, > hence closing it as won't fix. > > ---------- > nosy: +loewis > resolution: -> wont fix > status: open -> closed > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue5092> > _______________________________________ > > >
msg80777 - (view)	Author: Martin v. Löwis (loewis) *	Date: 2009-01-29 19:18
> I still do not understand what is going on when python executed thic > code. I have a local variable l in my parent process. No, you don't. It's a global variable, not a local one. > When I create a > child process, program makes first makes a copy of memory. Than what? It doesn't have to do anything with the multiprocessing at all. For comparison, just run the Python script def f(): del l l = [] f() It produces the same error, with no multiprocessing involved.
msg80812 - (view)	Author: Jerzy (Orlowski)	Date: 2009-01-30 10:53
OK, I see and if don't want l to exist in f() I have to: def f(): pass def a(): l=[] f() a() ???? Jurek Martin v. Löwis wrote: > Martin v. Löwis <martin@v.loewis.de> added the comment: > >> I still do not understand what is going on when python executed thic >> code. I have a local variable l in my parent process. > > No, you don't. It's a global variable, not a local one. > >> When I create a >> child process, program makes first makes a copy of memory. Than what? > > It doesn't have to do anything with the multiprocessing at all. > For comparison, just run the Python script > > def f(): > del l > l = [] > f() > > It produces the same error, with no multiprocessing involved. > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue5092> > _______________________________________ > > >
msg80813 - (view)	Author: Jerzy (Orlowski)	Date: 2009-01-30 11:01
And anyway, for me it's not OK if something in a code of a function like 'del' affect how variables are affected in whole function. It is really illogical. There code is in lines and line are one below another. The logical way is that a line of code affects the program ONLY when it is executed and ONLY from the time it is executed. A statement that is not executed (python never reach the place) should not affect the program in ANY way. You may think what you think, but for me it is a big bug in the heart of python Jerzy Martin v. Löwis wrote: > Martin v. Löwis <martin@v.loewis.de> added the comment: > >> I still do not understand what is going on when python executed thic >> code. I have a local variable l in my parent process. > > No, you don't. It's a global variable, not a local one. > >> When I create a >> child process, program makes first makes a copy of memory. Than what? > > It doesn't have to do anything with the multiprocessing at all. > For comparison, just run the Python script > > def f(): > del l > l = [] > f() > > It produces the same error, with no multiprocessing involved. > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue5092> > _______________________________________ > > >
msg80818 - (view)	Author: David W. Lambert (LambertDW)	Date: 2009-01-30 14:40
The alternative is unreasonable. I doubt you'd be happy with this: a = 'Something' def variable_both_global_and_local()->Exception('No good!'): del a # delete a from global name space a = 'anotherthing' # define a in local name space
msg80841 - (view)	Author: Jerzy (Orlowski)	Date: 2009-01-30 22:52
I am not an expert. But for me it is much better. If you cannot delete the global variable in a function (del makes the variable local anyway). So trying to delete a global variable should raise an exception "Cannot delete a global variable" or something like that. In a function variable should be global till the place when you define a local one. Example: a='Something' def f(): print a #prints the global variable a del a #Make an exception that a is global so it cannot be deleted a='anotherthing' #make a local a print a #print local a del a #delete local a print a #print global a f() Also, if there are two variable (global and local) with seme name, there should be a way to access either of them like 'print loc(a)' and 'print glob(a)'. This is just a suggestion Another way of resolving the problem would be making it impossible to make a local variable when there is anothe one with the same name. David W. Lambert pisze: > David W. Lambert <lambertdw@corning.com> added the comment: > > The alternative is unreasonable. I doubt you'd be happy with this: > > > a = 'Something' > > def variable_both_global_and_local()->Exception('No good!'): > del a # delete a from global name space > a = 'anotherthing' # define a in local name space > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue5092> > _______________________________________ > > >
msg80846 - (view)	Author: Martin v. Löwis (loewis) *	Date: 2009-01-31 01:32
> Another way of resolving the problem would be making it impossible to > make a local variable when there is anothe one with the same name. Please trust that there really is no problem. If you don't understand how Python works, ask on a mailing list. If you want to propose a change to Python, contact python-ideas. Don't be surprised when everybody tells you that things are fine the way they are.
msg80848 - (view)	Author: David W. Lambert (LambertDW)	Date: 2009-01-31 01:58
#Ah! Not a problem. You need globals() and locals() dictionaries. # as a python3 script, this message produces next couple lines output. #method one #yup, global a is gone #method two #{'gv': 'local here', 'name': 'gv'} #yup, global gv is gone print('method one') a = '' def Delete_a_global_variable(): global a del a Delete_a_global_variable() try: a print('whoops! this can never happen') except NameError: print('yup, global a is gone') print('method two') gv = '' # global variable def delete_chosen_variable(name): del globals()[name] gv = 'local here' print(locals()) delete_chosen_variable('gv') try: gv print('whoops! this can never happen') except NameError: print('yup, global gv is gone')