| msg135961 - (view) |
Author: kai zhu (kaizhu) |
Date: 2011-05-14 07:16 |
| i'm using the latest debian unstable python3.2 build on colinux (2011, may, 14) ## leak.py ## >>> import imp, leak; imp.reload(leak) ## will leak ~2.5mb per reload ## on i386 debian unstable machine (according to top). ## in my real world app (an automatic build system), ## i run out of memory after a number reloads :( class Foo(object): pass Foo.leaky_dictionary = {} for aa in range(256): for bb in range(256): Foo.leaky_dictionary[(bb << 8) | aa] = None $ python3.2 Python 3.2.1a0 (default, May 5 2011, 00:47:12) [GCC 4.6.1 20110428 (prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import imp, leak; imp.reload(leak) ## 11mb <module 'leak' from 'leak.py'> >>> import imp, leak; imp.reload(leak) ## 13mb <module 'leak' from 'leak.py'> >>> import imp, leak; imp.reload(leak) ## 16mb <module 'leak' from 'leak.py'> >>> import imp, leak; imp.reload(leak) ## 19mb <module 'leak' from 'leak.py'> |
|
|
| msg135963 - (view) |
Author: Ezio Melotti (ezio.melotti) *  |
Date: 2011-05-14 07:55 |
| I think this has to do with class attributes and reload(): wolf@hp:~/dev/py/py3k$ cat leak.py class Foo: pass Foo.l = list(range(65535)) wolf@hp:~/dev/py/py3k$ ./python Python 3.3a0 (default:4b122cac7ac5+, May 14 2011, 10:01:13) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import imp, leak; imp.reload(leak) <module 'leak' from 'leak.py'> [189091 refs] >>> imp.reload(leak) <module 'leak' from 'leak.py'> [254649 refs] >>> imp.reload(leak) <module 'leak' from 'leak.py'> [320207 refs] >>> imp.reload(leak) <module 'leak' from 'leak.py'> [385765 refs] >>> import gc; gc.collect() 28 [123927 refs] However calling gc.collect() explicitly seems to fix the problem. |
|
|
| msg135964 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2011-05-14 08:00 |
| It's simply because all classes form a cycle (Foo -> Foo.__mro__ -> Foo) A class and class attributes can only be freed with gc.collect(). Did you disable the garbage collector? |
|
|
| msg136020 - (view) |
Author: kai zhu (kaizhu) |
Date: 2011-05-15 08:14 |
| explicit gc.collect() doesn't seem to fix the leak in my application. my current fix is to not re-instantiate the class attribute (which cost ~7mb) during reload & instead reference one created earlier. i haven't pinpointed y, but i suspect its a corner case, which would rarely occur in general usage. my class also inherits from subprocess.Popen, which has a __del__ method, which might interfere w/ collection (although gc.garbage says otherwise ;). the reason i reload is that the script gets modified frequently, which the auto-build system will detect & reload. |
|
|
| msg136479 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2011-05-21 20:34 |
| > my class also inherits from subprocess.Popen, which has a __del__ method, which might interfere w/ collection The doc says __del__ *will* prevent collection. > (although gc.garbage says otherwise ;). Do you mean that gc.garbage is empty, when it should not be? Have you tried replacing __del__ with an explicit close or cleanup method? |
|
|
| msg194942 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2013-08-12 11:25 |
| Yes, __del__ will interfere with garbage collection before Python 3.4. This is pretty much expected (and is fixed in Python 3.4, but won't be backported). |
|
|