| msg64784 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *  |
Date: 2008-03-31 15:12 |
| The following function seems to 8 references each time it is run: import io, gc def f(): class C: pass c=C() assert isinstance(c, io.StringIO) is False gc.collect();gc.collect();gc.collect() This is because io.StringIO._abc_negative_cache contains a strong reference to each "class C", as soon as isinstance() is called. These are never released. Python3.0 does use WeakSet for these caches, and does not leak. This is the (deep) reason why test_io.IOTest.test_destructor() leaks in the following report: http://mail.python.org/pipermail/python-checkins/2008-March/067918.html (The test derives from io.FileIO, and it is the call to the base class' method which calls isinstance() and put the class in the cache) |
|
|
| msg87720 - (view) |
Author: Daniel Diniz (ajaksu2) *  |
Date: 2009-05-13 22:54 |
| Confirmed on release26-maint and trunk. |
|
|
| msg101907 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-03-29 22:21 |
| Now that WeakSet has been backported to trunk, I've backported the fix for this reference-leak (patch with test case attached). However, after making the patch, I discovered that old-style classes are not weak-referenceable. Consequently, the patch is not suitable for commiting. If old-style classes can be made weak-referenceable, then the patch should work. If not, then this bug is essentially impossible to solve in 2.x unless the ABC cache is done away with entirely. Other notes: Since abc.py is imported during bootstrapping before paths are set up, I needed to add _weakref as a built-in module. (It's already a built-in module in the py3k branch.) Since the patch tweaks Modules/Setup.dist, you need to "make distclean" and rerun ./configure after applying the patch. Also, on unpatched trunk the example posted by Amaury no longer seems to trigger the issue. However, the example I posted in Issue 8022 still triggers the leak. I used that as the basis for the test case in the patch. |
|
|
| msg101960 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2010-03-30 23:35 |
| Hmm, Benjamin pointed out that ABCs only support new-style classes, so old-style classes could be detected early instead of being added to the _abc_negative_cache. |
|
|
| msg101961 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2010-03-30 23:38 |
| By the way, Daniel, your patch doesn't look right. First, you shouldn't need all the sortedlist/sortedset hierarchy. Second, len(gc.get_objects()) is a truly horrible way of checking the classes have been destroyed. Just take a weakref to the class before deleting it, and check that calling the weakref returns None. (besides, we also have a refleak-detection run (regrtest -R) which can detect leaks even if you don't check them explicitly in your tests) |
|
|
| msg101962 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-03-30 23:48 |
| I hadn't realized that old style classes didn't support ABCs. That certainly simplifies things! I'm working on a new patch. |
|
|
| msg101963 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-03-30 23:58 |
| Are you sure the old-style classes don't support ABCs? ABCTestCase.validate_isinstance in Lib/test/test_collection.py specifically tests that both new-style and old-style classes work, unless I'm reading it wrong. (and those tests fail if I make ABCMeta.__instancecheck__ and ABCMeta.__subclasscheck__ always return False for old-style classes) |
|
|
| msg101964 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2010-03-31 00:02 |
| In those cases, it's because __subclasscheck__ is overridden. You can't register a old-style class. |
|
|
| msg101965 - (view) |
Author: Florent Xicluna (flox) * |
Date: 2010-03-31 00:06 |
| > ABCTestCase.validate_isinstance ... specifically tests that > both new-style and old-style classes work... I fixed some parts with issue #7624, and changeset r78800. |
|
|
| msg101967 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-03-31 00:36 |
| You can't register an old-style class, but many ABCs support duck-typing by implementing __subclasshook__. ABCMeta caches those results and stores a reference to old-style classes, sometimes in _abc_cache and sometimes in _abc_negative_cache. It can't simply return False. I guess that leaves two options: 1) Make old-style classes weak-referenceable and cache the results of __subclasshook__. 2) Refuse to cache old-style classes and call __subclasshook__ every time for old-style classes. Python 2.7a4+ (trunk:79493, Mar 30 2010, 19:19:13) >>> class old_iterable_class: ... def __iter__(self): ... pass ... >>> import collections >>> issubclass(old_iterable_class, collections.Iterable) True |
|
|
| msg102036 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2010-03-31 21:39 |
| > Make old-style classes weak-referenceable Now done (r79535). |
|
|
| msg102037 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-03-31 21:41 |
| Cool! I will revise the patch based on your comments about my test case. |
|
|
| msg102047 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-03-31 23:45 |
| New patches uploaded. I separated out the patch to add the test case, to make it easier to test before and after applying the fix. |
|
|
| msg104287 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2010-04-26 23:21 |
| Someone with a better knowledge of ABCs than me should probably do a final review of this. |
|
|
| msg104290 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-04-26 23:51 |
| Someone with appropriate permissions want to update the "Stage"? |
|
|
| msg105931 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-05-17 19:10 |
| Antoine, do you have a suggestion for someone with with better knowledge of ABCs to do the final review, so that I may very politely pester them? ;-) |
|
|
| msg105934 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2010-05-17 20:07 |
| > Antoine, do you have a suggestion for someone with with better > knowledge of ABCs to do the final review, so that I may very politely > pester them? ;-) I guess Benjamin could. |
|
|
| msg110801 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2010-07-19 19:22 |
| Patched the unit test, then ran the test before applying the fix which failed, after applying the fix the test ran successfully. Tested on Windows Vista 32 bit against 2.7 maintainance release. The patches are short and sweet, I see no reason why they can't go forward. |
|
|
| msg112983 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2010-08-05 14:21 |
| Can a committer take a look at this please. |
|
|
| msg112991 - (view) |
Author: Jack Diederich (jackdied) * |
Date: 2010-08-05 15:51 |
| This is a change in the codepath for instances that don't have __class__ defined. subclass = getattr(instance, '__class__', None) - if subclass in cls._abc_cache: + if subclass is not None and subclass in cls._abc_cache: I think the same thing happens in either case (from visual inspection of the code) but I'd rather not change it if we don't need to. |
|
|
| msg113485 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-08-09 22:02 |
| Jack, The change is necessary because "None in WeakSet()" would throw an exception. |
|
|
| msg114093 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2010-08-17 01:35 |
| Some comments: - The test should be in test_abc.py and should probably not use a collections class. - Use test_support.gc_collect(). - What's the __len__() stuff for? |
|
|
| msg114226 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-08-18 14:42 |
| Benjamin, Thanks for the feedback. Since you only commented on the test case, may I assume that the fix itself looked good to you? I will work on revising the test case based on your comments. Since I ran into the bug while working with the ABCs in the collections module, that biased my thinking when writing the test. :-) I needed to define__len__ because it's an abstract method in the ABC I used in the test (collections.Sized). I found that overriding it again in a sub-sub-class and calling it were necessary to trigger all of the ABC machinery leading to the leak. |
|
|
| msg114233 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-08-18 15:27 |
| Attached is a new test case, based on Benjamin's comments. |
|
|
| msg114476 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2010-08-21 03:03 |
| r84230 |
|
|
| msg114477 - (view) |
Author: Daniel Stutzbach (stutzbach) |
Date: 2010-08-21 03:05 |
| Thanks! :-) |
|
|
| msg135931 - (view) |
Author: H. (bluag) |
Date: 2011-05-13 17:15 |
| ImportError: No module named _weakrefset Here are some references while i was trying to install Pinax framework. http://paste.pound-python.org/show/6536/ And i saw that the _weakrefset.py is not included in the package. So I have copied from Python's source with version : 3.1.* to my d:\sosyal\ folder. and everything works fine. |
|
|
| msg135935 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2011-05-13 17:43 |
| bluag: the script you run contains a list of some modules required to start Python (I found a copy here: https://github.com/pinax/pinax/raw/master/scripts/pinax-boot.py ) This script is obviously out of date wrt the new version of Python. Please report this to the pinax project! |
|
|