Issue 13133: FD leaks in ZipFile.read(), ZipFile.extract() and also using explicit arc_member.close() (original) (raw)

Created on 2011-10-08 18:48 by Valery.Khamenya, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
zipfiletest.py Valery.Khamenya,2011-10-08 18:48
zipfile.patch Valery.Khamenya,2011-10-09 10:15
Messages (5)
msg145182 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-08 18:48
The attached file reproduces 3 types of FD leaks and leads to the error like: IOError: [Errno 24] Too many open files: '/tmp/1019' For example if executed with pypy.
msg145185 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-08 20:27
I can't reproduce this problem in either 2.7.2 or 3.3.0a0. > For example if executed with pypy. Do you mean that this problem is only reproducible when the attached script is run with pypy?
msg145188 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-08 20:44
"I can't reproduce this problem in either 2.7.2 or 3.3.0a0." You probably mean CPython implementation of Python. No, I didn't mean this implementation. "Do you mean that this problem is only reproducible when the attached script is run with pypy?" I can't say "only". I just could say yes, it is reproducible with pypy. Perhaps, there are others Python implementations that will suffer from the bug in current implementation of ZipFile
msg145191 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-08 21:58
Yes, in 2.7 many parts of the stdlib relies on reference counting to close files. But 3.2 introduced a ResourceWarning which is emitted (in debug mode) each time a __del__ closes a valuable resource like a file or a socket. This was done exactly for this reason - help other implementations with a different garbage collector. Now Lib/zipfile.py is probably much more gc-friendly: see how it uses a new member "close_fileobj", and the "with" statement in ZipFile.read(). PyPy will benefit of this when it migrates to 3.2; meanwhile, you could apply the same changes in pypy's own copy of zipfile.py.
msg145250 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-09 10:15
Amaury, I followed your advice. All relevant changes of 3.2 are backported via the patch attached. P.S. now I can install Twisted using pypy too. "pypy setup.py install" works fine for me.
History
Date User Action Args
2022-04-11 14:57:22 admin set github: 57342
2012-10-15 13:23:34 serhiy.storchaka set status: open -> closed
2011-10-09 10:15:17 Valery.Khamenya set files: + zipfile.patchkeywords: + patchstatus: pending -> openmessages: +
2011-10-08 21:58:42 amaury.forgeotdarc set status: open -> pendingnosy: + amaury.forgeotdarcmessages: + resolution: out of date
2011-10-08 20:44:28 Valery.Khamenya set messages: +
2011-10-08 20:27:59 meador.inge set nosy: + meador.ingemessages: +
2011-10-08 18:48:54 Valery.Khamenya create