| msg69532 - (view) |
Author: Darryl Dixon (esrever_otua) |
Date: 2008-07-11 04:54 |
| In at least Python 2.4, using cPickle.Pickler to try and pickle a nested chain of objects more than about 2590 objects deep causes the Python interpreter to segfault. This doesn't seem to happen when using the pure Python pickle module. It is not memory related (witness that the pure Python module can achieve depths much greater than this just fine), and does not seem to be directly related to system architecture (happens on both i386 and on x86_64 (32bit and 64bit)). Sample interpreter session to replicate: >>> # Let's cause cPickle to segfault: >>> from cPickle import Pickler as cPickler >>> class rec: ... child = None ... def __init__(self, counter): ... if counter > 0: ... self.child = rec(counter-1) ... >>> import sys >>> sys.setrecursionlimit(10000) >>> mychain = rec(2600) >>> from cStringIO import StringIO >>> stream = StringIO() >>> p = cPickler(stream, 1) >>> res = p.dump(mychain) Segmentation fault And now the same steps again using the pure Python Pickler: >>> import sys >>> from pickle import Pickler as pPickler >>> from cStringIO import StringIO >>> class rec: ... child = None ... def __init__(self, counter): ... if counter > 0: ... self.child = rec(counter-1) ... >>> sys.setrecursionlimit(20000) >>> mychain = rec(2600) >>> stream = StringIO() >>> p = pPickler(stream, 1) >>> p.dump(mychain) >>> len(stream.getvalue()) 48676 >>> |
|
|
| msg69534 - (view) |
Author: Martin v. Löwis (loewis) *  |
Date: 2008-07-11 06:49 |
| Can you try this for a newer version? For 2.4, such problems will not be fixed anymore. |
|
|
| msg69585 - (view) |
Author: Darryl Dixon (esrever_otua) |
Date: 2008-07-12 12:37 |
| Happens with Python 2.5.2 on 64bit also: Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import platform >>> platform.architecture() ('64bit', '') >>> from cPickle import Pickler >>> class rec: ... child = None ... def __init__(self, counter): ... if counter > 0: ... self.child = rec(counter-1) ... >>> import sys >>> sys.setrecursionlimit(10000) >>> mychain = rec(2600) >>> from cStringIO import StringIO >>> stream = StringIO() >>> p = Pickler(stream, 1) >>> res = p.dump(mychain) Segmentation fault |
|
|
| msg69756 - (view) |
Author: Darryl Dixon (esrever_otua) |
Date: 2008-07-16 03:27 |
| Hmm, looks like this dup's 2702... Funny how two people find the same thing within a short window of each other *sighs* so looks like it's probably fixed. I'll test /trunk against the failing testcase below and make sure all is OK. D |
|
|
| msg69759 - (view) |
Author: Darryl Dixon (esrever_otua) |
Date: 2008-07-16 04:07 |
| No, I've just tested /trunk, including r64595, and the Segmentation fault is still present, eg: Python 2.6b1+ (trunk:64998, Jul 16 2008, 15:50:22) [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.setrecursionlimit(40000) >>> class rec(object): ... child = None ... def __init__(self, counter): ... if counter > 0: ... self.child = rec(counter-1) ... >>> mychain = rec(2600) >>> from cPickle import Pickler >>> from cStringIO import StringIO >>> stream = StringIO() >>> p = Pickler(stream, 1) >>> p.dump(mychain) Segmentation fault |
|
|
| msg69803 - (view) |
Author: Facundo Batista (facundobatista) * |
Date: 2008-07-16 17:01 |
| Confirmed in... Python 2.6b1+ (trunk:65017M, Jul 16 2008, 13:37:00) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 ...with a more simple case: """ import sys, cPickle sys.setrecursionlimit(10405) class rec(object): child = None def __init__(self, counter): if counter > 0: self.child = rec(counter-1) mychain = rec(2600) cPickle.dumps(mychain) """ Note that if we put the recursion limit in 10405 we get a segfault, but if we put it 10404, we get a "RuntimeError: maximum recursion depth exceeded". Considering that 10400 is exactly 2600 * 4, maybe this is a useful hint. Another behaviour I got: With a recursion limit big big enough, doing rec(1670) works ok, and rec(1671) segfaults. And a very nasty one: I put rec(1671) to see in which recursion limit we have the conversion of RecursionLimit to SegFault. Testing, I tried with recursion limit in 6700, and sometimes it worked ok, and sometimes it segfaulted. Yes, *sometimes*, :( |
|
|
| msg69842 - (view) |
Author: Darryl Dixon (esrever_otua) |
Date: 2008-07-16 21:39 |
| That is a very interesting observation (x4), especially in light of #3373 Unfortunately I don't really have the (p|g)db -foo to debug either of these properly :( |
|
|
| msg71656 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-08-21 16:24 |
| Well, the standard recursion limit is precisely there to guard against segfaults when blowing up the C stack, so if you make the recursion limit much larger, it's quite normal to get segfaults. Therefore, I don't think this is a real bug. |
|
|
| msg71722 - (view) |
Author: Darryl Dixon (esrever_otua) |
Date: 2008-08-22 04:18 |
| Well, it's definitely a bug, or inconsistency, if you like, between cPickle and pickle. My gut says that probably there is some fault in cPickle that is causing this. When pickle.py can recurse to 10,000+ and cPickle segfaults at 2600 on a 64bit machine, something smells wrong. Especially as there have been other, similar, *fixable* bugs already in cPickle (cf #2702). Unfortunately I can only report the problem, I do not have the expertise to debug. regards, D |
|
|
| msg71733 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-08-22 09:23 |
| > Well, it's definitely a bug, or inconsistency, if you like, between > cPickle and pickle. There is clearly a problem with cPickle stack consumption and a new bug has been opened for this in #3640. What I don't agree with is your argument that pickle and cPickle should have the same recursion limits - it's just "foolish consistency" to ask for identical behaviour on such a low-level and implementation-dependent issue. |
|
|
| msg87936 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-05-16 21:23 |
| Now that #3640 has been fixed, this bug is probably fixed too (at least in trunk and py3k). |
|
|