Issue 11321: 9th import of module _pickle always crashes (original) (raw)
Created on 2011-02-25 15:51 by palm.kevin, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (11)
Author: Palm Kevin (palm.kevin)
Date: 2011-02-25 15:51
Please have a look to the following app:
#include "Python.h"
main(int argc, char **argv) { int i; printf("START\n"); for(i = 0; i < 20; i++){ Py_Initialize();
printf("Try import #%i ...", i);
PyRun_SimpleString("import _pickle\n");
printf("SUCCESS\n");
Py_Finalize();} printf("END\n"); }
If I run that app (Linux), then I get the following output:
START Try import #0 ...SUCCESS Try import #1 ...SUCCESS Try import #2 ...SUCCESS Try import #3 ...SUCCESS Try import #4 ...SUCCESS Try import #5 ...SUCCESS Try import #6 ...SUCCESS Try import #7 ...SUCCESS Segmentation fault
It systematically crashes on the 9th call ... Does anybody know the reason of this segmentation fault problem??
Author: Alyssa Coghlan (ncoghlan) * 
Date: 2011-02-27 13:05
I can definitely confirm that one:
~/devel/py3k$ gcc -I/usr/local/include/python3.3m -o .c -L/usr/local/lib/python3.3 -lpython3.3m -lm -lpthread -ldl -lutil -rdynamic ~/devel/py3k$ ./issue11321 START Try import #0 ...SUCCESS Try import #1 ...SUCCESS Try import #2 ...SUCCESS Try import #3 ...SUCCESS Try import #4 ...SUCCESS Try import #5 ...SUCCESS Try import #6 ...SUCCESS Try import #7 ...SUCCESS Segmentation fault
I can also confirm that it isn't a regression w.r.t 3.1:
~/devel/py3k$ gcc -I/usr/include/python3.1 -o .c -L/usr/lib/python3.1 -lpython3.1 -lm -lpthread -ldl -lutil -rdynamic ~/devel/py3k$ ./issue11321 START Try import #0 ...SUCCESS Try import #1 ...SUCCESS Try import #2 ...SUCCESS Try import #3 ...SUCCESS Try import #4 ...SUCCESS Try import #5 ...SUCCESS Try import #6 ...SUCCESS Try import #7 ...SUCCESS Segmentation fault
It is, however, a regression from the 2.x line (where, presumably, cPickle simply never got reinitialised)
Author: Alyssa Coghlan (ncoghlan) *
Date: 2011-02-27 13:17
I haven't worked out where the error is happening yet, but the 13 allocated static variables and the complete lack of a module cleanup function aren't going to be helping matters.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2011-02-27 13:43
Here's a sample stack trace of it blowing up:
Program received signal SIGSEGV, Segmentation fault. 0x0000000000479404 in type_dealloc (type=0x7ffff6a82f20) at Objects/typeobject.c:2523 2523 _PyObject_GC_UNTRACK(type);
(gdb) bt #0 0x0000000000479404 in type_dealloc (type=0x7ffff6a82f20) at Objects/typeobject.c:2523 #1 0x0000000000465a2b in dict_dealloc (mp=0x8f61b0) at Objects/dictobject.c:1020 #2 0x00000000004e43c8 in _PyImport_FixupExtensionUnicode (mod=0x7ffff7f68ef0, name=0x7fffffffcd40 "_pickle", filename=0x7ffff7f12570) at Python/import.c:611 #3 0x00000000004e7fcf in _PyImport_LoadDynamicModule (name=0x7fffffffcd40 "_pickle", pathname=0x7fffffffbc70 "/usr/local/lib/python3.3/lib-dynload/_pickle.cpython-33m.so", fp=) at ./Python/importdl.c:85 #4 0x00000000004e5f79 in import_submodule (mod=0x796b20, subname=0x7fffffffcd40 "_pickle", fullname=0x7fffffffcd40 "_pickle") at Python/import.c:2894 #5 0x00000000004e61ef in load_next (mod=0x796b20, altmod=, p_name=, buf=0x7fffffffcd40 "pickle", p_buflen=0x7fffffffcd38) at Python/import.c:2706 #6 0x00000000004e6931 in import_module_level (name=0x0, globals=0x922060, locals=, fromlist=0x796b20, level=) at Python/import.c:2422 #7 0x00000000004e6e74 in PyImport_ImportModuleLevel (name=0x7ffff7e6c550 "pickle", globals=0x922060, locals=0x922060, fromlist=0x796b20, level=) at Python/import.c:2474 #8 0x00000000004c642b in builtin___import (self=, args=, kwds=) at Python/bltinmodule.c:167 #9 0x000000000042c897 in PyObject_Call (func=0x7ffff7edf8c0, arg=0x8e79e0, kw=0xa2e79) at Objects/abstract.c:2149 #10 0x00000000004c6c73 in PyEval_CallObjectWithKeywords (func=0x7ffff7edf8c0, arg=0x7ffff7e85ef0, kw=0xa2e79) at Python/ceval.c:3739 #11 0x00000000004ca6de in PyEval_EvalFrameEx (f=0xae4490, throwflag=) at Python/ceval.c:2316 #12 0x00000000004cf343 in PyEval_EvalCodeEx (_co=0x7ffff7ea5690, globals=, locals=, args=0x0, argcount=, kws=, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3295 #13 0x00000000004cf5db in PyEval_EvalCode (co=0x7ffff6a82f20, globals=0x8e79e0, locals=0xa2e79) at Python/ceval.c:761 #14 0x0000000000419c7e in run_mod (command=, flags=) at Python/pythonrun.c:1769 #15 PyRun_StringFlags (command=, flags=) at Python/pythonrun.c:1703 #16 PyRun_SimpleStringFlags (command=, flags=) at Python/pythonrun.c:1276 #17 0x0000000000415c41 in main ()
So _pickle itself doesn't seem to be dying, but it is eventually leaving things in a bad state so that the next time around, the import machinery blows up. Note that switching to a different module (e.g. "_struct") eliminates the segfault.
Author: Andreas Stührk (Trundle) *
Date: 2011-02-27 14:03
It's because the _pickle module doesn't incref Pickler_Type and Unpickler_Type before calling PyModule_AddObject() (which steals a reference).
The attached patch fixes the issue.
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *
Date: 2011-02-27 23:34
The patch looks good. It may also fix , which fails in a similar fashion
Author: Alyssa Coghlan (ncoghlan) *
Date: 2011-02-28 00:24
While Andreas's patch does indeed prevent the crash, there is something more going on here.
I modified his patch to print out the reference counts immediately after the new INCREF commands.
With the INCREF commands commented out, it looks like this:
~/devel/py3k$ ./issue113213.3 START Try import #0 ...Initialising _pickle Pickler type references: 10 Unpickler type references: 8 Module initialised SUCCESS Try import #1 ...Initialising _pickle Pickler type references: 10 Unpickler type references: 8 Module initialised SUCCESS Try import #2 ...Initialising _pickle Pickler type references: 9 Unpickler type references: 7 Module initialised SUCCESS Try import #3 ...Initialising _pickle Pickler type references: 8 Unpickler type references: 6 Module initialised SUCCESS Try import #4 ...Initialising _pickle Pickler type references: 7 Unpickler type references: 5 Module initialised SUCCESS Try import #5 ...Initialising _pickle Pickler type references: 6 Unpickler type references: 4 Module initialised SUCCESS Try import #6 ...Initialising _pickle Pickler type references: 5 Unpickler type references: 3 Module initialised SUCCESS Try import #7 ...Initialising _pickle Pickler type references: 4 Unpickler type references: 2 Module initialised SUCCESS Try import #8 ...Initialising _pickle Pickler type references: 3 Unpickler type references: 1 Module initialised Segmentation fault
Note that it does the right thing the first time Py_Finalize is called, but the reference counts start going down after that.
When the INCREF is included, the count goes up initially and then levels out.
So I believe the simple patch is just masking the problem instead of fixing it. The correct answer is to follow the documentation and implement the module finalisation protocol for _pickle (i.e. move all the static globals into the module struct and implement module traversal and finalisation functions).
Author: Andreas Stührk (Trundle) *
Date: 2011-02-28 01:05
That initial increase you can see happens because the module's dict is copied in _PyImport_FixupExtensionUnicode().
Author: Alyssa Coghlan (ncoghlan) *
Date: 2011-02-28 01:31
Looking at the example of _struct.c, I'll withdraw my objection to the patch. The segfault is due to the attempt to destroy a statically allocated type object, and the only viable solution to that is to ensure the reference count never drops to 0, even during Py_Finalize.
I also created to cover the fact that _pickle really shouldn't be stashing things in static globals now that a better alternative is available.
Author: Roundup Robot (python-dev) 
Date: 2011-07-15 19:03
New changeset 1ae0b7b8de0b by Antoine Pitrou in branch '3.2': Issue #11321: Fix a crash with multiple imports of the _pickle module when http://hg.python.org/cpython/rev/1ae0b7b8de0b
New changeset 6674272754da by Antoine Pitrou in branch 'default': Issue #11321: Fix a crash with multiple imports of the _pickle module when http://hg.python.org/cpython/rev/6674272754da
Author: Antoine Pitrou (pitrou) *
Date: 2011-07-15 19:04
Should be fixed now, thank you!
History
Date
User
Action
Args
2022-04-11 14:57:13
admin
set
github: 55530
2011-07-15 19:04:09
pitrou
set
status: open -> closed
versions: - Python 3.1
messages: +
resolution: fixed
stage: resolved
2011-07-15 19:03:25
python-dev
set
nosy: + python-dev
messages: +
2011-03-25 02:05:30
vstinner
set
nosy: + vstinner
2011-02-28 01:31:16
ncoghlan
set
nosy: + loewis
messages: +
versions: + Python 3.3
2011-02-28 01:15:21
ncoghlan
link
2011-02-28 01:05:34
Trundle
set
nosy:brett.cannon, amaury.forgeotdarc, ncoghlan, pitrou, alexandre.vassalotti, Trundle, palm.kevin
messages: +
2011-02-28 00:24:24
ncoghlan
set
nosy:brett.cannon, amaury.forgeotdarc, ncoghlan, pitrou, alexandre.vassalotti, Trundle, palm.kevin
messages: +
2011-02-27 23:34:26
amaury.forgeotdarc
set
nosy: + amaury.forgeotdarc
messages: +
2011-02-27 14:03:35
Trundle
set
files: + issue11321.patch
nosy: + Trundle
messages: +
keywords: + patch
2011-02-27 13:43:35
ncoghlan
set
nosy:brett.cannon, ncoghlan, pitrou, alexandre.vassalotti, palm.kevin
messages: +
2011-02-27 13:17:56
ncoghlan
set
nosy:brett.cannon, ncoghlan, pitrou, alexandre.vassalotti, palm.kevin
messages: +
2011-02-27 13:05:25
ncoghlan
set
nosy:brett.cannon, ncoghlan, pitrou, alexandre.vassalotti, palm.kevin
messages: +
2011-02-26 17:28:46
eric.araujo
set
nosy: + alexandre.vassalotti, brett.cannon, ncoghlan, pitrou
2011-02-25 15:52:09
palm.kevin
set
versions: + Python 3.1
2011-02-25 15:51:21
palm.kevin
create