[Python-Dev] need help with frozen module/marshal/gc issue involving sub-interpreters for importlib bootstrapping (original) (raw)

Brett Cannon brett at python.org
Mon Feb 6 15:57:56 CET 2012

So my grand quest for bootstrapping importlib into CPython is damn close to coming to fruition; I have one nasty bug blocking my way and I can't figure out what could be causing it. I'm hoping someone here will either know the solution off the top of their head or will have the time to have a quick look to see if they can figure it out as my brain is mush at this point.

First, the bug tracking all of this is http://bugs.python.org/issue2377 and the repo where I have been doing my work is ssh:// hg at hg.python.org/sandbox/bcannon/#bootstrap_importlib (change as needed if you want an HTTPS checkout). Everything works fine as long as you don't use sub-interpreters via test_capi (sans some test failures based on some assumptions which can easily be fixed; the bug I'm talking about is the only real showstopper at this point).

Here is the issue: if you run test_capi the code triggers an assertion of test_subinterps (__main__.TestPendingCalls) ... Assertion failed: (gc->gc.gc_refs != 0), function visit_decref, file Modules/gcmodule.c, line 327.. If you run the test under gdb you will discover that the assertion is related to ref counts when collecting for a generation (basically the ref updating is hitting 0 when it shouldn't).

Now the odd thing is that this is happening while importing frozen module code (something I didn't touch) which is calling marshal (something else I didn't touch) and while it is in the middle of unmarshaling the frozen module code it is triggering the assertion.

Does anyone have any idea what is going on? Am I possibly doing something stupid with refcounts which is only manifesting when using sub-interpreters? All relevant code for bootstrapping is contained in Python/pythonrun.c:import_init() (with a little tweaking in the _io module to delay importing the os module and making import.c always use import instead of using the C code). I'm storing the import function in the PyInterpreterState to keep separate state from the other interpreters (i.e. separate sys modules so as to use the proper sys.modules, etc.). But as I said, this all works in a single interpreter view of the world (the entire test suite doesn't trigger a nasty error like this).

Thanks for any help people can provide me on this now 5 year quest to get this work finished.

-Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20120206/e8a8a824/attachment.html>

More information about the Python-Dev mailing list