Issue 15578: Crash when modifying sys.modules during import (original) (raw)

Issue15578

Created on 2012-08-07 20:50 by twouters, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
eric.snow_issue15578.diff eric.snow,2012-08-18 05:59
Messages (6)
msg167640 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2012-08-07 20:50
In a submodule of a package, replacing the parent package in sys.modules during import of the package (meaning the only reference to it is in sys.modules) and then importing itself (or something from itself) crashes the interpreter: centurion:~/python/crasher > cat sa/__init__.py import v1 centurion:~/python/crasher > cat sa/v1/__init__.py import sys sys.modules['sa'] = sys.modules[__name__] import sa centurion:~/python/crasher > python -c 'import sa' Segmentation fault (core dumped) It seems the crash is not entirely deterministic, as we've had the original code this was reduced from run in Python 2.4 and 2.6 for years, and only discovered the crash when switching to 2.7. However, running the reduced testcase crashes for me in all of Python 2.7, 2.6, 2.4 and 2.2, in debug builds and opt builds. Given the nature of the bug I expect debug builds to crash reliably. I haven't found the actual source of the problem, but what seems to happen is that the import mechanism has a borrowed reference to the 'sa' module around, assuming it'll stay alive while the submodules are imported because it's also stored in sys.modules. This assumption is incorrect. However, changing the import in sa/__init__.py into an absolute or explicit relative import seems to fix the problem, which means this probably won't affect Python 3.
msg167642 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-08-07 21:24
If I remember correctly there were non-trivial improvements made to the way modules are finalized in 2.7 compared to earlier versions. That could be what exposed the bug. The pre-2.7 code might have been leaving another reference alive because of your more complex real world example, whereas the simplified example doesn't.
msg168495 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-08-18 05:59
Here's the deal. import_module_level() gets called for v1 from sa (where "globals" comes from). In that function it first calls get_parent(), which returns a borrowed reference to the sa module object. Then that parent object is passed to load_next() where the actual load of v1 will take place (and the segfault happens). The problem is that get_parent() returns a borrowed reference. When the sa module is replaced in sys.modules, the old sa module is decref'ed. That's fine except load_next is using that same module as the parent. Enter segfault, stage left. Here's a quick patch that fixes the failure (along with a test).
msg185285 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-26 17:59
Eric has commit rights now so he can commit his own fix. =)
msg274953 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-08 02:09
New changeset 2d6dd8402d77 by Eric Snow in branch '2.7': Issue #15578: Correctly incref the parent module while importing. https://hg.python.org/cpython/rev/2d6dd8402d77
msg274965 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-08 05:26
New changeset 731e5617cc8d by Gregory P. Smith in branch '2.7': Fix placement of Misc/NEWS item for issue #15578. https://hg.python.org/cpython/rev/731e5617cc8d
History
Date User Action Args
2022-04-11 14:57:33 admin set github: 59783
2016-09-08 05:26:28 python-dev set messages: +
2016-09-08 02:10:06 eric.snow set status: open -> closedresolution: fixedstage: patch review -> resolved
2016-09-08 02:09:04 python-dev set nosy: + python-devmessages: +
2013-03-26 17:59:59 brett.cannon set assignee: eric.snownosy: - brett.cannon
2013-03-26 17:59:40 brett.cannon set nosy:twouters, brett.cannon, gregory.p.smith, r.david.murray, eric.snow, nordauxmessages: +
2012-08-18 05:59:46 eric.snow set files: + eric.snow_issue15578.diffnosy: + eric.snow, brett.cannonmessages: + keywords: + patchstage: patch review
2012-08-09 17:42:27 loewis set versions: - Python 2.6
2012-08-08 08:13:17 nordaux set nosy: + nordaux
2012-08-07 21:24:31 r.david.murray set nosy: + r.david.murraymessages: +
2012-08-07 21:21:40 gregory.p.smith set nosy: + gregory.p.smith
2012-08-07 20:50:29 twouters create