Issue 3919: PySys_SetObject crashes after Py_NewInterpreter(). (original) (raw)

Somewhere between Python 3.0a3 and Python 3.0b3, a call to PySys_SetObject() after having used Py_NewInterpreter() to create a sub interpreter causes a crash. This appears to be due to interp->sysdict being NULL after Py_NewInterpreter() called.

As illustration of problem, consider program for Python 2.X.

#include <Python.h>

int main(int argc, char *argv[]) { Py_Initialize(); PySys_SetObject("xxx", PyLong_FromLongLong(1)); fprintf(stderr, "sysdict=%d\n", !!PyThreadState_Get()->interp->sysdict); fflush(stderr); PyRun_SimpleString("import sys\n" "print >> sys.stderr, 'xxx =', sys.xxx\n");

Py_NewInterpreter();
fprintf(stderr, "sysdict=%d\n", !!PyThreadState_Get()->interp->sysdict);
fflush(stderr);
PySys_SetObject("yyy", PyLong_FromLongLong(2));
PyRun_SimpleString("import sys\n"
                   "print >> sys.stderr, 'yyy =', sys.yyy\n");

Py_Finalize();
return 0;

}

This when run yields:

sysdict=1 xxx = 1 sysdict=1 yyy = 2

Now, for Python 3.0 variant of same program:

#include <Python.h>

int main(int argc, char *argv[]) { Py_Initialize(); fprintf(stderr, "sysdict=%d\n", !!PyThreadState_Get()->interp->sysdict); fflush(stderr); PySys_SetObject("xxx", PyLong_FromLongLong(1)); PyRun_SimpleString("import sys\n" "print('xxx =',sys.xxx, file=sys.stderr)\n");

Py_NewInterpreter();
fprintf(stderr, "sysdict=%d\n", !!PyThreadState_Get()->interp->sysdict);
fflush(stderr);
PySys_SetObject("yyy", PyLong_FromLongLong(2));
PyRun_SimpleString("import sys\n"
                   "print('yyy =',sys.yyy, file=sys.stderr)\n");

Py_Finalize();
return 0;

}

I get for Python 3.0a3:

sysdict=1 xxx = 1 sysdict=1 object : AttributeError("'module' object has no attribute 'stderr'",) type : AttributeError refcount: 4 address : 0xf1180 lost sys.stderr

I am not concerned here about loss of sys.stderr, although that could be a separate issue for all I know.

The important bit here is that sysdict is set after Py_NewInterpreter().

In Python 3.0b3/3.0rc1 I instead get:

sysdict=1 xxx = 1 sysdict=0 Bus error

This is because PySys_SetObject() is presumably crashing because sysdict is not set in interp object.

I tried to ask about this on python-3000 Google group, but that message ended up in some moderation queue and has vanished. Thus quote part of that message below.

""" From what I can tell so far the problem is that 'interp->sysdict' is NULL after calling Py_NewInterpreter() to create a secondary sub interpreter.

Reading through code and using a debugger, at this point this seems to be due to condition if code:

   sysmod = _PyImport_FindExtension("sys", "sys");
   if (bimod != NULL && sysmod != NULL) {
           interp->sysdict = PyModule_GetDict(sysmod);
           if (interp->sysdict == NULL)
                   goto handle_error;
           Py_INCREF(interp->sysdict);
           PySys_SetPath(Py_GetPath());
           PyDict_SetItemString(interp->sysdict, "modules",
                                interp->modules);
           _PyImportHooks_Init();
           initmain();
           if (!Py_NoSiteFlag)
                   initsite();
   }

in Py_NewInterpreter() not executing due to _PyImport_FindExtension("sys", "sys") returning NULL.

Down in _PyImport_FindExtension(), it appears that the reason it fails is because of following returning with NULL.

   def = (PyModuleDef*)PyDict_GetItemString(extensions,

filename);

   .....

           if (def->m_base.m_init == NULL)
                   return NULL;

In other words, whatever m_base.m_init is meant to be is NULL when perhaps it isn't meant to be.

(gdb) call ((PyModuleDef*)PyDict_GetItemString(extensions,"builtins"))-

m_base.m_init $9 = (PyObject ()()) 0 (gdb) call ((PyModuleDef*)PyDict_GetItemString(extensions,"sys"))- m_base.m_init $10 = (PyObject ()()) 0

I am going to keep tracking through to try and work out why, but posting this initial information in case this rings a bell with anyone. """

Is this expected behaviour? Or, is it necessary now to perform some special initialisation after having called Py_NewInterpreter() to get builtins and sys modules setup?

This problem originally came up with mod_wsgi, which worked fine with Python 3.0a3, but fails on more recent releases because of this.

I know sub interpreters may be seen as evil by some and the target of removal, but are they still meant to work for now at least?