[Python-Dev] Problem between deallocation of modules and func_globals (original) (raw)

Brett Cannon brett at python.org
Fri Jan 19 22:33:35 CET 2007

On 1/19/07, M.-A. Lemburg <mal at egenix.com> wrote:

On 2007-01-18 20:53, Brett Cannon wrote: > I have discovered an issue relating to funcglobals for functions and > the deallocation of the module it is contained within. Let's say you > store a reference to the function encodings.searchfunction from the > 'encodings' module (this came up in C code, but I don't see why it > couldn't happen in Python code). Then you delete the one reference to > the module that is stored in sys.modules, leading to its deallocation. > That triggers the setting of None to every value in > encodings.dict. > > Oops, now the global namespace for that module has everything valued > at None. The dict doesn't get deallocated since a reference is held > by encodings.searchfunction.funcglobals and there is still a > reference to that (technically held in the interpreter's > codecsearchpath field). So the function can still execute, but > throws exceptions like AttributeError because a module variable that > once held a dict now has None and thus doesn't have the 'get' method.

That's a typical error situation you get in del methods at the time the interpreter is shut down.

Yeah, but in this case this is at the end of Py_Initialize() for the stuff I am doing to the interpreter. =)

The main reason for setting everything to None first is to break circular references and make sure that at least some of the object destructors can run.

I know the reason, it just happens to occur at a bad time for me.

> My question is whether this is at all worth trying to rectify. Since > Google didn't turn anything up I am going to guess this is not exactly > a common thing. =) That would lead me to believe some (probably > most) of you will say, "just leave it alone and work around it".

If you can come up with a better way, sure :-) > The other option I can think of is to store a reference to the module > instead of just to its dict in the function. The problem with > that is we end up with a circular dependency of the functions in > modules having a reference to the module but then the module having a > reference to the functions. I tried not having the values in the > module's dict set to None if the reference count was above 1 and > that solved this issue, but that leads to dangling references on > anything in that dict that does not have a reference stored away > somewhere else like encodings.searchfunction. > > Anybody have any ideas on how to deal with this short of rewriting > some codecs stuff so that they don't depend on global state in the > module or just telling me to just live with it? I'm not exactly sure which global state you are referring to. The aliase map, the cache used by the search function ?

encodings._cache .

Note that the search function registry is a global managed in the thread state (it's not stored in any module).

Right, but that is not the issue. If you have deleted the reference to the encodings module from sys.modules it then sets encodings._cache to None. After the deletion, if you try to encode/decode a unicode string you can an AttributeError about how encodings._cache does not have a 'get' method since it is now None instead of a dict. The function is fine and still runs, it's just that the global state it depends on is no longer the way it assume it should be.

-Brett

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jan 19 2007) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

More information about the Python-Dev mailing list