[Python-Dev] variable name resolution in exec is incorrect (original) (raw)

Guido van Rossum guido at python.org
Thu May 27 02:38:04 CEST 2010

On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

On 27/05/10 06:07, Colin H wrote:

In original Python, the snippet would have given an error whether you thought of it as being in a class or function context, which is how anyone who knew Python then would have expected. Consistency is not a bug. When nested function namespaces were introduced, the behavior of exec was left unchanged. Backward compatibility is not a bug. Generally, most other behaviour did change - locals in enclosing scopes did become available in the nested function namespace, which was not backward compatible.  Why is a special case made to retain consistency and backward compatibility for code run using exec()? It's all python code. Inconsistent backward compatibility might be considered a bug. Because strings are opaque to the compiler. The lexical scoping has *no idea* what is inside the string, and the exec operation only has 3 things available to it:  - the code object compiled from the string  - the supplied globals namespace  - the supplied locals namespace It isn't a special case, it's the only way it can possible work. Consider a more complex example:  def getexecstr():  y = 3  return "print(y)"  exec(getexecstr()) Should that code work? Or consider this one:  def getexecstr():  y = 3  return "print y"  def runexecstr(strtorun):  y = 5  exec(strtorun)  runexecstr(getexecstr()) Should that work? If yes, should it print 3 or 5? Lexical scoping only works for code that is compiled as part of a single operation - the separation between the compilation of the individual string and the code defining that string means that the symbol table analysis needed for lexical scoping can't cross the boundary.

Hi Nick,

I don't think Colin was asking for such things. His use case is clarify by this post of his:

On Wed, May 26, 2010 at 7:03 AM, Colin H <hawkett at gmail.com> wrote:

A really good reason why you would want to provide a separate locals dictionary is to get access to the stuff that was defined in the exec()'d code block. Unfortunately this use case is broken by the current behaviour. The only way to get the definitions from the exec()'d code block is to supply a single dictionary, and then try to weed out the definitions from amongst all the other globals, which is very difficult if you don't know in advance what was in the code block you exec()'d.

I think here's an example of what he's asking for

def define_stuff(user_code): context = {'FOO': 42, 'BAR': 10**100} stuff = {} exec(user_code, context, stuff) return stuff

In some other place, define_stuff() is called like this:

user_code = """ EXTRA = 1.1 def func(): return FOO * BAR * EXTRA """ stuff = define_stuff(user_code) func = stuff['func'] print(func())

This can't find the EXTRA variable found. (Another example would be defining a recursive function -- the function can't find itself.)

The alternative (which Colin complains about) is for define_stuff() to use a single namespace, initialized with the context, and to return that -- but he's right that in that case FOO and BAR would be exported as part of stuff, which may not be the intention. (Another solution would be to put FOO and BAR in the builtins dict -- but that has other problems of course.)

So put simply - the bug is that a class namespace is used, but its not a class.

Well, really, the bug is that no closure is created even though there are separate globals and locals. Unfortunately this is because while at the point where the code is executed the two different contexts are clearly present, at the point where it is compiled this is not the case -- the compiler normally uses syntactic clues to decide whether to generate code using closures, in particular, the presence of nested functions. Note that define_stuff() could be equivalently written like this, where it is more obvious that the compiler doesn't know about the separate namespaces:

def define_stuff(user_code): context = {...} stuff = {} compiled_code = compile(user_code, "", "exec") exec(user_code, context, stuff) return stuff

This is not easy to fix. The best short-term work-around is probably a hack like this:

def define_stuff(user_code): context = {...} stuff = {} stuff.update(context) exec(user_code, stuff) for key in context: if key in stuff and stuff[key] == context[key]: del stuff[key] return stuff

-- --Guido van Rossum (python.org/~guido)

More information about the Python-Dev mailing list