[Python-Dev] [PEP 558] thinking through locals() semantics (original) (raw)

Guido van Rossum guido at python.org
Mon May 27 12:15:37 EDT 2019

I re-ran your examples and found that some of them fail.

On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith <njs at pobox.com> wrote:

First, I want to say: I'm very happy with PEP 558's changes to flocals. It solves the weird threading bugs, and exposes the fundamental operations you need for debugging in a simple and clean way, while leaving a lot of implementation flexibility for future Python VMs. It's a huge improvement over what we had before.

I'm not as sure about the locals() parts of the proposal. It might be fine, but there are some complex trade-offs here that I'm still trying to wrap my head around. The rest of this document is me thinking out loud to try to clarify these issues.

##### What are we trying to solve? There are two major questions, which are somewhat distinct: - What should the behavior of locals() be in CPython? - How much of that should be part of the language definition, vs CPython implementation details? The status quo is that for locals() inside function scope, the behavior is quite complex and subtle, and it's entirely implementation defined. In the current PEP draft, there are some small changes to the semantics, and also it promotes them becoming part of the official language semantics. I think the first question, about semantics, is the more important one. If we're promoting them to the language definition, the main effect is just to make it more important we get the semantics right. ##### What are the PEP's proposed semantics for locals()? They're kinda subtle. [Nick: please double-check this section, both for errors and because I think it includes some edge cases that the PEP currently doesn't mention.] For module/class scopes, locals() has always returned a mapping object which acts as a "source of truth" for the actual local environment – mutating the environment directly changes the mapping object, and vice-versa. That's not going to change. In function scopes, things are more complicated. The *local environment* is conceptually well-defined, and includes: - local variables (current source of truth: "fast locals" array) - closed-over variables (current source of truth: cell objects) - any arbitrary key/values written to frame.flocals that don't correspond to local or closed-over variables, e.g. you can do frame.flocals[object()] = 10, and then later read it out again. However, the mapping returned by locals() does not directly reflect this local environment. Instead, each function frame has a dict associated with it. locals() returns this dict. The dict always holds any non-local/non-closed-over variables, and also, in certain circumstances, we write a snapshot of local and closed-over variables back into the dict. Specifically, we write back: - Whenever locals() is called - Whenever exec() or eval() is called without passing an explicit locals argument - After every trace/profile event, if a Python-level tracing/profiling function is registered. (Note: in CPython, the use of Python-level tracing/profiling functions is extremely rare. It's more common in alternative implementations like PyPy. For example, the coverage package uses a C-level tracing function on CPython, which does not trigger locals updates, but on PyPy it uses a Python-level tracing function, which does trigger updates.) In addition, the PEP doesn't say, but I think that any writes to flocals immediately update both the environment and the locals dict. These semantics have some surprising consequences. Most obviously, in function scope (unlike other scopes), mutating locals() does not affect the actual local environment: def f(): a = 1 locals()["a"] = 2 assert a == 1 The writeback rules can also produce surprising results: def f(): loc1 = locals() # Since it's a snapshot created at the time of the call # to locals(), it doesn't contain 'loc1': assert "loc1" not in loc1 loc2 = locals() # Now loc1 has changed: assert "loc1" in loc1 However, the results here are totally different if a Python-level tracing/profiling function is installed – in particular, the first assertion fails. The interaction between flocals and and locals() is also subtle: def f(): a = 1 loc = locals() assert "loc" not in loc # Regular variable updates don't affect 'loc' a = 2 assert loc["a"] == 1 # But debugging updates do: sys.getframe().flocals["a"] = 3 assert a == 3

That assert fails; a is still 2 here for me.

assert loc["a"] == 3 # But it's not a full writeback assert "loc" not in loc # Mutating 'loc' doesn't affect flocals: loc["a"] = 1 assert sys.getframe().flocals["a"] == 1 # Except when it does: loc["b"] = 3 assert sys.getframe().flocals["b"] == 3

All of this can be explained by realizing loc is sys._getframe().f_locals. IOW locals() always returns the dict in f_locals.

Again, the results here are totally different if a Python-level tracing/profiling function is installed.

And you can also hit these subtleties via 'exec' and 'eval': def f(): a = 1 loc = locals() assert "loc" not in loc # exec() triggers writeback, and then mutates the locals dict exec("a = 2; b = 3") # So now the current environment has been reflected into 'loc' assert "loc" in loc # Also loc["a"] has been changed to reflect the exec'ed assignments assert loc["a"] == 2 # But if we look at the actual environment, directly or via # flocals, we can see that 'a' has not changed: assert a == 1 assert sys.getframe().flocals["a"] == 1 # loc["b"] changed as well: assert loc["b"] == 3 # And this does show up in flocals: assert sys.getframe().flocals["b"] == 3

This works indeed. My understanding is that the bytecode interpreter, when accessing the value of a local variable, ignores f_locals and always uses the "fast" array. But exec() and eval() don't use fast locals, their code is always compiled as if it appears in a module-level scope.

While the interpreter is running and no debugger is active, in a function scope f_locals is not used at all, the interpreter only interacts with the fast array and the cells. It is initialized by the first locals() call for a function scope, and locals() copies the fast array and the cells into it. Subsequent calls in the same function scope keep the same value for f_locals and re-copy fast and cells into it. This also clears out deleted local variables and emptied cells, but leaves "strange" keys (like "b" in the examples) unchanged.

The truly weird case happen when Python-level tracers are present, then the contents of f_locals is written back to the fast array and cells at certain points. This is intended for use by pdb (am I the only user of pdb left in the world?), so one can step through a function and mutate local variables. I find this essential in some cases.

Of course, many of these edge cases are pretty obscure, so it's not clear how much they matter. But I think we can at least agree that this isn't the one obvious way to do it :-).

##### What's the landscape of possible semantics? I did some brainstorming, and came up with 4 sets of semantics that seem plausible enough to at least consider: - [PEP]: the semantics in the current PEP draft.

To be absolutely clear this copies the fast array and cells to f_locals when locals() is called, but never copies back, except when Python-level tracing/profiling is on.

- [PEP-minus-tracing]: same as [PEP], except dropping the writeback on Python-level trace/profile events.

But this still copies the fast array and cells to f_locals when a Python trace function is called, right? It just doesn't write back.

- [snapshot]: in function scope, each call to locals() returns a new, static snapshot of the local environment, removing all this writeback stuff. Something like:

def locals(): frame = getcallerframe() if isfunctionscope(frame): # make a point-in-time copy of the "live" proxy object return dict(frame.flocals) else: # in module/class scope, return the actual local environment return frame.flocals

This is the most extreme variant, and in this case there is no point in having f_locals at all for a function scope (since nothing uses it). I'm not 100% sure that you understand this.

- [proxy]: Simply return the .flocals object, so in all contexts locals() returns a live mutable view of the actual environment:

def locals(): return getcallerframe().flocals

So this is PEP without any writeback. But there is still copying from the fast array and cells to f_locals. Does that only happen when locals() is called? Or also when a Python-level trace/profiling function is called?

My problem with all variants except what's in the PEP is that it would leave pdb no way (short of calling into the C API using ctypes) of writing back local variables.

##### How to evaluate our options? I can think of a lot of criteria that all-else-being-equal we would like Python to meet. (Of course, in practice they conflict.) Consistency across APIs: it's surprising if locals() and frame.flocals do different things. This argues for [proxy]. Consistency across contexts: it's surprising if locals() has acts differently in module/class scope versus function scope. This argues for [proxy]. Consistent behavior when the environment shifts, or small maintenance changes are made: it's nice if code that works today keeps working tomorrow. On this criterion, I think [snapshot] > [proxy] > [PEP-minus-tracing] >>> [PEP]. [PEP] is particularly bad here because a very rare environmental change that almost no-one tests and mostly only happens when debugging (i.e., enabling tracing) causes a radical change in semantics. Simplicity of explaining to users: all else being equal, it's nice if our docs are short and clear and the language fits in your head. On this criterion, I think: [proxy] > [snapshot] > [PEP-minus-tracing] > [PEP]. As evidence that the current behavior is confusing, see: - Ned gets confused and writes a long blog post after he figures it out: https://nedbatchelder.com/blog/201211/trickylocals.html - A linter that warns against mutating locals(): https://lgtm.com/rules/10030096/ Simplicity of implementation: "If the implementation is easy to explain, it may be a good idea." Since we need the proxy code anyway to implement flocals, I think it's: [proxy] (free) > [snapshot] (one 'if' statement) > ([PEP] = [PEP-minus-tracing]). Impact on other interpreter implementations: all else being equal, we'd like to give new interpreters maximal freedom to do clever things. (And local variables are a place where language VMs often expend a lot of cleverness.) [proxy] and [snapshot] are both easily implemented in terms of flocals, so they basically don't constrain alternative implementations at all. I'm not as sure about [PEP] and [PEP-minus-tracing]. I originally thought they must be horrible. On further thought, I'm not convinced they're that bad, since the need to support people doing silly stuff like frame.flocals[object()] = 10 means that implementations will already need to sometimes attach something like a dict object to their function frames. But perhaps alternative implementations would like to disallow this, or are OK with making it really slow but care about locals() performance more. Anyway, it's definitely ([proxy] = [snapshot]) > ([PEP] = [PEP-minus-tracing]), but I'm not sure whether the '>' is large or small. Backwards compatibility: help(locals) says: NOTE: Whether or not updates to this dictionary will affect name lookups in the local scope and vice-versa is implementation dependent and not covered by any backwards compatibility guarantees. So that claims that there are ~no backwards compatibility issues here. I'm going to ignore that; no matter what the docs say, we still don't want to break everyone's code. And unfortunately, I can't think of any realistic way to do a gradual transition, with like deprecation warnings and all that (can anyone else?), so whatever we do will be a flag-day change. Of our four options, [PEP] is intuitively the closest to what CPython has traditionally done. But what exactly breaks under the different approaches? I'll split this off into its own section.

##### Backwards compatibility I'll split this into three parts: code that treats locals() as read-only, exec()/eval(), and code that mutates locals(). I believe (but haven't checked) that the majority of uses of locals() are in simple cases like: def f(): .... print("{a} {b}".format(**locals())) Luckily, this code remains totally fine under all four of our candidates. exec() and eval() are an interesting case. In Python 2, exec'ing some assignments actually did mutate the local environment, e.g. you could do: # Python 2 def f(): exec "a = 1" assert a == 1 In Python 3, this was changed, so now exec() inside a function cannot mutate the enclosing scope. We got some bug reports about this change, and there are a number of questions on stackoverflow about it, e.g.: - https://bugs.python.org/issue4831 - https://stackoverflow.com/questions/52217525/how-can-i-change-the-value-of-variable-in-a-exec-function - https://stackoverflow.com/questions/50995581/eval-exec-with-assigning-variable-python In all released versions of Python, eval() was syntactically unable to rebind variables, so eval()'s interaction with the local environment was undefined. However, in 3.8, eval() will be able to rebind variables using the ':=' operator, so this interaction will become user-visible. Presumably we'll want eval() to match exec(). OK, with that background out of the way, let's look at our candidates. If we adopt [proxy], then that will mean exec()-inside-functions will go back to the Python 2 behavior, where executing assignments in the enclosing scope actually changes the enclosing scope. This will likely break some code out there that's relying on the Python 3 behavior, though I don't know how common that is. (I'm guessing not too common? Using the same variable inside and outside an 'exec' and trusting that they won't be the same seems like an unusual thing to do. But I don't know.) With [PEP] and [PEP-minus-tracing], exec() is totally unchanged. With [snapshot], there's technically a small difference: if you call locals() and then exec(), the exec() no longer triggers an implicit writeback to the dict that locals() returned. I think we can ignore this, and say that for all three of these, exec() is unlikely to produce backwards compatibility issues. OK, finally, let's talk about code that calls locals() and then mutates the return value. The main difference between our candidates is how they handle mutation, so this seems like the most important case to focus on. Conveniently, Mark Shannon and friends have statically analyzed a large corpus of Python code and made a list of cases where people do this: https://lgtm.com/rules/10030096/alerts/ Thanks! I haven't gone through the whole list, but I read through the first few in the hopes of getting a better sense of what kind of code does this in the real world and how it would be impacted by our different options. https://lgtm.com/projects/g/pydata/xarray/snapshot/a2ac6af744584c8afed3d56d00c7d6ace85341d9/files/xarray/plot/plot.py?sort=name&dir=ASC&mode=heatmap#L701 Current: raises if a Python-level trace/profile function is set [PEP]: raises if a Python-level trace/profile function is set [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: always raises Comment: uses locals() to capture a bunch of passed in kwargs so it can pass them as **kwargs to another function, and treats them like a snapshot. The authors were clearly aware of the dangers, because this pattern appears multiple times in this file, and all the other places make an explicit copy of locals() before using it, but this place apparently got missed. Fix is trivial: just do that here too. https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-codegen/src/main/resources/python/api.mustache Current: ok [PEP]: ok [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: ok Comment: this is inside a tool used to generate Python wrappers for REST APIs. The vast majority of entries in the lgtm database are from code generated by this tool. This was tricky to analyze, because it's complex templated code, and it does mutate locals(). But I'm pretty confident that the mutations end up just... not mattering, and all possible generated code works under all of our candidates. Which is lucky, because if we did have to fix this it wouldn't be trivial: fixing up the code generator itself wouldn't be too hard, but it'll take years for everyone to regenerate their old wrappers. https://lgtm.com/projects/g/saltstack/salt/snapshot/bb0950e5eafbb897c8e969e3f20fd297d8ba2006/files/salt/utils/thin.py?sort=name&dir=ASC&mode=heatmap#L258 Current: ok [PEP]: ok [PEP-minus-tracing]: ok [snapshot]: raises [proxy]: ok Comment: the use of locals() here is totally superfluous – it repeatedly reads and writes to locals()[mod], and in all cases this could be replaced by a simple variable, or any other dict. And it's careful not to assign to any name that matches an actual local variable, so it works fine with [proxy] too. But it does assume that multiple calls to locals() return the same dict, so [snapshot] breaks it. In this case the fix is trivial: just use a variable. https://lgtm.com/projects/g/materialsproject/pymatgen/snapshot/fd6900ed1040a4d35f2cf2b3506e6e3d7cdf77db/files/pymatgen/ext/jhu.py?sort=name&dir=ASC&mode=heatmap#L52 Current: buggy if a trace/profile function is set [PEP]: buggy if a trace/profile function is set [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: raises Comment: Another example of collecting kwargs into a dict. Actually this code is always buggy, because they seem to think that dict.pop("a", "b") removes both "a" and "b" from the dict... but it would raise an exception on [proxy], because they use one of those local variables after popping it from locals(). Fix is trivial: take an explicit snapshot of locals() before modifying it. ##### Conclusion so far [PEP-minus-tracing] seems to strictly dominate [PEP]. It's equal or better on all criteria, and actually more compatible with all the legacy code I looked at, even though it's technically less consistent with what CPython used to do. Unless someone points out some new argument, I think we can reject the writeback-when-tracing part of the PEP draft. Choosing between the remaining three is more of a judgement call. I'm leaning towards saying that on net, [snapshot] beats [PEP-minus-tracing]: it's dramatically simpler, and the backwards incompatibilities that we've found so far seem pretty minor, on par with what we do in every point release. (In fact, in 3/4 of the cases I looked at, [snapshot] is actually what users seemed to trying to use in the first place.) For [proxy] versus [snapshot], a lot depends on what we think of changing the semantics of exec(). [proxy] is definitely more consistent and elegant, and if we could go back in time I think it's what we'd have done from the start. Its compatibility is maybe a bit worse than [snapshot] on non-exec() cases, but this seems pretty minor overall (it often doesn't matter, and if it does just write dict(locals()) instead of locals(), like you would in non-function scope). But the change in exec() semantics is an actual language change, even though it may not affect much real code, so that's what stands out for me. I'd very much like to hear about any considerations I've missed, and any opinions on the "judgement call" part.

To me it looks as if you're working with the wrong mental model of how it currently works. Unfortunately there's no way to find out how it currently works without reading the source code, since from Python code you cannot access the fast array and cells directly, and by just observing f_locals you get a slightly wrong picture.

Perhaps we should give the two operations that copy from/to the fast array and cells names, so we can talk about them more easily.

Sorry I didn't snip stuff.

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20190527/b6f6ba58/attachment-0001.html>

More information about the Python-Dev mailing list