[Python-Dev] PEP 572 semantics (original) (raw)

Tim Peters tim.peters at gmail.com
Wed Jul 4 13:21:19 EDT 2018

Just a quickie:

[Steve Dower]

> The PEP uses the phrase "an assignment expression occurs in a comprehension" - what does this mean?

It's about static analysis of the source code, at compile-time, to establish scopes. So "occurs in" means your eyeballs see an assignment expression in a comprehension.

Does it occur when/where it is compiled, instantiated, or executed? This is important because where it occurs determines which scope will be modified. For sanity sake, I want to assume that it means compiled,

Probably ;-) I just don't know exactly what those distinctions mean to you.

but now what happens when that scope is gone?

Nothing new. There's no new concept of "scope" here, which is why the PEP doesn't say a whole lot about scope.

>>> def f():

> ... return (a := i for i in range(5))

> ...

Same as now, i is local to the synthetic nested function created for the genexp. The scope of a is determined by pretending the assignment occurred in the block containing the outermost (textually - static analysis) comprehension. In this case, a = anything before the return would establish that a is local to f, so that's the answer: a is local to f. If a had been declared global in f, then a in the genexp would be the same global a. And similarly if a had been declared nonlocal.in f.

In all cases the scope resolution is inherited from the closest containing non-comprehension/genexp block, with the twist if that if a name is unknown in that block, the name is established as being local to that block. So your example is actually the subtlest case.

>>> list(f())

> [0, 1, 2, 3, 4] # or a new error because the scope has gone?

Function scopes in Python have "indefinite extent", and nothing about that changes. So, ya, that's the output - same as today if you changed your example to delete the "a :=" part.

Internally, f's local a was left bound to 4, but there's no way to see that here because the genexp has been run to exhaustion and reference-counting has presumably thrown everything away by now.

a

???

Same thing typing a would result in if you had never typed list(f()).

Here's a variation:

def f():

yield (a := i for i in range(5)) yield a

Then:

g = f() list(next(g)) [0, 1, 2, 3, 4] next(g) 4

I'll push back real hard on doing the assignment in the scope where the generator is executed:

>

> >>> def dosecureop(name, numbers):

> ... authorised = checkauthorised(name)

This instance of authorized is local to do_secure_op.

... if not all(numbers):

> ... raise ValueError()

> ... if not authorised:

> ... raise SecurityError()

> ... print('You made it!')

> ...

> >>> dosecureop('whatever', (authorised := i for i in [1, 2, 3]))

And this instance of authorized is a global (because the genexp appears in top-level code, so its containing block is the module). The two instances have nothing to do with each other.

You made it!

Yup - you did!

>>> authorised

> NameError: name 'authorised' is undefined

It would display 3 instead.

From the any()/all() examples, it seems clear that the target scope for

> the assignment has to be referenced from the generator scope (but not

> for other comprehension types, which can simply do one transfer of the

> assigned name after fully evaluating all the contents).

I don't think that follows. It may in some cases. For example,

def f(): i = None # not necessary, just making i's scope obvious def g(ignore): return i+1 return [g(i := j) for j in range(3)]

While the list comprehension is executing, it needs to rebind f's i on each iteration so that the call to g() on each iteration can see i's then-current value.

Will this reference keep the frame object alive for as long as the generator exists? Can it be a weak reference? Are assignments just going to be silently ignored when the frame they should assign to is gone? I'd like to see these clarified in the main text.

Here you're potentially asking questions about how closures work in Python (in the most-likely case that an embedded assignment-statement target resolves to an enclosing function-local scope), but the PEP doesn't change anything about how closures already work. Closures are implemented via "cell objects", one per name, which already supply both "read" and "write" access to both the owning and referencing scopes.

def f(): a = 42 return (a+1 for i in range(3))

That works fine today, and a cell object is used in the synthesized genexp function to get read access to f's local a. But references to a in f also use that cell object. - the thing that lives in f's frame isn't really the binding for a, but a reference to the cell object that holds a's current binding. The cell object doesn't care whether f's frame goes away (except that the cell object's refcount drops by one when f's frame vanishes). Nothing about that changes if the synthesized genexp wants write access instead.

While a gentle introduction to how closures are implemented in Python would be a cool thing, this PEP is the last place to include one ;-)

It may help to realize that there's nothing here that can't be done today by explicitly writing nested functions with appropriate scope declarations, in a straightforward way. There's nothing inherently new.

Huh! Not so much "a quickie" after all :-( So I'll stop here for now. Thank you for the critical reading and feedback! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180704/bc74b922/attachment-0001.html>

More information about the Python-Dev mailing list