[Python-Dev] PEP 572: Now with 25% less reference implementation! (original) (raw)

Dmitry Malinovsky damalinov at gmail.com
Fri Apr 20 00:45:37 EDT 2018

Hello Chris, and thank you for working on this PEP!

What do you think about using variable type hints with this syntax? I tried to search through python-dev and couldn't find a single post discussing that question. If I missed it somehow, could you please include its conclusions into the PEP?

For instance, as I understand now the parser will fail on this snippet:

while data: bytes := stream.read():
    print("Received data:", data)

Do brackets help?

while (data: bytes := stream.read()):
    print("Received data:", data)

IIUC, in 3.7 It is invalid syntax to specify a type hint for a for loop item; should brackets help? Currently they don't:

Python 3.7.0b3+ (heads/3.7:7dcfd6c, Mar 30 2018, 21:30:34)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> for (x: int) in [1,2,3]:
  File "<stdin>", line 1
    for (x: int) in [1,2,3]:
      ^
SyntaxError: invalid syntax

Thanks,

On 20 Apr 2018, at 09:10, Chris Angelico <rosuav at gmail.com> wrote:

Working on the reference implementation for PEP 572 is turning out to be a massive time sink, both on my personal schedule and on the PEP's discussion. I can't just hold off all discussion on a topic until I figure out whether something is possible or not, because that could take me several days, even a week or more. And considering the massive backlash against the proposal, it seems like a poor use of my time to try to prove that something's impossible, find that I don't know enough about grammar editing to be able to say anything more than "well, I couldn't do it, but someone else might be able to", and then try to resume the discussion with no more certainty than we had before. So here's the PEP again, simplified. I'm fairly sure it's just going to be another on a growing list of rejected PEPs to my name, and I'm done with trying to argue some of these points. Either the rules get simplified, or they don't. Trying to simplify the rules and maintain perfect backward compatibility is just making the rules even more complicated. PEP 572, if accepted, will change the behaviour of certain constructs inside comprehensions, mainly due to interactions with class scope that make no sense unless you know how they're implemented internally. The official tutorial pretends that comprehensions are "equivalent to" longhand: https://docs.python.org/3/tutorial/datastructures.html?highlight=equivalent#list-comprehensions https://docs.python.org/3/howto/functional.html?highlight=equivalent#generator-expressions-and-list-comprehensions and this is an inaccuracy for the sake of simplicity. PEP 572 will make this far more accurate; the only difference is that the comprehension is inside a function. Current semantics are far more bizarre than that. Do you want absolutely 100% backward compatibility? Then reject this PEP. Or better still, keep using Python 3.7, and don't upgrade to 3.8, in case something breaks. Do you want list comprehensions that make better sense? Then accept that some code will need to change, if it tried to use the same name in multiple scopes, or tried to use ancient Python 2 semantics with a yield expression in the outermost iterable. I'm pretty much ready for pronouncement. https://www.python.org/dev/peps/pep-0572/ ChrisA PEP: 572 Title: Assignment Expressions Author: Chris Angelico <rosuav at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018, 04-Apr-2018, 17-Apr-2018

Abstract ======== This is a proposal for creating a way to assign to variables within an expression. Additionally, the precise scope of comprehensions is adjusted, to maintain consistency and follow expectations. Rationale ========= Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts. Merely introducing a way to assign as an expression would create bizarre edge cases around comprehensions, though, and to avoid the worst of the confusions, we change the definition of comprehensions, causing some edge cases to be interpreted differently, but maintaining the existing behaviour in the majority of situations. Syntax and semantics ==================== In any context where arbitrary Python expressions can be used, a **named expression** can appear. This is of the form target := expr where expr is any valid Python expression, and target is any valid assignment target. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:: # Handle a matched regex if (match := pattern.search(data)) is not None: ... # A more explicit alternative to the 2-arg form of iter() invocation while (value := readnextitem()) is not None: ... # Share a subexpression between a comprehension filter clause and its output filtereddata = [y for x in data if (y := f(x)) is not None] Differences from regular assignment statements ---------------------------------------------- Most importantly, since := is an expression, it can be used in contexts where statements are illegal, including lambda functions and comprehensions. An assignment statement can assign to multiple targets, left-to-right:: x = y = z = 0 The equivalent assignment expression is parsed as separate binary operators, and is therefore processed right-to-left, as if it were spelled thus:: assert 0 == (x := (y := (z := 0))) Augmented assignment is not supported in expression form::

x +:= 1 File "", line 1 x +:= 1 ^ SyntaxError: invalid syntax Otherwise, the semantics of assignment are identical in statement and expression forms. Alterations to comprehensions ----------------------------- The current behaviour of list/set/dict comprehensions and generator expressions has some edge cases that would behave strangely if an assignment expression were to be used. Therefore the proposed semantics are changed, removing the current edge cases, and instead altering their behaviour only in a class scope. As of Python 3.7, the outermost iterable of any comprehension is evaluated in the surrounding context, and then passed as an argument to the implicit function that evaluates the comprehension. Under this proposal, the entire body of the comprehension is evaluated in its implicit function. Names not assigned to within the comprehension are located in the surrounding scopes, as with normal lookups. As one special case, a comprehension at class scope will eagerly bind any name which is already defined in the class scope. A list comprehension can be unrolled into an equivalent function. With Python 3.7 semantics:: numbers = [x + y for x in range(3) for y in range(4)] # Is approximately equivalent to def (iterator): result = [] for x in iterator: for y in range(4): result.append(x + y) return result numbers = (iter(range(3))) Under the new semantics, this would instead be equivalent to:: def (): result = [] for x in range(3): for y in range(4): result.append(x + y) return result numbers = () When a class scope is involved, a naive transformation into a function would prevent name lookups (as the function would behave like a method):: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " prefixednames = [prefix + name for name in names] With Python 3.7 semantics, this will evaluate the outermost iterable at class scope, which will succeed; but it will evaluate everything else in a function:: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " def (iterator): result = [] for name in iterator: result.append(prefix + name) return result prefixednames = (iter(names)) The name prefix is thus searched for at global scope, ignoring the class name. Under the proposed semantics, this name will be eagerly bound; and the same early binding then handles the outermost iterable as well. The list comprehension is thus approximately equivalent to:: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " def (names=names, prefix=prefix): result = [] for name in names: result.append(prefix + name) return result prefixednames = () With list comprehensions, this is unlikely to cause any confusion. With generator expressions, this has the potential to affect behaviour, as the eager binding means that the name could be rebound between the creation of the genexp and the first call to next(). It is, however, more closely aligned to normal expectations. The effect is ONLY seen with names that are looked up from class scope; global names (eg range()) will still be late-bound as usual. One consequence of this change is that certain bugs in genexps will not be detected until the first call to next(), where today they would be caught upon creation of the generator. Recommended use-cases ===================== Simplifying list comprehensions ------------------------------- A list comprehension can map and filter efficiently by capturing the condition:: results = [(x, y, x/y) for x in inputdata if (y := f(x)) > 0] Similarly, a subexpression can be reused within the main expression, by giving it a name on first use:: stuff = [[y := f(x), x/y] for x in range(5)] # There are a number of less obvious ways to spell this in current # versions of Python, such as: # Inline helper function stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] # Extra 'for' loop - potentially could be optimized internally stuff = [[y, x/y] for x in range(5) for y in [f(x)]] # Using a mutable cache object (various forms possible) c = {} stuff = [[c.update(y=f(x)) or c['y'], x/c['y']] for x in range(5)] In all cases, the name is local to the comprehension; like iteration variables, it cannot leak out into the surrounding context. Capturing condition values -------------------------- Assignment expressions can be used to good effect in the header of an if or while statement:: # Proposed syntax while (command := input("> ")) != "quit": print("You entered:", command) # Capturing regular expression match objects # See, for instance, Lib/pydoc.py, which uses a multiline spelling # of this effect if match := re.search(pat, text): print("Found:", match.group(0)) # Reading socket data until an empty string is returned while data := sock.read(): print("Received data:", data) # Equivalent in current Python, not caring about function return value while input("> ") != "quit": print("You entered a command.") # To capture the return value in current Python demands a four-line # loop header. while True: command = input("> "); if command == "quit": break print("You entered:", command) Particularly with the while loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value. Rejected alternative proposals ============================== Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above. Alternative spellings --------------------- Broadly the same semantics as the current proposal, but spelled differently. 1. EXPR as NAME:: stuff = [[f(x) as y, x/y] for x in range(5)] Since EXPR as NAME already has meaning in except and with statements (with different semantics), this would create unnecessary confusion or require special-casing (eg to forbid assignment within the headers of these statements). 2. EXPR -> NAME:: stuff = [[f(x) -> y, x/y] for x in range(5)] This syntax is inspired by languages such as R and Haskell, and some programmable calculators. (Note that a left-facing arrow y <- f(x) is_ _not possible in Python, as it would be interpreted as less-than and unary_ _minus.) This syntax has a slight advantage over 'as' in that it does not_ _conflict with with and except statements, but otherwise is_ _equivalent._ _3. Adorning statement-local names with a leading dot::_ _stuff = [[(f(x) as .y), x/.y] for x in range(5)] # with "as"_ _stuff = [[(.y := f(x)), x/.y] for x in range(5)] # with ":="_ _This has the advantage that leaked usage can be readily detected, removing_ _some forms of syntactic ambiguity. However, this would be the only place_ _in Python where a variable's scope is encoded into its name, making_ _refactoring harder._ _4. Adding a where: to any statement to create local name bindings::_ _value = x**2 + 2*x where:_ _x = spam(1, 4, 7, q)_ _Execution order is inverted (the indented body is performed first, followed_ _by the "header"). This requires a new keyword, unless an existing keyword_ _is repurposed (most likely with:). See PEP 3150 for prior discussion_ _on this subject (with the proposed keyword being given:)._ _5. TARGET from EXPR::_ _stuff = [[y from f(x), x/y] for x in range(5)]_ _This syntax has fewer conflicts than as does (conflicting only with the_ _raise Exc from Exc notation), but is otherwise comparable to it. Instead_ _of paralleling with expr as target: (which can be useful but can also be_ _confusing), this has no parallels, but is evocative._ _Special-casing conditional statements_ _-------------------------------------_ _One of the most popular use-cases is if and while statements. Instead_ _of a more general solution, this proposal enhances the syntax of these two_ _statements to add a means of capturing the compared value::_ _if re.search(pat, text) as match:_ _print("Found:", match.group(0))_ _This works beautifully if and ONLY if the desired condition is based on the_ _truthiness of the captured value. It is thus effective for specific_ _use-cases (regex matches, socket reads that return '' when done), and_ _completely useless in more complicated cases (eg where the condition is_ _f(x) < 0 and you want to capture the value of f(x)). It also has_ _no benefit to list comprehensions._ _Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction_ _of possible use-cases, even in if/while statements._ _Special-casing comprehensions_ _-----------------------------_ _Another common use-case is comprehensions (list/set/dict, and genexps). As_ _above, proposals have been made for comprehension-specific solutions._ _1. where, let, or given::_ _stuff = [(y, x/y) where y = f(x) for x in range(5)]_ _stuff = [(y, x/y) let y = f(x) for x in range(5)]_ _stuff = [(y, x/y) given y = f(x) for x in range(5)]_ _This brings the subexpression to a location in between the 'for' loop and_ _the expression. It introduces an additional language keyword, which creates_ _conflicts. Of the three, where reads the most cleanly, but also has the_ _greatest potential for conflict (eg SQLAlchemy and numpy have where_ _methods, as does tkinter.dnd.Icon in the standard library)._ _2. with NAME = EXPR::_ _stuff = [(y, x/y) with y = f(x) for x in range(5)]_ _As above, but reusing the with keyword. Doesn't read too badly, and needs_ _no additional language keyword. Is restricted to comprehensions, though,_ _and cannot as easily be transformed into "longhand" for-loop syntax. Has_ _the C problem that an equals sign in an expression can now create a name_ _binding, rather than performing a comparison. Would raise the question of_ _why "with NAME = EXPR:" cannot be used as a statement on its own._ _3. with EXPR as NAME::_ _stuff = [(y, x/y) with f(x) as y for x in range(5)]_ _As per option 2, but using as rather than an equals sign. Aligns_ _syntactically with other uses of as for name binding, but a simple_ _transformation to for-loop longhand would create drastically different_ _semantics; the meaning of with inside a comprehension would be_ _completely different from the meaning as a stand-alone statement, while_ _retaining identical syntax._ _Regardless of the spelling chosen, this introduces a stark difference between_ _comprehensions and the equivalent unrolled long-hand form of the loop. It is_ _no longer possible to unwrap the loop into statement form without reworking_ _any name bindings. The only keyword that can be repurposed to this task is_ _with, thus giving it sneakily different semantics in a comprehension than_ _in a statement; alternatively, a new keyword is needed, with all the costs_ _therein._ _Lowering operator precedence_ _----------------------------_ _There are two logical precedences for the := operator. Either it should_ _bind as loosely as possible, as does statement-assignment; or it should bind_ _more tightly than comparison operators. Placing its precedence between the_ _comparison and arithmetic operators (to be precise: just lower than bitwise_ _OR) allows most uses inside while and if conditions to be spelled_ _without parentheses, as it is most likely that you wish to capture the value_ _of something, then perform a comparison on it::_ _pos = -1_ _while pos := buffer.find(searchterm, pos + 1) >= 0: ... Once find() returns -1, the loop terminates. If := binds as loosely as = does, this would capture the result of the comparison (generally either True or False), which is less useful. While this behaviour would be convenient in many situations, it is also harder to explain than "the := operator behaves just like the assignment statement", and as such, the precedence for := has been made as close as possible to that of =. Migration path ============== The semantic changes to list/set/dict comprehensions, and more so to generator expressions, may potentially require migration of code. In many cases, the changes simply make legal what used to raise an exception, but there are some edge cases that were previously legal and now are not, and a few corner cases with altered semantics. The Outermost Iterable ---------------------- As of Python 3.7, the outermost iterable in a comprehension is special: it is evaluated in the surrounding context, instead of inside the comprehension. Thus it is permitted to contain a yield expression, to use a name also used elsewhere, and to reference names from class scope. Also, in a genexp, the outermost iterable is pre-evaluated, but the rest of the code is not touched until the genexp is first iterated over. Class scope is now handled more generally (see above), but if other changes require the old behaviour, the iterable must be explicitly elevated from the comprehension:: # Python 3.7 def f(x): return [x for x in x if x] def g(): return [x for x in [(yield 1)]] # With PEP 572 def f(x): return [y for y in x if y] def g(): sentitem = (yield 1) return [x for x in [sentitem]] This more clearly shows that it is g(), not the comprehension, which is able to yield values (and is thus a generator function). The entire comprehension is consistently in a single scope. The following expressions would, in Python 3.7, raise exceptions immediately. With the removal of the outermost iterable's special casing, they are now equivalent to the most obvious longhand form:: gen = (x for x in rage(10)) # NameError gen = (x for x in 10) # TypeError (not iterable) gen = (x for x in range(1/0)) # ZeroDivisionError def (): for x in rage(10): yield x gen = () # No exception yet tng = next(gen) # NameError Open questions ============== Importing names into comprehensions ----------------------------------- A list comprehension can use and update local names, and they will retain their values from one iteration to another. It would be convenient to use this feature to create rolling or self-effecting data streams:: progressivesums = [total := total + value for value in data] This will fail with UnboundLocalError due to total not being initalized. Simply initializing it outside of the comprehension is insufficient - unless the comprehension is in class scope:: class X: total = 0 progressivesums = [total := total + value for value in data] At other scopes, it may be beneficial to have a way to fetch a value from the surrounding scope. Should this be automatic? Should it be controlled with a keyword? Hypothetically (and using no new keywords), this could be written:: total = 0 progressivesums = [total := total + value import nonlocal total for value in data] Translated into longhand, this would become:: total = 0 def (total=total): result = [] for value in data: result.append(total := total + value) return result progressivesums = () ie utilizing the same early-binding technique that is used at class scope. Frequently Raised Objections ============================ Why not just turn existing assignment into an expression? --------------------------------------------------------- C and its derivatives define the = operator as an expression, rather than a statement as is Python's way. This allows assignments in more contexts, including contexts where comparisons are more common. The syntactic similarity between if (x == y) and if (x = y) belies their drastically different semantics. Thus this proposal uses := to clarify the distinction. This could be used to create ugly code! --------------------------------------- So can anything else. This is a tool, and it is up to the programmer to use it where it makes sense, and not use it where superior constructs can be used. With assignment expressions, why bother with assignment statements? ------------------------------------------------------------------- The two forms have different flexibilities. The := operator can be used inside a larger expression; the = statement can be augmented to += and its friends. The assignment statement is a clear declaration of intent: this value is to be assigned to this target, and that's it. Why not use a sublocal scope and prevent namespace pollution? ------------------------------------------------------------- Previous revisions of this proposal involved sublocal scope (restricted to a single statement), preventing name leakage and namespace pollution. While a definite advantage in a number of situations, this increases complexity in many others, and the costs are not justified by the benefits. In the interests of language simplicity, the name bindings created here are exactly equivalent to any other name bindings, including that usage at class or module scope will create externally-visible names. This is no different from for loops or other constructs, and can be solved the same way: del the name once it is no longer needed, or prefix it with an underscore. Names bound within a comprehension are local to that comprehension, even in the outermost iterable, and can thus be used freely without polluting the surrounding namespace. (The author wishes to thank Guido van Rossum and Christoph Groth for their suggestions to move the proposal in this direction. [2]) Style guide recommendations =========================== As this adds another way to spell some of the same effects as can already be done, it is worth noting a few broad recommendations. These could be included in PEP 8 and/or other style guides. 1. If either assignment statements or assignment expressions can be used, prefer statements; they are a clear declaration of intent. 2. If using assignment expressions would lead to ambiguity about execution order, restructure it to use statements instead. Acknowledgements ================ The author wishes to thank Guido van Rossum and Nick Coghlan for their considerable contributions to this proposal, and to members of the core-mentorship mailing list for assistance with implementation. References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/assignment-expressions) .. [2] Pivotal post regarding inline assignment semantics (https://mail.python.org/pipermail/python-ideas/2018-March/049409.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/damalinov%40gmail.com

More information about the Python-Dev mailing list