[Python-Dev] cpython: Implement PEP 380 (original) (raw)

[Python-Dev] cpython: Implement PEP 380 - 'yield from' (closes #11682)

Georg Brandl g.brandl at gmx.net
Fri Jan 13 16:17:09 CET 2012

Caution, long review ahead.

On 01/13/2012 12:43 PM, nick.coghlan wrote:

http://hg.python.org/cpython/rev/d64ac9ab4cd0 changeset: 74356:d64ac9ab4cd0 user: Nick Coghlan <ncoghlan at gmail.com> date: Fri Jan 13 21:43:40 2012 +1000 summary: Implement PEP 380 - 'yield from' (closes #11682) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -318,7 +318,7 @@

There should probably be a "versionadded" somewhere on this page.

.. productionlist:: yieldatom: "(" yieldexpression ")" - yieldexpression: "yield" [expressionlist] + yieldexpression: "yield" [expressionlist | "from" expression]

The :keyword:yield expression is only used when defining a generator function, and can only be used in the body of a function definition. Using a @@ -336,7 +336,10 @@ the generator's methods, the function can proceed exactly as if the :keyword:yield expression was just another external call. The value of the :keyword:yield expression after resuming depends on the method which resumed -the execution. +the execution. If :meth:_next_ is used (typically via either a +:keyword:for or the :func:next builtin) then the result is :const:None, +otherwise, if :meth:send is used, then the result will be the value passed +in to that method. .. index:: single: coroutine @@ -346,12 +349,29 @@ where should the execution continue after it yields; the control is always transferred to the generator's caller. -The :keyword:yield statement is allowed in the :keyword:try clause of a +:keyword:yield expressions are allowed in the :keyword:try clause of a :keyword:try ... :keyword:finally construct. If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:close method will be called, allowing any pending :keyword:finally clauses to execute. +When yield from expression is used, it treats the supplied expression as +a subiterator. All values produced by that subiterator are passed directly +to the caller of the current generator's methods. Any values passed in with +:meth:send and any exceptions passed in with :meth:throw are passed to +the underlying iterator if it has the appropriate methods. If this is not the +case, then :meth:send will raise :exc:AttributeError or :exc:TypeError, +while :meth:throw will just raise the passed in exception immediately. + +When the underlying iterator is complete, the :attr:~StopIteration.value +attribute of the raised :exc:StopIteration instance becomes the value of +the yield expression. It can be either set explicitly when raising +:exc:StopIteration, or automatically when the sub-iterator is a generator +(by returning a value from the sub-generator). + +The parentheses can be omitted when the :keyword:yield expression is the +sole expression on the right hand side of an assignment statement. + .. index:: object: generator The following generator's methods can be used to control the execution of a @@ -444,6 +464,10 @@ The proposal to enhance the API and syntax of generators, making them usable as simple coroutines. + :pep:0380 - Syntax for Delegating to a Subgenerator + The proposal to introduce the :token:yieldfrom syntax, making delegation + to sub-generators easy. + .. primaries: PEP 3155: Qualified name for classes and functions ================================================== @@ -208,7 +224,6 @@ how they might be accessible from the global scope. Example with (non-bound) methods:: - >>> class C: ... def meth(self): ... pass

This looks like a spurious (and syntax-breaking) change.

diff --git a/Grammar/Grammar b/Grammar/Grammar --- a/Grammar/Grammar +++ b/Grammar/Grammar @@ -121,7 +121,7 @@ |'**' test) # The reason that keywords are test nodes instead of NAME is that using NAME # results in an ambiguity. ast.c makes sure it's a NAME. -argument: test [compfor] | test '=' test # Really [keyword '='] test +argument: (test) [compfor] | test '=' test # Really [keyword '='] test

This looks like a change without effect?

diff --git a/Include/genobject.h b/Include/genobject.h --- a/Include/genobject.h +++ b/Include/genobject.h @@ -11,20 +11,20 @@ struct frame; /* Avoid including frameobject.h */

typedef struct { - PyObjectHEAD - /* The gi prefix is intended to remind of generator-iterator. */ + PyObjectHEAD + /* The gi prefix is intended to remind of generator-iterator. */ - /* Note: giframe can be NULL if the generator is "finished" */ - struct frame *giframe; + /* Note: giframe can be NULL if the generator is "finished" */ + struct frame *giframe; - /* True if generator is being executed. */ - int girunning; + /* True if generator is being executed. */ + int girunning; - /* The code object backing the generator */ - PyObject *gicode; + /* The code object backing the generator */ + PyObject *gicode; - /* List of weak reference. */ - PyObject *giweakreflist; + /* List of weak reference. */ + PyObject *giweakreflist; } PyGenObject;

While these change tabs into spaces, it should be 4 spaces, not 8.

@@ -34,6 +34,7 @@

PyAPIFUNC(PyObject *) PyGenNew(struct frame *); PyAPIFUNC(int) PyGenNeedsFinalizing(PyGenObject *); +PyAPIFUNC(int) PyGenFetchStopIterationValue(PyObject **); _#ifdef cplusplus }

Does this API need to be public? If yes, it needs to be documented.

diff --git a/Include/opcode.h b/Include/opcode.h --- a/Include/opcode.h +++ b/Include/opcode.h @@ -7,116 +7,117 @@

/* Instruction opcodes for compiled code */ -#define POPTOP 1 -#define ROTTWO 2 -#define ROTTHREE 3 -#define DUPTOP 4 +#define POPTOP 1 +#define ROTTWO 2 +#define ROTTHREE 3 +#define DUPTOP 4 #define DUPTOPTWO 5 -#define NOP 9 +#define NOP 9 -#define UNARYPOSITIVE 10 -#define UNARYNEGATIVE 11 -#define UNARYNOT 12 +#define UNARYPOSITIVE 10 +#define UNARYNEGATIVE 11 +#define UNARYNOT 12 -#define UNARYINVERT 15 +#define UNARYINVERT 15 -#define BINARYPOWER 19 +#define BINARYPOWER 19 -#define BINARYMULTIPLY 20 +#define BINARYMULTIPLY 20 -#define BINARYMODULO 22 -#define BINARYADD 23 -#define BINARYSUBTRACT 24 -#define BINARYSUBSCR 25 +#define BINARYMODULO 22 +#define BINARYADD 23 +#define BINARYSUBTRACT 24 +#define BINARYSUBSCR 25 #define BINARYFLOORDIVIDE 26 #define BINARYTRUEDIVIDE 27 #define INPLACEFLOORDIVIDE 28 #define INPLACETRUEDIVIDE 29 -#define STOREMAP 54 -#define INPLACEADD 55 -#define INPLACESUBTRACT 56 -#define INPLACEMULTIPLY 57 +#define STOREMAP 54 +#define INPLACEADD 55 +#define INPLACESUBTRACT 56 +#define INPLACEMULTIPLY 57 -#define INPLACEMODULO 59 -#define STORESUBSCR 60 -#define DELETESUBSCR 61 +#define INPLACEMODULO 59 +#define STORESUBSCR 60 +#define DELETESUBSCR 61 -#define BINARYLSHIFT 62 -#define BINARYRSHIFT 63 -#define BINARYAND 64 -#define BINARYXOR 65 -#define BINARYOR 66 -#define INPLACEPOWER 67 -#define GETITER 68 -#define STORELOCALS 69 -#define PRINTEXPR 70 +#define BINARYLSHIFT 62 +#define BINARYRSHIFT 63 +#define BINARYAND 64 +#define BINARYXOR 65 +#define BINARYOR 66 +#define INPLACEPOWER 67 +#define GETITER 68 +#define STORELOCALS 69 +#define PRINTEXPR 70 #define LOADBUILDCLASS 71 +#define YIELDFROM 72 -#define INPLACELSHIFT 75 -#define INPLACERSHIFT 76 -#define INPLACEAND 77 -#define INPLACEXOR 78 -#define INPLACEOR 79 -#define BREAKLOOP 80 +#define INPLACELSHIFT 75 +#define INPLACERSHIFT 76 +#define INPLACEAND 77 +#define INPLACEXOR 78 +#define INPLACEOR 79 +#define BREAKLOOP 80 #define WITHCLEANUP 81 -#define RETURNVALUE 83 -#define IMPORTSTAR 84 +#define RETURNVALUE 83 +#define IMPORTSTAR 84 -#define YIELDVALUE 86 -#define POPBLOCK 87 -#define ENDFINALLY 88 -#define POPEXCEPT 89 +#define YIELDVALUE 86 +#define POPBLOCK 87 +#define ENDFINALLY 88 +#define POPEXCEPT 89 -#define HAVEARGUMENT 90 /* Opcodes from here have an argument: */ +#define HAVEARGUMENT 90 /* Opcodes from here have an argument: */ -#define STORENAME 90 /* Index in name list */ -#define DELETENAME 91 /* "" */ -#define UNPACKSEQUENCE 92 /* Number of sequence items */ -#define FORITER 93 +#define STORENAME 90 /* Index in name list */ +#define DELETENAME 91 /* "" */ +#define UNPACKSEQUENCE 92 /* Number of sequence items */ +#define FORITER 93 #define UNPACKEX 94 /* Num items before variable part + (Num items after variable part << 8) */ -#define STOREATTR 95 /* Index in name list */ -#define DELETEATTR 96 /* "" */ -#define STOREGLOBAL 97 /* "" */ -#define DELETEGLOBAL 98 /* "" */ +#define STOREATTR 95 /* Index in name list */ +#define DELETEATTR 96 /* "" */ +#define STOREGLOBAL 97 /* "" */ +#define DELETEGLOBAL 98 /* "" */ -#define LOADCONST 100 /* Index in const list */ -#define LOADNAME 101 /* Index in name list */ -#define BUILDTUPLE 102 /* Number of tuple items */ -#define BUILDLIST 103 /* Number of list items */ -#define BUILDSET 104 /* Number of set items */ -#define BUILDMAP 105 /* Always zero for now */ -#define LOADATTR 106 /* Index in name list */ -#define COMPAREOP 107 /* Comparison operator */ -#define IMPORTNAME 108 /* Index in name list */ -#define IMPORTFROM 109 /* Index in name list */ +#define LOADCONST 100 /* Index in const list */ +#define LOADNAME 101 /* Index in name list */ +#define BUILDTUPLE 102 /* Number of tuple items */ +#define BUILDLIST 103 /* Number of list items */ +#define BUILDSET 104 /* Number of set items */ +#define BUILDMAP 105 /* Always zero for now */ +#define LOADATTR 106 /* Index in name list */ +#define COMPAREOP 107 /* Comparison operator */ +#define IMPORTNAME 108 /* Index in name list */ +#define IMPORTFROM 109 /* Index in name list */ -#define JUMPFORWARD 110 /* Number of bytes to skip */ -#define JUMPIFFALSEORPOP 111 /* Target byte offset from beginning of code */ -#define JUMPIFTRUEORPOP 112 /* "" */ -#define JUMPABSOLUTE 113 /* "" */ -#define POPJUMPIFFALSE 114 /* "" */ -#define POPJUMPIFTRUE 115 /* "" */ +#define JUMPFORWARD 110 /* Number of bytes to skip */ +#define JUMPIFFALSEORPOP 111 /* Target byte offset from beginning of code */ +#define JUMPIFTRUEORPOP 112 /* "" */ +#define JUMPABSOLUTE 113 /* "" */ +#define POPJUMPIFFALSE 114 /* "" */ +#define POPJUMPIFTRUE 115 /* "" */ -#define LOADGLOBAL 116 /* Index in name list */ +#define LOADGLOBAL 116 /* Index in name list */ -#define CONTINUELOOP 119 /* Start of loop (absolute) */ -#define SETUPLOOP 120 /* Target address (relative) */ -#define SETUPEXCEPT 121 /* "" */ -#define SETUPFINALLY 122 /* "" */ +#define CONTINUELOOP 119 /* Start of loop (absolute) */ +#define SETUPLOOP 120 /* Target address (relative) */ +#define SETUPEXCEPT 121 /* "" */ +#define SETUPFINALLY 122 /* "" */ -#define LOADFAST 124 /* Local variable number */ -#define STOREFAST 125 /* Local variable number */ -#define DELETEFAST 126 /* Local variable number */ +#define LOADFAST 124 /* Local variable number */ +#define STOREFAST 125 /* Local variable number */ +#define DELETEFAST 126 /* Local variable number */ -#define RAISEVARARGS 130 /* Number of raise arguments (1, 2 or 3) */ +#define RAISEVARARGS 130 /* Number of raise arguments (1, 2 or 3) */ /* CALLFUNCTIONXXX opcodes defined below depend on this definition */ -#define CALLFUNCTION 131 /* #args + (#kwargs<<8) */ -#define MAKEFUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */ -#define BUILDSLICE 133 /* Number of items */ +#define CALLFUNCTION 131 /* #args + (#kwargs<<8) */ +#define MAKEFUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */ +#define BUILDSLICE 133 /* Number of items */ #define MAKECLOSURE 134 /* same as MAKEFUNCTION */ #define LOADCLOSURE 135 /* Load free variable from closure */

Not sure putting these and all the other cosmetic changes into an already big patch is such a good idea...

diff --git a/Include/pyerrors.h b/Include/pyerrors.h --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -51,6 +51,11 @@ Pyssizet written; /* only for BlockingIOError, -1 otherwise */ } PyOSErrorObject;

+typedef struct { + PyExceptionHEAD + PyObject *value; +} PyStopIterationObject; + /* Compatibility typedefs */ typedef PyOSErrorObject PyEnvironmentErrorObject; #ifdef MSWINDOWS @@ -380,6 +385,8 @@ const char reason / UTF-8 encoded string */ ); +/* create a StopIteration exception with the given value */ +PyAPIFUNC(PyObject *) PyStopIterationCreate(PyObject *);

About this API see below.

diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2267,7 +2267,6 @@

func = PyObjectGetAttrString(o, name); if (func == NULL) { - PyErrSetString(PyExcAttributeError, name); return 0; } @@ -2311,7 +2310,6 @@ func = PyObjectGetAttrString(o, name); if (func == NULL) { - PyErrSetString(PyExcAttributeError, name); return 0; } vastart(va, format);

These two changes also look suspiciously unrelated?

+PyObject * +PyStopIterationCreate(PyObject *value) +{ + return PyObjectCallFunctionObjArgs(PyExcStopIteration, value, NULL); +}

I think this function is rather questionable. It is only used once at all. If kept, it should rather be named _PyE{rr,xc}_CreateStopIteration. But since it's so trivial, it should be removed altogether.

diff --git a/Objects/genobject.c b/Objects/genobject.c --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -5,6 +5,9 @@ #include "structmember.h" #include "opcode.h"

+static PyObject *genclose(PyGenObject *gen, PyObject *args); +static void genundelegate(PyGenObject *gen); + static int gentraverse(PyGenObject *gen, visitproc visit, void *arg) { @@ -90,12 +93,18 @@ /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ - if (result == PyNone && f->fstacktop == NULL) { - PyDECREF(result); - result = NULL; - /* Set exception if not called by geniternext() */ - if (arg) + if (result && f->fstacktop == NULL) { + if (result == PyNone) { + /* Delay exception instantiation if we can */ PyErrSetNone(PyExcStopIteration); + } else { + PyObject *e = PyStopIterationCreate(result); + if (e != NULL) { + PyErrSetObject(PyExcStopIteration, e); + PyDECREF(e); + }

Wouldn't PyErr_SetObject(PyExc_StopIteration, value) suffice here anyway?

+/* + * If StopIteration exception is set, fetches its 'value' + * attribute if any, otherwise sets pvalue to None. + * + * Returns 0 if no exception or StopIteration is set. + * If any other exception is set, returns -1 and leaves + * pvalue unchanged. + */ + +int +PyGenFetchStopIterationValue(PyObject **pvalue) { + PyObject *et, *ev, *tb; + PyObject *value = NULL; + + if (PyErrExceptionMatches(PyExcStopIteration)) { + PyErrFetch(&et, &ev, &tb); + PyXDECREF(et); + PyXDECREF(tb); + if (ev) { + value = ((PyStopIterationObject *)ev)->value; + PyDECREF(ev); + }

PyErr_Fetch without PyErr_Restore clears the exception, that should be mentioned in the docstring.

Georg

More information about the Python-Dev mailing list