[Python-Dev] Assign(expr* targets, expr value) (original) (raw)

anatoly techtonik techtonik at gmail.com
Fri Nov 22 12:21:49 CET 2013


On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin at python.org> wrote:

2013/11/15 anatoly techtonik <techtonik at gmail.com>:

On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:

2013/11/12 anatoly techtonik <techtonik at gmail.com>:

On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin at python.org> wrote:

2013/11/10 anatoly techtonik <techtonik at gmail.com>:

http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl

In Assign(expr* targets, expr value), why the first argument is a list? x = y = 42 Thanks. Speaking of this ASDL. expr* targets means that multiple entities of expr under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But expr definition contains things that can not be used as left side assignment targets: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, exprcontext ctx) | Subscript(expr value, slice slice, exprcontext ctx) | Starred(expr value, exprcontext ctx) | Name(identifier id, exprcontext ctx) | List(expr* elts, exprcontext ctx) | Tuple(expr* elts, exprcontext ctx) If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve? Only valid expression targets are allowed during AST construction. See setexprcontext in ast.c. Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html There is no setexprcontext in my copy of CPython code, which seems to be some alpha of Python 3.4 It's actually called setcontext.

Ok. So what is the process?

SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE

Is that right?

Is it possible to fix ADSL to move expr that are allowed in Assign into expr subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?

I'm not sure what you mean by an expr subset. Transform this: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, exprcontext ctx) | Subscript(expr value, slice slice, exprcontext ctx) | Starred(expr value, exprcontext ctx) | Name(identifier id, exprcontext ctx) | List(expr* elts, exprcontext ctx) | Tuple(expr* elts, exprcontext ctx) to this: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | exprasgn exprasgn = Attribute(expr value, identifier attr, exprcontext ctx) | Subscript(expr value, slice slice, exprcontext ctx) | Starred(expr value, exprcontext ctx) | Name(identifier id, exprcontext ctx) | List(expr* elts, exprcontext ctx) | Tuple(expr* elts, exprcontext ctx) I doubt ASDL will let you do that.

asdl.py is plain broken - wrong number of arguments passed to output function asdl_c.py worked ok with fixed ASDL and generated - diff attached. I don't know what to check further - on Windows without Visual Studio.

anatoly t. -------------- next part -------------- --- C:/__py/_pydotorg/devinabox/cpython/Parser/Python-ast.c Fri Nov 22 14:19:30 2013 +++ C:/__py/_pydotorg/devinabox/cpython/Parser/Python-ast.c-patched Fri Nov 22 14:19:11 2013 @@ -161,10 +161,6 @@ static PyTypeObject *Break_type; static PyTypeObject *Continue_type; static PyTypeObject *expr_type; -static char *expr_attributes[] = {

-}; static PyObject* ast2obj_expr(void*); static PyTypeObject *BoolOp_type; _Py_IDENTIFIER(values); @@ -276,6 +272,13 @@ "value", }; static PyTypeObject *Ellipsis_type; +static PyTypeObject *expr_asgn_type; +static PyTypeObject *expr_asgn_type; +static char *expr_asgn_attributes[] = {

+}; +static PyObject* ast2obj_expr_asgn(void*); static PyTypeObject *Attribute_type; _Py_IDENTIFIER(attr); _Py_IDENTIFIER(ctx); @@ -853,7 +856,7 @@ if (!Continue_type) return 0; expr_type = make_type("expr", &AST_type, NULL, 0); if (!expr_type) return 0;

static int obj2ast_slice(PyObject* obj, slice_ty* out, PyArena* arena); @@ -1568,8 +1579,7 @@ }

expr_ty -BoolOp(boolop_ty op, asdl_seq * values, int lineno, int col_offset, PyArena

+BoolOp(boolop_ty op, asdl_seq * values, PyArena *arena) { expr_ty p; if (!op) { @@ -1583,14 +1593,11 @@ p->kind = BoolOp_kind; p->v.BoolOp.op = op; p->v.BoolOp.values = values;

expr_ty -BinOp(expr_ty left, operator_ty op, expr_ty right, int lineno, int col_offset,

+BinOp(expr_ty left, operator_ty op, expr_ty right, PyArena *arena) { expr_ty p; if (!left) { @@ -1615,14 +1622,11 @@ p->v.BinOp.left = left; p->v.BinOp.op = op; p->v.BinOp.right = right;

expr_ty -UnaryOp(unaryop_ty op, expr_ty operand, int lineno, int col_offset, PyArena

+UnaryOp(unaryop_ty op, expr_ty operand, PyArena *arena) { expr_ty p; if (!op) { @@ -1641,14 +1645,11 @@ p->kind = UnaryOp_kind; p->v.UnaryOp.op = op; p->v.UnaryOp.operand = operand;

expr_ty -Lambda(arguments_ty args, expr_ty body, int lineno, int col_offset, PyArena

+Lambda(arguments_ty args, expr_ty body, PyArena *arena) { expr_ty p; if (!args) { @@ -1667,14 +1668,11 @@ p->kind = Lambda_kind; p->v.Lambda.args = args; p->v.Lambda.body = body;

expr_ty -IfExp(expr_ty test, expr_ty body, expr_ty orelse, int lineno, int col_offset,

+IfExp(expr_ty test, expr_ty body, expr_ty orelse, PyArena *arena) { expr_ty p; if (!test) { @@ -1699,14 +1697,11 @@ p->v.IfExp.test = test; p->v.IfExp.body = body; p->v.IfExp.orelse = orelse;

expr_ty -Dict(asdl_seq * keys, asdl_seq * values, int lineno, int col_offset, PyArena

+Dict(asdl_seq * keys, asdl_seq * values, PyArena *arena) { expr_ty p; p = (expr_ty)PyArena_Malloc(arena, sizeof(*p)); @@ -1715,13 +1710,11 @@ p->kind = Dict_kind; p->v.Dict.keys = keys; p->v.Dict.values = values;

expr_ty -Set(asdl_seq * elts, int lineno, int col_offset, PyArena *arena) +Set(asdl_seq * elts, PyArena *arena) { expr_ty p; p = (expr_ty)PyArena_Malloc(arena, sizeof(*p)); @@ -1729,14 +1722,11 @@ return NULL; p->kind = Set_kind; p->v.Set.elts = elts;

expr_ty -ListComp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset,

+ListComp(expr_ty elt, asdl_seq * generators, PyArena *arena) { expr_ty p; if (!elt) { @@ -1750,14 +1740,11 @@ p->kind = ListComp_kind; p->v.ListComp.elt = elt; p->v.ListComp.generators = generators;

expr_ty -SetComp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena

+SetComp(expr_ty elt, asdl_seq * generators, PyArena *arena) { expr_ty p; if (!elt) { @@ -1771,14 +1758,11 @@ p->kind = SetComp_kind; p->v.SetComp.elt = elt; p->v.SetComp.generators = generators;

expr_ty -DictComp(expr_ty key, expr_ty value, asdl_seq * generators, int lineno, int

+DictComp(expr_ty key, expr_ty value, asdl_seq * generators, PyArena *arena) { expr_ty p; if (!key) { @@ -1798,14 +1782,11 @@ p->v.DictComp.key = key; p->v.DictComp.value = value; p->v.DictComp.generators = generators;

expr_ty -GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset,

+GeneratorExp(expr_ty elt, asdl_seq * generators, PyArena *arena) { expr_ty p; if (!elt) { @@ -1819,13 +1800,11 @@ p->kind = GeneratorExp_kind; p->v.GeneratorExp.elt = elt; p->v.GeneratorExp.generators = generators;

expr_ty -Yield(expr_ty value, int lineno, int col_offset, PyArena *arena) +Yield(expr_ty value, PyArena *arena) { expr_ty p; p = (expr_ty)PyArena_Malloc(arena, sizeof(*p)); @@ -1833,13 +1812,11 @@ return NULL; p->kind = Yield_kind; p->v.Yield.value = value;

expr_ty -YieldFrom(expr_ty value, int lineno, int col_offset, PyArena *arena) +YieldFrom(expr_ty value, PyArena *arena) { expr_ty p; if (!value) { @@ -1852,14 +1829,12 @@ return NULL; p->kind = YieldFrom_kind; p->v.YieldFrom.value = value;

expr_ty -Compare(expr_ty left, asdl_int_seq * ops, asdl_seq * comparators, int lineno,

+Compare(expr_ty left, asdl_int_seq * ops, asdl_seq * comparators, PyArena

{ expr_ty p; if (!left) { @@ -1874,14 +1849,12 @@ p->v.Compare.left = left; p->v.Compare.ops = ops; p->v.Compare.comparators = comparators;

expr_ty Call(expr_ty func, asdl_seq * args, asdl_seq * keywords, expr_ty starargs,

{ expr_ty p; if (!func) { @@ -1898,13 +1871,11 @@ p->v.Call.keywords = keywords; p->v.Call.starargs = starargs; p->v.Call.kwargs = kwargs;

expr_ty -Num(object n, int lineno, int col_offset, PyArena *arena) +Num(object n, PyArena *arena) { expr_ty p; if (!n) { @@ -1917,13 +1888,11 @@ return NULL; p->kind = Num_kind; p->v.Num.n = n;

expr_ty -Str(string s, int lineno, int col_offset, PyArena *arena) +Str(string s, PyArena *arena) { expr_ty p; if (!s) { @@ -1936,13 +1905,11 @@ return NULL; p->kind = Str_kind; p->v.Str.s = s;

expr_ty -Bytes(bytes s, int lineno, int col_offset, PyArena *arena) +Bytes(bytes s, PyArena *arena) { expr_ty p; if (!s) { @@ -1955,13 +1922,11 @@ return NULL; p->kind = Bytes_kind; p->v.Bytes.s = s;

expr_ty -NameConstant(singleton value, int lineno, int col_offset, PyArena *arena) +NameConstant(singleton value, PyArena *arena) { expr_ty p; if (!value) { @@ -1974,29 +1939,36 @@ return NULL; p->kind = NameConstant_kind; p->v.NameConstant.value = value;

expr_ty -Ellipsis(int lineno, int col_offset, PyArena *arena) +Ellipsis(PyArena *arena) { expr_ty p; p = (expr_ty)PyArena_Malloc(arena, sizeof(*p)); if (!p) return NULL; p->kind = Ellipsis_kind;

expr_ty +expr_asgn(PyArena *arena) +{

+} + +expr_asgn_ty Attribute(expr_ty value, identifier attr, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena) {

@@ -2012,7 +1984,7 @@ "field ctx is required for Attribute"); return NULL; }

-expr_ty +expr_asgn_ty Subscript(expr_ty value, slice_ty slice, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena) {

@@ -2044,7 +2016,7 @@ "field ctx is required for Subscript"); return NULL; }

-expr_ty +expr_asgn_ty Starred(expr_ty value, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena) {

@@ -2071,7 +2043,7 @@ "field ctx is required for Starred"); return NULL; }

-expr_ty +expr_asgn_ty Name(identifier id, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena) {

@@ -2097,7 +2069,7 @@ "field ctx is required for Name"); return NULL; }

-expr_ty +expr_asgn_ty List(asdl_seq * elts, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena) {

-expr_ty +expr_asgn_ty Tuple(asdl_seq * elts, expr_context_ty ctx, int lineno, int col_offset, PyArena *arena) {

@@ -3010,6 +2982,29 @@ result = PyType_GenericNew(Ellipsis_type, NULL, NULL); if (!result) goto failed; break;

+failed:

+} + +PyObject* +ast2obj_expr_asgn(void* _o) +{

@@ -4082,8 +4077,8 @@ targets = asdl_seq_new(len, arena); if (targets == NULL) goto failed; for (i = 0; i < len; i++) {

@@ -4844,35 +4839,11 @@ int isinstance;

 PyObject *tmp = NULL;

@@ -4916,7 +4887,7 @@ PyErr_SetString(PyExc_TypeError, "required field "values" missing from BoolOp"); return 1; }

+} + +int +obj2ast_expr_asgn(PyObject* obj, expr_asgn_ty* out, PyArena* arena) +{

@@ -5938,7 +5960,7 @@ return 0; }



More information about the Python-Dev mailing list