cpython: abf87e1fbc62 (original) (raw)

Mercurial > cpython

changeset 88512:abf87e1fbc62

Issue #20226: Major improvements to Argument Clinic. * You may now specify an expression as the default value for a parameter! Example: "sys.maxsize - 1". This support is intentionally quite limited; you may only use values that can be represented as static C values. * Removed "doc_default", simplified support for "c_default" and "py_default". (I'm not sure we still even need "py_default", but I'm leaving it in for now in case a use presents itself.) * Parameter lines support a trailing '\\' as a line continuation character, allowing you to break up long lines. * The argument parsing code generated when supporting optional groups now uses PyTuple_GET_SIZE instead of PyTuple_GetSize, leading to a 850% speedup in parsing. (Just kidding, this is an unmeasurable difference.) * A bugfix for the recent regression where the generated prototype from pydoc for builtins would be littered with unreadable "=<object ...>"" default values for parameters that had no default value. * Converted some asserts into proper failure messages. * Many doc improvements and fixes. [#20226]

Larry Hastings larry@hastings.org
date	Thu, 16 Jan 2014 11:32:01 -0800
parents	18a71ae9e715
children	cd3fdf21a6e4
files	Doc/c-api/arg.rst Doc/howto/clinic.rst Lib/inspect.py Lib/test/test_inspect.py Misc/NEWS Modules/_cursesmodule.c Modules/_dbmmodule.c Modules/_opcode.c Modules/_testcapimodule.c Modules/posixmodule.c Modules/zlibmodule.c Tools/clinic/clinic.py Tools/clinic/clinic_test.py
diffstat	13 files changed, 472 insertions(+), 187 deletions(-)[+] [-] Doc/c-api/arg.rst 2 Doc/howto/clinic.rst 261 Lib/inspect.py 89 Lib/test/test_inspect.py 5 Misc/NEWS 3 Modules/_cursesmodule.c 4 Modules/_dbmmodule.c 6 Modules/_opcode.c 4 Modules/_testcapimodule.c 4 Modules/posixmodule.c 4 Modules/zlibmodule.c 4 Tools/clinic/clinic.py 250 Tools/clinic/clinic_test.py 23

line wrap: on

line diff

--- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -294,6 +294,8 @@ Other objects the object pointer is stored. If the Python object does not have the required type, :exc:TypeError is raised. +.. _o_ampersand: + O& (object) [converter, anything] Convert a Python object to a C variable through a converter function. This takes two arguments: the first is a function, the second is the address of a C

--- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -127,6 +127,12 @@ 4. Cut the docstring and paste it in bet margin, with no line wider than 80 characters. (Argument Clinic will preserve indents inside the docstring.)

If the old docstring had a first line that looked like a function
signature, throw that line away. (The docstring doesn't need it
anymore--when you use help() on your builtin in the future,
the first line will be built automatically based on the function's
signature.) + Sample:: /*[clinic input] @@ -196,6 +202,10 @@ 8. Declare each of the parameters to the name_of_parameter: converter = default_value

Argument Clinic's support for "default values" is quite sophisticated;
please see :ref:the section below on default values <default_values>
for more information. + Add a blank line below the parameters. What's a "converter"? It establishes both the type @@ -513,16 +523,6 @@ The base function would now be named p[](#l2.27) and the impl function would now be named pickler_dumper_impl()``.

-The NULL default value ----------------------- - -For string and object parameters, you can set them to None to indicate -that there is no default. However, that means the C variable will be -initialized to Py_None. For convenience's sakes, there's a special -value called NULL for just this case: from Python's perspective it -behaves like a default value of None, but the C variable is initialized -with NULL. - Converting functions using PyArg_UnpackTuple -------------------------------------------- @@ -654,35 +654,70 @@ the same converters. All arguments to Argument Clinic converters are keyword-only. All Argument Clinic converters accept the following arguments: -py_default

The default value for this parameter when defined in Python.
Specifically, the value that will be used in the inspect.Signature
string.
If a default value is specified for the parameter, defaults to
repr(default), else defaults to None.
Specified as a string.

c_default
The default value for this parameter when defined in C.
Specifically, this will be the initializer for the variable declared
in the "parse function". See :ref:the section on default values <default_values>
for how to use this.
Specified as a string.

annotation
The annotation value for this parameter. Not currently supported,
because PEP 8 mandates that the Python library may not use
annotations.

-c_default

The default value for this parameter when defined in C.
Specifically, this will be the initializer for the variable declared
in the "parse function".
Specified as a string. +In addition, some converters accept additional arguments. Here is a list +of these arguments, along with their meanings: +

bitwise
Only supported for unsigned integers. The native integer value of this
Python argument will be written to the parameter without any range checking,
even for negative values.

converter
Only supported by the object converter. Specifies the name of a
:ref:C "converter function" <o_ampersand>
to use to convert this object to a native type.

encoding
Only supported for strings. Specifies the encoding to use when converting
this string from a Python str (Unicode) value into a C char * value.

-required

If a parameter takes a default value, Argument Clinic infers that the
parameter is optional. However, you may want a parameter to take a
default value in C, but not behave in Python as if the parameter is
optional. Passing in required=True to a converter tells Argument
Clinic that this parameter is not optional, even if it has a default
value.

length
Only supported for strings. If true, requests that the length of the
string be passed in to the impl function, just after the string parameter,
in a parameter named <parameter_name>_length.

nullable
Only supported for strings. If true, this parameter may also be set to
None, in which case the C parameter will be set to NULL.

subclass_of
Only supported for the object converter. Requires that the Python
value be a subclass of a Python type, as expressed in C.

types
Only supported for the object (and self) converter. Specifies
the C type that will be used to declare the variable. Default value is
"PyObject *".

(The need for required may be obviated by c_default, which is
newer but arguably a better solution.)

types
A string containing a list of Python types (and possibly pseudo-types);
this restricts the allowable Python argument to values of these types.
(This is not a general-purpose facility; as a rule it only supports
specific lists of types as shown in the legacy converter table.)

-annotation

The annotation value for this parameter. Not currently supported,
because PEP 8 mandates that the Python library may not use
annotations.

zeroes
Only supported for strings. If true, embedded NUL bytes ('\\0') are
permitted inside the value.

+ +Please note, not every possible combination of arguments will work. +Often these arguments are implemented internally by specific PyArg_ParseTuple +format units, with specific behavior. For example, currently you cannot +call str and pass in zeroes=True without also specifying an encoding; +although it's perfectly reasonable to think this would work, these semantics don't +map to any existing format unit. So Argument Clinic doesn't support it. (Or, at +least, not yet.) Below is a table showing the mapping of legacy converters into real Argument Clinic converters. On the left is the legacy converter, @@ -720,9 +755,9 @@ on the right is the text you'd replace i 'u' Py_UNICODE 'U' unicode 'w*' Py_buffer(types='bytearray rwbuffer') -'y#' str(type='bytes', length=True) +'y#' str(types='bytes', length=True) 'Y' PyByteArrayObject -'y' str(type='bytes') +'y' str(types='bytes') 'y*' Py_buffer 'Z#' Py_UNICODE(nullable=True, length=True) 'z#' str(nullable=True, length=True) @@ -789,6 +824,90 @@ This restriction doesn't seem unreasonab hard-coded encoding strings for parameters whose format units start with e. +.. _default_values: + +Parameter default values +------------------------ + +Default values for parameters can be any of a number of values. +At their simplest, they can be string, int, or float literals:: +

foo: str = "abc"
bar: int = 123
bat: float = 45.6

+ +They can also use any of Python's built-in constants:: +

yep: bool = True
nope: bool = False
nada: object = None

+ +There's also special support for a default value of NULL, and +for simple expressions, documented in the following sections. + + +The NULL default value +-------------------------- + +For string and object parameters, you can set them to None to indicate +that there's no default. However, that means the C variable will be +initialized to Py_None. For convenience's sakes, there's a special +value called NULL for just this reason: from Python's perspective it +behaves like a default value of None, but the C variable is initialized +with NULL. + +Expressions specified as default values +--------------------------------------- + +The default value for a parameter can be more than just a literal value. +It can be an entire expression, using math operators and looking up attributes +on objects. However, this support isn't exactly simple, because of some +non-obvious semantics. + +Consider the following example:: +

foo: Py_ssize_t = sys.maxsize - 1

+ +sys.maxsize can have different values on different platforms. Therefore +Argument Clinic can't simply evaluate that expression locally and hard-code it +in C. So it stores the default in such a way that it will get evaluated at +runtime, when the user asks for the function's signature. + +What namespace is available when the expression is evaluated? It's evaluated +in the context of the module the builtin came from. So, if your module has an +attribute called "max_widgets", you may simply use it:: +

foo: Py_ssize_t = max_widgets

+ +If the symbol isn't found in the current module, it fails over to looking in +sys.modules. That's how it can find sys.maxsize for example. (Since you +don't know in advance what modules the user will load into their interpreter, +it's best to restrict yourself to modules that are preloaded by Python itself.) + +Evaluating default values only at runtime means Argument Clinic can't compute +the correct equivalent C default value. So you need to tell it explicitly. +When you use an expression, you must also specify the equivalent expression +in C, using the c_default parameter to the converter:: +

foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1

+ +Another complication: Argument Clinic can't know in advance whether or not the +expression you supply is valid. It parses it to make sure it looks legal, but +it can't actually know. You must be very careful when using expressions to +specify values that are guaranteed to be valid at runtime + +Finally, because expressions must be representable as static C values, there +are many restrictions on legal expressions. Here's a list of Python features +you're not permitted to use: + +* Function calls. +* Inline if statements (3 if foo else 5). +* Automatic sequence unpacking (*[1, 2, 3]). +* List/set/dict comprehensions and generator expressions. +* Tuple/list/set/dict literals. + + + Using a return converter ------------------------ @@ -1096,7 +1215,73 @@ any arguments. You can still use a self converter, a return converter, and specify a type argument to the object converter for METH_O. -Using Argument Clinic in Python files +The #ifdef trick +---------------------------------------------- + +If you're converting a function that isn't available on all platforms, +there's a trick you can use to make life a little easier. The existing +code probably looks like this:: +

#ifdef HAVE_FUNCTIONNAME
static module_functionname(...)
{
...
}
#endif /* HAVE_FUNCTIONNAME */

+ +And then in the PyMethodDef structure at the bottom you'll have:: +

#ifdef HAVE_FUNCTIONNAME
{'functionname', ... },
#endif /* HAVE_FUNCTIONNAME */

+ +In this scenario, you should change the code to look like the following:: +

#ifdef HAVE_FUNCTIONNAME
/*[clinic input]
module.functionname
...
[clinic start generated code]*/
static module_functionname(...)
{
...
}
#endif /* HAVE_FUNCTIONNAME */

+ +Run Argument Clinic on the code in this state, then refresh the file in +your editor. Now you'll have the generated code, including the #define +for the PyMethodDef, like so:: +

#ifdef HAVE_FUNCTIONNAME
/*[clinic input]
...
[clinic start generated code]*/
...
#define MODULE_FUNCTIONNAME [](#l2.290)
```
   {'functionname', ... },[](#l2.291)
```
...
/[clinic end generated code: checksum=...]/
static module_functionname(...)
{
...
}
#endif /* HAVE_FUNCTIONNAME */

+ +Change the #endif at the bottom as follows:: +

#else
#define MODULE_FUNCTIONNAME
#endif /* HAVE_FUNCTIONNAME */

+ +Now you can remove the #ifdefs around the PyMethodDef structure +at the end, and replace those three lines with MODULE_FUNCTIONNAME. +If the function is available, the macro turns into the PyMethodDef +static value, including the trailing comma; if the function isn't +available, the macro turns into nothing. Perfect + +(This is the preferred approach for optional functions; in the future, +Argument Clinic may generate the entire PyMethodDef structure.) + ------------------------------------- It's actually possible to use Argument Clinic to preprocess Python files.

--- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1954,6 +1954,8 @@ class Signature: if not s: return None

   Parameter = cls._parameter_cls[](#l3.7)

+ if s.endswith("/)"): kind = Parameter.POSITIONAL_ONLY s = s[:-2] + ')' @@ -1969,55 +1971,74 @@ class Signature: if not isinstance(module, ast.Module): return None

   # ast.FunctionDef[](#l3.16)
   f = module.body[0][](#l3.17)

parameters = [] empty = Parameter.empty invalid = object()

```
   def parse_attribute(node):[](#l3.23)
```

       if not isinstance(node.ctx, ast.Load):[](#l3.24)

```
           return None[](#l3.25)
```

```
   module = None[](#l3.26)
```
```
   module_dict = {}[](#l3.27)
```

   module_name = getattr(func, '__module__', None)[](#l3.28)

```
   if module_name:[](#l3.29)
```

       module = sys.modules.get(module_name, None)[](#l3.30)

```
       if module:[](#l3.31)
```

           module_dict = module.__dict__[](#l3.32)

   sys_module_dict = sys.modules[](#l3.33)

```
       value = node.value[](#l3.35)
```
```
       o = parse_node(value)[](#l3.36)
```
```
       if o is invalid:[](#l3.37)
```
```
           return invalid[](#l3.38)
```

```
   def parse_name(node):[](#l3.39)
```

       assert isinstance(node, ast.arg)[](#l3.40)

       if node.annotation != None:[](#l3.41)

           raise ValueError("Annotations are not currently supported")[](#l3.42)

```
       return node.arg[](#l3.43)
```

       if isinstance(value, ast.Name):[](#l3.45)

```
           name = o[](#l3.46)
```

           if name not in sys.modules:[](#l3.47)

```
               return invalid[](#l3.48)
```

           o = sys.modules[name][](#l3.49)

       return getattr(o, node.attr, invalid)[](#l3.51)

```
   def wrap_value(s):[](#l3.52)
```
```
       try:[](#l3.53)
```

           value = eval(s, module_dict)[](#l3.54)

```
       except NameError:[](#l3.55)
```
```
           try:[](#l3.56)
```

               value = eval(s, sys_module_dict)[](#l3.57)

```
           except NameError:[](#l3.58)
```

               raise RuntimeError()[](#l3.59)

```
   def parse_node(node):[](#l3.61)
```

       if isinstance(node, ast.arg):[](#l3.62)

           if node.annotation != None:[](#l3.63)

               raise ValueError("Annotations are not currently supported")[](#l3.64)

```
           return node.arg[](#l3.65)
```

       if isinstance(node, ast.Num):[](#l3.66)

```
           return node.n[](#l3.67)
```

       if isinstance(node, ast.Str):[](#l3.68)

```
           return node.s[](#l3.69)
```

       if isinstance(node, ast.NameConstant):[](#l3.70)

```
           return node.value[](#l3.71)
```

       if isinstance(node, ast.Attribute):[](#l3.72)

           return parse_attribute(node)[](#l3.73)

       if isinstance(node, ast.Name):[](#l3.74)

       if isinstance(value, str):[](#l3.75)

           return ast.Str(value)[](#l3.76)

       if isinstance(value, (int, float)):[](#l3.77)

           return ast.Num(value)[](#l3.78)

       if isinstance(value, bytes):[](#l3.79)

           return ast.Bytes(value)[](#l3.80)

       if value in (True, False, None):[](#l3.81)

           return ast.NameConstant(value)[](#l3.82)

```
       raise RuntimeError()[](#l3.83)
```

   class RewriteSymbolics(ast.NodeTransformer):[](#l3.85)

       def visit_Attribute(self, node):[](#l3.86)

```
           a = [][](#l3.87)
```
```
           n = node[](#l3.88)
```

           while isinstance(n, ast.Attribute):[](#l3.89)

               a.append(n.attr)[](#l3.90)

```
               n = n.value[](#l3.91)
```

           if not isinstance(n, ast.Name):[](#l3.92)

               raise RuntimeError()[](#l3.93)

```
           a.append(n.id)[](#l3.94)
```

           value = ".".join(reversed(a))[](#l3.95)

           return wrap_value(value)[](#l3.96)

       def visit_Name(self, node):[](#l3.98)
           if not isinstance(node.ctx, ast.Load):[](#l3.99)

               return invalid[](#l3.100)

```
           return node.id[](#l3.101)
```
```
       return invalid[](#l3.102)
```

               raise ValueError()[](#l3.103)

           return wrap_value(node.id)[](#l3.104)

def p(name_node, default_node, default=empty):

       name = parse_node(name_node)[](#l3.107)

       name = parse_name(name_node)[](#l3.108)
       if name is invalid:[](#l3.109)
           return None[](#l3.110)
       if default_node:[](#l3.111)

           o = parse_node(default_node)[](#l3.112)

```
           try:[](#l3.113)
```

               default_node = RewriteSymbolics().visit(default_node)[](#l3.114)

               o = ast.literal_eval(default_node)[](#l3.115)

           except ValueError:[](#l3.116)

               o = invalid[](#l3.117)
           if o is invalid:[](#l3.118)
               return None[](#l3.119)
           default = o if o is not invalid else default[](#l3.120)

--- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -1601,12 +1601,15 @@ class TestSignatureObject(unittest.TestC self.assertTrue(isinstance(signature, inspect.Signature)) def p(name): return signature.parameters[name].default self.assertEqual(p('s'), 'avocado')

   self.assertEqual(p('b'), b'bytes')[](#l4.7)
   self.assertEqual(p('d'), 3.14)[](#l4.8)
   self.assertEqual(p('i'), 35)[](#l4.9)

   self.assertEqual(p('c'), sys.maxsize)[](#l4.10)
   self.assertEqual(p('n'), None)[](#l4.11)
   self.assertEqual(p('t'), True)[](#l4.12)
   self.assertEqual(p('f'), False)[](#l4.13)

   self.assertEqual(p('local'), 3)[](#l4.14)

   self.assertEqual(p('sys'), sys.maxsize)[](#l4.15)

   self.assertEqual(p('exp'), sys.maxsize - 1)[](#l4.16)

def test_signature_on_non_function(self): with self.assertRaisesRegex(TypeError, 'is not a callable object'):

--- a/Misc/NEWS +++ b/Misc/NEWS @@ -88,6 +88,9 @@ Tests Tools/Demos ----------- +- Issue #20226: Argument Clinic now permits simple expressions

(e.g. "sys.maxsize - 1") as default values for parameters. +

Issue #19936: Added executable bits or shebang lines to Python scripts which requires them. Disable executable bits and shebang lines in test and benchmark files in order to prevent using a random system python, and in

--- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -618,7 +618,7 @@ curses_window_addch(PyObject *self, PyOb int group_right_1 = 0; long attr = 0;

switch (PyTuple_Size(args)) {

switch (PyTuple_GET_SIZE(args)) { case 1: if (!PyArg_ParseTuple(args, "O:addch", &ch)) return NULL;

@@ -650,7 +650,7 @@ curses_window_addch(PyObject self, PyOb static PyObject curses_window_addch_impl(PyObject self, int group_left_1, int x, int y, PyObject ch, int group_right_1, long attr) -/[clinic end generated code: checksum=44ed958b891cde91205e584c766e048f3999714f]/ +/[clinic end generated code: checksum=b073327add8197b6ba7fb96c87062422c8312954]/ { PyCursesWindowObject *cwself = (PyCursesWindowObject *)self; int coordinates_group = group_left_1;

--- a/Modules/_dbmmodule.c +++ b/Modules/_dbmmodule.c @@ -297,7 +297,7 @@ dbm_dbm_get(PyObject *self, PyObject *ar int group_right_1 = 0; PyObject *default_value = NULL;

switch (PyTuple_Size(args)) {

switch (PyTuple_GET_SIZE(args)) { case 1: if (!PyArg_ParseTuple(args, "s#:get", &key, &key_length)) return NULL;

@@ -318,7 +318,7 @@ dbm_dbm_get(PyObject *self, PyObject *ar static PyObject dbm_dbm_get_impl(dbmobject dp, const char key, Py_ssize_clean_t key_length, int group_right_1, PyObject default_value) -/[clinic end generated code: checksum=28cf8928811bde51e535d67ae98ea039d79df717]/ +/[clinic end generated code: checksum=2c3209571267017f1b9abbd19e1b521849fd5d4a]/ { datum dbm_key, val; @@ -450,7 +450,7 @@ dbm.open as dbmopen flags: str="r" How to open the file. "r" for reading, "w" for writing, etc.

mode: int(doc_default="0o666") = 0o666

mode: int(py_default="0o666") = 0o666 If creating a new file, the mode bits for the new file (e.g. os.O_RDWR).

--- a/Modules/_opcode.c +++ b/Modules/_opcode.c @@ -39,7 +39,7 @@ static PyObject * int oparg = 0; int _return_value;

switch (PyTuple_Size(args)) {

switch (PyTuple_GET_SIZE(args)) { case 1: if (!PyArg_ParseTuple(args, "i:stack_effect", &opcode)) return NULL;

@@ -64,7 +64,7 @@ exit: static int _opcode_stack_effect_impl(PyModuleDef module, int opcode, int group_right_1, int oparg) -/[clinic end generated code: checksum=e880e62dc7b0de73403026eaf4f8074aa106358b]/ +/[clinic end generated code: checksum=47e76ec27523da4ab192713642d32482cd743aa4]*/ { int effect; if (HAS_ARG(opcode)) {

--- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2870,7 +2870,7 @@ PyDoc_STRVAR(docstring_with_signature_an ); PyDoc_STRVAR(docstring_with_signature_with_defaults, -"docstring_with_signature_with_defaults(s='avocado', d=3.14, i=35, c=sys.maxsize, n=None, t=True, f=False)\n" +"docstring_with_signature_with_defaults(s='avocado', b=b'bytes', d=3.14, i=35, n=None, t=True, f=False, local=the_number_three, sys=sys.maxsize, exp=sys.maxsize - 1)\n" "\n" "\n" "\n" @@ -3317,6 +3317,8 @@ PyInit__testcapi(void) Py_INCREF(&PyInstanceMethod_Type); PyModule_AddObject(m, "instancemethod", (PyObject *)&PyInstanceMethod_Type);

PyModule_AddIntConstant(m, "the_number_three", 3);

+ TestError = PyErr_NewException("_testcapi.error", NULL, NULL); Py_INCREF(TestError); PyModule_AddObject(m, "error", TestError);

--- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2401,7 +2401,7 @@ class dir_fd_converter(CConverter): /*[clinic input] -os.stat -> object(doc_default='stat_result') +os.stat path : path_t(allow_fd=True) Path to be examined; can be string, bytes, or open-file-descriptor int. @@ -2523,7 +2523,7 @@ posix_lstat(PyObject *self, PyObject ar #define OS_ACCESS_DIR_FD_CONVERTER dir_fd_unavailable #endif /[clinic input] -os.access -> object(doc_default='True if granted, False otherwise') +os.access path: path_t(allow_fd=True) Path to be tested; can be string, bytes, or open-file-descriptor int.

--- a/Modules/zlibmodule.c +++ b/Modules/zlibmodule.c @@ -202,7 +202,7 @@ zlib_compress(PyModuleDef *module, PyObj int group_right_1 = 0; int level = 0;

switch (PyTuple_Size(args)) {

switch (PyTuple_GET_SIZE(args)) { case 1: if (!PyArg_ParseTuple(args, "y*:compress", &bytes)) return NULL;

@@ -227,7 +227,7 @@ zlib_compress(PyModuleDef module, PyObj static PyObject zlib_compress_impl(PyModuleDef module, Py_buffer bytes, int group_right_1, int level) -/[clinic end generated code: checksum=2c59af563a4595c5ecea4011701f482ae350aa5f]/ +/[clinic end generated code: checksum=66c4d16d0b8b9dd423648d9ef00d6a89d3363665]/ { PyObject *ReturnVal = NULL; Byte *input, *output = NULL;

--- a/Tools/clinic/clinic.py +++ b/Tools/clinic/clinic.py @@ -55,6 +55,13 @@ class Null: NULL = Null() +class Unknown:

def repr(self):
```
   return '<Unknown>'[](#l12.9)
```

+ +unknown = Unknown() + + def _text_accumulator(): text = [] def output(): @@ -197,7 +204,7 @@ def version_splitter(s): accumulator = [] def flush(): if not accumulator:

       raise ValueError('Malformed version string: ' + repr(s))[](#l12.21)

       raise ValueError('Unsupported version string: ' + repr(s))[](#l12.22)
   version.append(int(''.join(accumulator)))[](#l12.23)
   accumulator.clear()[](#l12.24)

@@ -596,7 +603,7 @@ static {impl_return_type} count_min = sys.maxsize count_max = -1

   add("switch (PyTuple_Size(args)) {{\n")[](#l12.30)

   add("switch (PyTuple_GET_SIZE(args)) {{\n")[](#l12.31)
   for subset in permute_optional_groups(left, required, right):[](#l12.32)
       count = len(subset)[](#l12.33)
       count_min = min(count_min, count)[](#l12.34)

@@ -1069,6 +1076,7 @@ class Clinic: self.filename = filename self.modules = collections.OrderedDict() self.classes = collections.OrderedDict()

```
   self.functions = [][](#l12.39)
```

global clinic clinic = self @@ -1343,29 +1351,7 @@ class Parameter: def is_keyword_only(self): return self.kind == inspect.Parameter.KEYWORD_ONLY -py_special_values = {

NULL: "None",

-} - -def py_repr(o):

special = py_special_values.get(o)
if special:
```
   return special[](#l12.54)
```
return repr(o)

- - -c_special_values = {

NULL: "NULL",
None: "Py_None",

-} - -def c_repr(o):

special = c_special_values.get(o)
if special:
```
   return special[](#l12.66)
```
if isinstance(o, str):

   return '"' + quoted_for_c_string(o) + '"'[](#l12.68)

return repr(o)

+ def add_c_converter(f, name=None): if not name: @@ -1407,8 +1393,7 @@ class CConverter(metaclass=CConverterAut """ For the init function, self, name, function, and default must be keyword-or-positional parameters. All other

parameters (including "required" and "doc_default")
must be keyword-only.

parameters must be keyword-only. """
The C type to use for this variable.

@@ -1418,23 +1403,23 @@ class CConverter(metaclass=CConverterAut # The Python default value for this parameter, as a Python value. # Or the magic value "unspecified" if there is no default.

Or the magic value "unknown" if this value is a cannot be evaluated
at Argument-Clinic-preprocessing time (but is presumed to be valid
at runtime).
default = unspecified
If not None, default must be isinstance() of this type.
(You can also specify a tuple of types.)
default_type = None

"default" as it should appear in the documentation, as a string.
Or None if there is no default.
doc_default = None

"default" converted into a str for rendering into Python code.
py_default = None

- # "default" converted into a C value, as a string. # Or None if there is no default. c_default = None

"default" converted into a Python value, as a string.
Or None if there is no default.
py_default = None

+ # The default value used to initialize the C variable when # there is no default, but not specifying a default may # result in an "uninitialized variable" warning. This can @@ -1485,12 +1470,12 @@ class CConverter(metaclass=CConverterAut # Only used by format units ending with '#'. length = False

def init(self, name, function, default=unspecified, *, doc_default=None, c_default=None, py_default=None, required=False, annotation=unspecified, **kwargs):

def init(self, name, function, default=unspecified, *, c_default=None, py_default=None, annotation=unspecified, **kwargs): self.function = function self.name = name

if default is not unspecified:

       if self.default_type and not isinstance(default, self.default_type):[](#l12.125)

       if self.default_type and not isinstance(default, (self.default_type, Unknown)):[](#l12.126)
           if isinstance(self.default_type, type):[](#l12.127)
               types_str = self.default_type.__name__[](#l12.128)
           else:[](#l12.129)

@@ -1498,23 +1483,19 @@ class CConverter(metaclass=CConverterAut fail("{}: default value {!r} for field {} is not of type {}".format( self.class.name, default, name, types_str)) self.default = default

       self.py_default = py_default if py_default is not None else py_repr(default)[](#l12.134)

       self.doc_default = doc_default if doc_default is not None else self.py_default[](#l12.135)

       self.c_default = c_default if c_default is not None else c_repr(default)[](#l12.136)

```
   else:[](#l12.137)
```

       self.py_default = py_default[](#l12.138)

       self.doc_default = doc_default[](#l12.139)

       self.c_default = c_default[](#l12.140)

   self.c_default = c_default[](#l12.142)

   self.py_default = py_default[](#l12.143)

+ if annotation != unspecified: fail("The 'annotation' parameter is not currently permitted.")

   self.required = required[](#l12.147)
   self.converter_init(**kwargs)[](#l12.148)

def converter_init(self): pass def is_optional(self):

   return (self.default is not unspecified) and (not self.required)[](#l12.154)

   return (self.default is not unspecified)[](#l12.155)

def render(self, parameter, data): """ @@ -1655,8 +1636,9 @@ class bool_converter(CConverter): c_ignored_default = '0' def converter_init(self):

   self.default = bool(self.default)[](#l12.163)

   self.c_default = str(int(self.default))[](#l12.164)

   if self.default is not unspecified:[](#l12.165)

       self.default = bool(self.default)[](#l12.166)

       self.c_default = str(int(self.default))[](#l12.167)

class char_converter(CConverter): type = 'char' @@ -1665,7 +1647,7 @@ class char_converter(CConverter): c_ignored_default = "'\0'" def converter_init(self):

   if len(self.default) != 1:[](#l12.175)

   if isinstance(self.default, str) and (len(self.default) != 1):[](#l12.176)
       fail("char_converter: illegal default value " + repr(self.default))[](#l12.177)

@@ -1797,8 +1779,8 @@ class object_converter(CConverter): @add_legacy_c_converter('s#', length=True) -@add_legacy_c_converter('y', type="bytes") -@add_legacy_c_converter('y#', type="bytes", length=True) +@add_legacy_c_converter('y', types="bytes") +@add_legacy_c_converter('y#', types="bytes", length=True) @add_legacy_c_converter('z', nullable=True) @add_legacy_c_converter('z#', nullable=True, length=True) class str_converter(CConverter): @@ -1993,8 +1975,8 @@ class CReturnConverter(metaclass=CReturn # Or the magic value "unspecified" if there is no default. default = None

def init(self, *, doc_default=None, **kwargs):

   self.doc_default = doc_default[](#l12.196)

def init(self, *, py_default=None, **kwargs):

   self.py_default = py_default[](#l12.198)
   try:[](#l12.199)
       self.return_converter_init(**kwargs)[](#l12.200)
   except TypeError as e:[](#l12.201)

@@ -2212,6 +2194,7 @@ class DSLParser: self.indent = IndentStack() self.kind = CALLABLE self.coexist = False

   self.parameter_continuation = ''[](#l12.206)

def directive_version(self, required): global version @@ -2244,15 +2227,18 @@ class DSLParser: self.block.signatures.append(c) def at_classmethod(self):

   assert self.kind is CALLABLE[](#l12.214)

   if self.kind is not CALLABLE:[](#l12.215)

       fail("Can't set @classmethod, function is not a normal callable")[](#l12.216)
   self.kind = CLASS_METHOD[](#l12.217)

def at_staticmethod(self):

   assert self.kind is CALLABLE[](#l12.220)

   if self.kind is not CALLABLE:[](#l12.221)

       fail("Can't set @staticmethod, function is not a normal callable")[](#l12.222)
   self.kind = STATIC_METHOD[](#l12.223)

def at_coexist(self):

   assert self.coexist == False[](#l12.226)

```
   if self.coexist:[](#l12.227)
```

       fail("Called @coexist twice!")[](#l12.228)
   self.coexist = True[](#l12.229)

@@ -2503,6 +2489,7 @@ class DSLParser: if not self.indent.infer(line): return self.next(self.state_function_docstring, line)

   self.parameter_continuation = ''[](#l12.236)
   return self.next(self.state_parameter, line)[](#l12.237)

@@ -2516,6 +2503,10 @@ class DSLParser: p.group = -p.group def state_parameter(self, line):

   if self.parameter_continuation:[](#l12.244)

       line = self.parameter_continuation + ' ' + line.lstrip()[](#l12.245)

       self.parameter_continuation = ''[](#l12.246)

+ if self.ignore_line(line): return @@ -2529,6 +2520,11 @@ class DSLParser: # we indented, must be to new parameter docstring column return self.next(self.state_parameter_docstring_start, line)

```
   line = line.rstrip()[](#l12.255)
```
```
   if line.endswith('\\'):[](#l12.256)
```

       self.parameter_continuation = line[:-1][](#l12.257)

```
       return[](#l12.258)
```

+ line = line.lstrip() if line in ('*', '/', '[', ']'): @@ -2547,48 +2543,123 @@ class DSLParser: else: fail("Function " + self.function.name + " has an unsupported group configuration. (Unexpected state " + str(self.parameter_state) + ")")

   ast_input = "def x({}): pass".format(line)[](#l12.267)

   base, equals, default = line.rpartition('=')[](#l12.268)

```
   if not equals:[](#l12.269)
```
```
       base = default[](#l12.270)
```

       default = None[](#l12.271)
   module = None[](#l12.272)
   try:[](#l12.273)

       ast_input = "def x({}): pass".format(base)[](#l12.274)
       module = ast.parse(ast_input)[](#l12.275)
   except SyntaxError:[](#l12.276)

```
       pass[](#l12.277)
```

```
       try:[](#l12.278)
```
```
           default = None[](#l12.279)
```

           ast_input = "def x({}): pass".format(line)[](#l12.280)

           module = ast.parse(ast_input)[](#l12.281)

```
       except SyntaxError:[](#l12.282)
```

           pass[](#l12.283)
   if not module:[](#l12.284)
       fail("Function " + self.function.name + " has an invalid parameter declaration:\n\t" + line)[](#l12.285)

function_args = module.body[0].args parameter = function_args.args[0]

```
   py_default = None[](#l12.290)
```

- parameter_name = parameter.arg name, legacy, kwargs = self.parse_converter(parameter.annotation)

   if function_args.defaults:[](#l12.295)

       expr = function_args.defaults[0][](#l12.296)

       # mild hack: explicitly support NULL as a default value[](#l12.297)

       if isinstance(expr, ast.Name) and expr.id == 'NULL':[](#l12.298)

```
           value = NULL[](#l12.299)
```

       elif isinstance(expr, ast.Attribute):[](#l12.300)

```
   if not default:[](#l12.301)
```
```
       value = unspecified[](#l12.302)
```

       if 'py_default' in kwargs:[](#l12.303)

           fail("You can't specify py_default without specifying a default value!")[](#l12.304)

```
   else:[](#l12.305)
```

       default = default.strip()[](#l12.306)

       ast_input = "x = {}".format(default)[](#l12.307)

```
       try:[](#l12.308)
```

           module = ast.parse(ast_input)[](#l12.309)

           # blacklist of disallowed ast nodes[](#l12.311)

           class DetectBadNodes(ast.NodeVisitor):[](#l12.312)

```
               bad = False[](#l12.313)
```

               def bad_node(self, node):[](#l12.314)

                   self.bad = True[](#l12.315)

               # inline function call[](#l12.317)

               visit_Call = bad_node[](#l12.318)

               # inline if statement ("x = 3 if y else z")[](#l12.319)

               visit_IfExp = bad_node[](#l12.320)

               # comprehensions and generator expressions[](#l12.322)

               visit_ListComp = visit_SetComp = bad_node[](#l12.323)

               visit_DictComp = visit_GeneratorExp = bad_node[](#l12.324)

               # literals for advanced types[](#l12.326)

               visit_Dict = visit_Set = bad_node[](#l12.327)

               visit_List = visit_Tuple = bad_node[](#l12.328)

               # "starred": "a = [1, 2, 3]; *a"[](#l12.330)

               visit_Starred = bad_node[](#l12.331)

               # allow ellipsis, for now[](#l12.333)

               # visit_Ellipsis = bad_node[](#l12.334)

           blacklist = DetectBadNodes()[](#l12.336)

           blacklist.visit(module)[](#l12.337)

           if blacklist.bad:[](#l12.338)

               fail("Unsupported expression as default value: " + repr(default))[](#l12.339)

           expr = module.body[0].value[](#l12.341)

           # mild hack: explicitly support NULL as a default value[](#l12.342)

           if isinstance(expr, ast.Name) and expr.id == 'NULL':[](#l12.343)

```
               value = NULL[](#l12.344)
```

               py_default = 'None'[](#l12.345)

               c_default = "NULL"[](#l12.346)

           elif (isinstance(expr, ast.BinOp) or[](#l12.347)

               (isinstance(expr, ast.UnaryOp) and not isinstance(expr.operand, ast.Num))):[](#l12.348)

               c_default = kwargs.get("c_default")[](#l12.349)

               if not (isinstance(c_default, str) and c_default):[](#l12.350)

                   fail("When you specify an expression (" + repr(default) + ") as your default value,\nyou MUST specify a valid c_default.")[](#l12.351)

               py_default = default[](#l12.352)

               value = unknown[](#l12.353)

           elif isinstance(expr, ast.Attribute):[](#l12.354)

```
               a = [][](#l12.355)
```
```
               n = expr[](#l12.356)
```

               while isinstance(n, ast.Attribute):[](#l12.357)

                   a.append(n.attr)[](#l12.358)

                   n = n.value[](#l12.359)

               if not isinstance(n, ast.Name):[](#l12.360)

                   fail("Unsupported default value " + repr(default) + " (looked like a Python constant)")[](#l12.361)

               a.append(n.id)[](#l12.362)

               py_default = ".".join(reversed(a))[](#l12.363)

               c_default = kwargs.get("c_default")[](#l12.365)

               if not (isinstance(c_default, str) and c_default):[](#l12.366)

                   fail("When you specify a named constant (" + repr(py_default) + ") as your default value,\nyou MUST specify a valid c_default.")[](#l12.367)

```
               try:[](#l12.369)
```

                   value = eval(py_default)[](#l12.370)

               except NameError:[](#l12.371)

                   value = unknown[](#l12.372)

```
           else:[](#l12.373)
```

               value = ast.literal_eval(expr)[](#l12.374)

               py_default = repr(value)[](#l12.375)

               if isinstance(value, (bool, None.__class__)):[](#l12.376)

                   c_default = "Py_" + py_default[](#l12.377)

               elif isinstance(value, str):[](#l12.378)

                   c_default = '"' + quoted_for_c_string(value) + '"'[](#l12.379)

```
               else:[](#l12.380)
```

                   c_default = py_default[](#l12.381)

       except SyntaxError as e:[](#l12.383)

           fail("Syntax error: " + repr(e.text))[](#l12.384)

       except (ValueError, AttributeError):[](#l12.385)

           value = unknown[](#l12.386)
           c_default = kwargs.get("c_default")[](#l12.387)

           py_default = default[](#l12.388)
           if not (isinstance(c_default, str) and c_default):[](#l12.389)
               fail("When you specify a named constant (" + repr(py_default) + ") as your default value,\nyou MUST specify a valid c_default.")[](#l12.390)

```
           a = [][](#l12.392)
```
```
           n = expr[](#l12.393)
```

           while isinstance(n, ast.Attribute):[](#l12.394)

               a.append(n.attr)[](#l12.395)

```
               n = n.value[](#l12.396)
```

           if not isinstance(n, ast.Name):[](#l12.397)

               fail("Malformed default value (looked like a Python constant)")[](#l12.398)

```
           a.append(n.id)[](#l12.399)
```

           py_default = ".".join(reversed(a))[](#l12.400)

           kwargs["py_default"] = py_default[](#l12.401)

           value = eval(py_default)[](#l12.402)

```
       else:[](#l12.403)
```

           value = ast.literal_eval(expr)[](#l12.404)

```
   else:[](#l12.405)
```
```
       value = unspecified[](#l12.406)
```

       kwargs.setdefault('c_default', c_default)[](#l12.407)

       kwargs.setdefault('py_default', py_default)[](#l12.408)

dict = legacy_converters if legacy else converters legacy_str = "legacy " if legacy else "" @@ -2777,7 +2848,7 @@ class DSLParser: if p.converter.is_optional(): a.append('=') value = p.converter.default

           a.append(p.converter.doc_default)[](#l12.416)

           a.append(p.converter.py_default)[](#l12.417)
       s = fix_right_bracket_count(p.right_bracket_count)[](#l12.418)
       s += "".join(a)[](#l12.419)
       if add_comma:[](#l12.420)

@@ -2788,9 +2859,18 @@ class DSLParser: add(fix_right_bracket_count(0)) add(')')

   # if f.return_converter.doc_default:[](#l12.425)

```
   # PEP 8 says:[](#l12.426)
```
```
   #[](#l12.427)
```

   #     The Python standard library will not use function annotations[](#l12.428)

   #     as that would result in a premature commitment to a particular[](#l12.429)

   #     annotation style. Instead, the annotations are left for users[](#l12.430)

   #     to discover and experiment with useful annotation styles.[](#l12.431)

```
   #[](#l12.432)
```

   # therefore this is commented out:[](#l12.433)

```
   #[](#l12.434)
```

   # if f.return_converter.py_default:[](#l12.435)
   #     add(' -> ')[](#l12.436)

   #     add(f.return_converter.doc_default)[](#l12.437)

   #     add(f.return_converter.py_default)[](#l12.438)

docstring_first_line = output() @@ -2998,8 +3078,8 @@ def main(argv): # print(" ", short_name + "".join(parameters)) print()

   print("All converters also accept (doc_default=None, required=False, annotation=None).")[](#l12.446)

   print("All return converters also accept (doc_default=None).")[](#l12.447)

   print("All converters also accept (c_default=None, py_default=None, annotation=None).")[](#l12.448)

   print("All return converters also accept (py_default=None).")[](#l12.449)
   sys.exit(0)[](#l12.450)

if ns.make:

--- a/Tools/clinic/clinic_test.py +++ b/Tools/clinic/clinic_test.py @@ -231,20 +231,20 @@ xyz self._test_clinic(""" verbatim text here lah dee dah -/[copy] +/[copy input] def -[copy]/ +[copy start generated code]/ abc -/[copy checksum: 03cfd743661f07975fa2f1220c5194cbaff48451]/ +/[copy end generated code: checksum=03cfd743661f07975fa2f1220c5194cbaff48451]/ xyz """, """ verbatim text here lah dee dah -/[copy] +/[copy input] def -[copy]/ +[copy start generated code]/ def -/[copy checksum: 7b18d017f89f61cf17d47f92749ea6930a3f1deb]/ +/[copy end generated code: checksum=7b18d017f89f61cf17d47f92749ea6930a3f1deb]/ xyz """) @@ -292,17 +292,6 @@ os.access p = function.parameters['path'] self.assertEqual(1, p.converter.args['allow_fd'])

def test_param_docstring(self):

   function = self.parse_function("""[](#l13.35)

-module os -os.stat as os_stat_fn -> object(doc_default='stat_result') -

path: str
```
  Path to be examined""")[](#l13.40)
```

   p = function.parameters['path'][](#l13.41)

   self.assertEqual("Path to be examined", p.docstring)[](#l13.42)

   self.assertEqual(function.return_converter.doc_default, 'stat_result')[](#l13.43)

- def test_function_docstring(self): function = self.parse_function(""" module os

cpython: abf87e1fbc62 (original) (raw)

Mercurial > cpython

changeset 88512:abf87e1fbc62

The C type to use for this variable.

Or the magic value "unknown" if this value is a cannot be evaluated

at Argument-Clinic-preprocessing time (but is presumed to be valid

at runtime).

If not None, default must be isinstance() of this type.

(You can also specify a tuple of types.)

"default" as it should appear in the documentation, as a string.

Or None if there is no default.

"default" converted into a str for rendering into Python code.

"default" converted into a Python value, as a string.

Or None if there is no default.