(original) (raw)

On 01/19/2014 08:30 PM, Nick Coghlan wrote:

Guido, Larry and I thrashed out the required semantics for parameter groups at PyCon US last year (and I believe the argument clinic PEP describes those accurately).

They're mainly needed to represent oddball signatures like range() and slice().

However, I'm inclined to say that the affected functions should simply not support introspection until Python 3.5.

It's not just a matter of the data model, there's also the matter of defining the string representation.

Au contraire!, says the man writing patches.

How it works right now: Argument Clinic publishes the signature information for builtins as the first line of the builtin's docstring. It gets clipped off before \_\_doc\_\_ returns it, and it's made available as \_\_text\_signature\_\_. inspect.Signature retrieves this string and passes it in to ast.parse(). It then walks the resulting tree, producing the Function and Parameter objects as it goes.

Argument Clinic has the information about positional-only parameters and optional groups, but it can't express it in this text signature. The text signature is parsed by ast.parse(), and ast.parse() only understands Python syntax. So the information is thrown away. Boo hoo.

I was trying to change this with PEP 457:

http://www.python.org/dev/peps/pep-0457/

PEP 457 proposed new Python syntax for positional-only parameters and optional groups. The syntax looks like Argument Clinic's syntax for these features, but with commas replacing newlines:

Positional-only parameters are delimited with a single slash. In "def foo(self, /, a)", self is positional-only and a is not.
Optional groups are parameters wrapped with square brackets, with the brackets never appearing between a parameter and a comma. In "def foo(a, \[b, c\])", b and c are in an optional group together. Groups can be nested with some restrictions.

To be clear: I wasn't proposing we actually add this syntax to Python! Just that, \*if\* we added support for positional-only parameters and optional groups, it would be this syntax. I did propose that documentation and tools like Argument Clinic could use it now.

PEP 457 didn't get any support. People said: "We don't need to define syntax if we're not going to use it. If you really need this you can do it privately."

Okay! I'll do it privately! Hopefully this week!

Step 1: Extend the format of the \_\_text\_signature\_\_ to use PEP 457 syntax. Then \*strip\* that extra syntax in inspect.Signature, producing a sanitized string that is parseable by ast.parse(), and storing the extra information on the side. Pass the sanitized string in to ast.parse(), walk the parse tree, and merge the positional-only / optional group information into the Parameter objects as I go. inspect.Signature objects can already represent positional-only parameters, and I plan to add a new member ("group", see below) that lets them represent optional groups too. Hooray! inspect.Signature objects can now represent almost every Python callable!

Step 2: make pydoc print the optional groups information, aka the PEP 457 square brackets, because that's uncontroversial. pydoc on builtins has printed things like "range(\[start\], stop, \[step\])" for decades. Everybody intuitively understands it, everybody likes it.

Step 3, if people want it: Also make pydoc display which parameters are positional-only. I was going to propose that in a couple of days, when I was getting near ready to implement it. But I guess we're discussing it now. Unsurprisingly, I propose to use the PEP 457 syntax here.

\--------------------

Regarding Step 3 above, here's something to consider. str(inspect.signature(foo)) produces a very nice-looking string, which currently is (almost!) always parseable by ast.parse() like so:

sig = inspect.signature(foo)
if sig:
ast.parse("def foo" + str(sig) + ": pass")

On the one hand, this mechanical round-trip ability is kind of a nice property.

On the other hand, it does restrict the format of the string--it can't have our square brackets, it can't have our "/,". Meanwhile \_\_str\_\_ makes no guarantee that the string it returns is a valid Python expression.

We could do both. Make inspect.Signature.\_\_str\_\_ only return Python compatible syntax, and allow another mechanism for pydoc to produce something more informative (but not Python compatible) for the user, like so:

>>> str(inspect.signature(\_pickle.Pickler.dump))
'(self, obj)'

\>>> pydoc.plain(pydoc.render\_doc(pickle.Pickler.dump))
"...\\n\_pickle.Pickler.dump(self, /, obj)\\n..."

Or we could do it the other way around.

So I guess I'm proposing four alternatives:
1) inspect.Signature.\_\_str\_\_() must always be parsable by ast.parse().
2) inspect.Signature.\_\_str\_\_() must always be parsable by ast.parse(). Add another method to inspect.Signature that can use PEP 457 syntax, use that from pydoc.
3) inspect.Signature.\_\_str\_\_() produces PEP 457 syntax. Add another method to inspect.Signature producing a text representation that is parseable by ast.parse().
4) inspect.Signature.\_\_str\_\_() produces PEP 457 syntax.

\--------------------

In case you thought we were done, there's one more wrinkle: str(inspect.signature(foo)) \*already\* marks positional-only parameters! I discovered that in the last day or two.

Late last week I noticed that "self" is \*always\* positional-only for builtins. It doesn't matter if the builtin is METH\_KEYWORDS. So, in my burgeoning "add support for more builtins" patch, in inspect.Signature I marked self parameters on builtins as positional-only. I was rewarded with this:

>>> str(inspect.signature(\_pickle.Pickler.dump))
'(, obj)'

Yes, inspect.Signature.\_\_str\_\_() uses angle brackets to denote positional-only parameters.

I think this behavior needs to be removed. It's undocumented and way non-obvious. I'm not aware of this syntax getting support from much of anybody--the only reason it's survived this long is because nobody besides me has ever seen it in the wild.

\--------------------

To address Yury's proposal:

So before committing to the parameters groups idea, I'd like to propose somewhat simpler, but powerful enough to solve our todays problems solution.

What if we add a notion of "optional" parameters?

Your proposal gets a "no, absolutely not" vote from me.

1. We already have a notion of "optional parameters". Parameters
with default values are optional.

2. Your proposed syntax doesn't mention how we'd actually establish
default values for parameters. So it's insufficient to handle
existing code.

3. Your syntax/semantics, as described, can't convey the concept of
optional groups. So it's insufficient to solve the problem it sets
out to solve.

4. Your proposed syntax changes 80% of existing code--any parameter
with a default value. I don't know how you concluded this was
"simpler".

Here's my counter-proposal.

1. We add a new class to inspect named "ParameterGroup". This class
will have only one public member, "parent". ParameterGroup.parent
will always be either None or a different inspect.ParameterGroup
instance.

2. We add a new member to inspect.Parameter named "group". "group"
will be either None or an instance of inspect.ParameterGroup.

3. We add a new method on Parameter named "is_optional()".
"is_optional()" returns True if the function can be called without
providing the parameter. Here's the implementation:

def is_optional(self):

return (self.default is not self._empty) or (self.group is not
None)

4. Textual representations intended for displaying to the user are
permitted to use square brackets to denote optional groups. They
might also be permitted to use "/" to delimit positional-only
parameters from other types of parameters, if the community accepts
this.

5. We don't change any Python language semantics and we don't break
any existing code.

Under my proposal:

bytearray([source, [encoding, [errors]]])

source.group != encoding.group

encoding.group != errors.group

source.group.parent == None

encoding.group.parent == source.group

errors.group.parent == encoding.group

source.is_optional() == encoding.is_optional() ==
errors.is_optional() == True

curses.window.addch([x, y,] ch, [attr])

x.group == y.group

x.group != attr.group

x.group.parent == attr.group.parent == None

x.is_optional() == y.is_optional() == attr.is_optional() == True

ch.is_optional() == False

Sorry about the length of this email,

/arry