[Python-Dev] Experiences with Creating PEP 484 Stub Files (original) (raw)

Phil Thompson phil at riverbankcomputing.com
Tue Feb 9 17:06:25 EST 2016

On 9 Feb 2016, at 8:54 pm, Guido van Rossum <guido at python.org> wrote:

[Just adding to Andrew's response] On Tue, Feb 9, 2016 at 9:58 AM, Andrew Barnert via Python-Dev <python-dev at python.org> wrote: On Feb 9, 2016, at 03:44, Phil Thompson <phil at riverbankcomputing.com> wrote:

There are a number of things I'd like to express but cannot find a way to do so... - objects that implement the buffer protocol That seems like it should be filed as a bug with the typing repo. Presumably this is just an empty type that registers bytes, bytearray, and memoryview, and third-party classes have to register with it manually? Hm, there's no way to talk about these in regular Python code either, is there? I think that issue should be resolved first. Probably by adding something to collections.abc. And then we can add the corresponding name to typing.py. This will take time though (have to wait for 3.6) so I'd recommend 'Any' for now (and filing those bugs).

Ok.

- type objects You can use 'type' for this (i.e. the builtin). You can't specify any properties for types though; that's a feature request: https://github.com/python/typing/issues/107 -- but it may be a while before we address it (it's not entirely clear how it should work, and we have many other pressing issues still).

Yes, I can use type.

- slice objects Can't you just use the concrete types type and slice tor these two? I don't think you need generic or abstract "any metaclass, whether inheriting from type or not" or "any class that meets the slice protocol", do you? Can't you use 'slice' (i.e. the builtin)? Mypy supports that.

Yes, I can use slice.

- capsules

That one seems reasonable. But maybe there should just be a types.Capsule Type or types.PyCapsule, and then you can just check that the same as any other concrete type? But how often do you need to verify that something is a capsule, without knowing that it's the right capsule? At runtime, you'd usually use PyCapsuleIsValid, not PyCapsuleCheckExacf, right? So should the type checker be tracking the name too?

- sequences of fixed size (ie. specified in the same way as Tuple) That's kind of a poor data structure. :-( Why can't you use Tuple here?

Because allowing any sequence is more flexible that only allowing a tuple.

How would you disambiguate between a sequence of one int and a sequence of 0 or more ints if they're both spelled "Sequence[int]"? That isn't a problem for Tuple, because it's assumed to be heterogeneous, so Tuple[int] can only be a 1-tuple. (This was actually discussed in some depth. I thought it would be a problem, because some types--including tuple itself--are sometimes used as homogenous arbitrary-length containers and sometimes as heterogeneous fixed-length containers, but Guido and others had some good answers for that, even if I can't remember what they were.) We solved that by allowing Tuple[int, ...] to spell a homogeneous tuple of integers.

- distinguishing between instance and class attributes. Where? Are you building a protocol that checks the data members of a type for conformance or something? If so, why is an object that has "spam" and "eggs" as instance attributes but "cheese" as a class attribute not usable as an object conforming to the protocol with all three attributes? (Also, does @property count as a class or instance attribute? What about an arbitrary data descriptor? Or a non-data descriptor?) It's a known mypy bug. :-( It's somewhat convoluted to fix. https://github.com/JukkaL/mypy/issues/1097 Some things Andrew snipped: The documentation is incomplete - there is no mention of Set or Tuple for example. Tuple is here: https://docs.python.org/3/library/typing.html#typing.Tuple

Yes, I missed that.

collections.Set maps to typing.AbstractSet (https://docs.python.org/3/library/typing.html#typing.AbstractSet; present twice in the docs somehow :-( ). typing.Set (corresponding to builtins.set) is indeed missing, I've a note of that: http://bugs.python.org/issue26322.

I found the documentation confusing regarding Optional. Intuitively it seems to be the way to specify arguments with default values. However it is explained in terms of (for example) Union[str, None] and I (intuitively but incorrectly) read that as meaning "a str or None" as opposed to "a str or nothing". But it does mean 'str or None'. The type of an argument doesn't have any bearing on whether it may be omitted from the argument list by the caller -- these are orthogonal concepts (though sadly the word optional might apply to both). It's possible (though unusual) to have an optional argument that must be a str when given; it's also possible to have a mandatory argument that may be a str or None.

In the case of Python wrappers around a C++ library then every optional argument will have to have a specific type when given.

So you are saying that a mandatory argument that may be a str or None would be specified as Union[str, None]? But the docs say that that is the underlying implementation of Option[str] - which (to me) means an optional argument that should be a string when given.

Can you help improve the wording in the docs (preferably by filing an issue)?

When I eventually understand what it means...

bytes can be used as shorthand for bytes, bytearray and memoryview - but what about objects that really only support bytes? Shouldn't the shorthand be called something like AnyBytes? We debated that, but found it too annoying to have to import and write write AnyBytes in so many places. The type checker may not be precise for cases that only accept bytes, but hopefully it's more useful in general this way.

Is there any recommended way to test the validity and completeness of stub files? What's the recommended way to parse them? That's also an open issue. For a quick check I tend to just point mypy at a stub file, since it is the most mature implementation of PEP 484 to date (Google's pytype is still working on PEP 484 compatibility). While this doesn't always catch all errors, it will at least find syntax errors and cases that mypy doesn't support. :-)

Ok I'll try that.

Phil

More information about the Python-Dev mailing list