[Python-Dev] PEP 561: Distributing and Packaging Type Information (original) (raw)

Ethan Smith ethan at ethanhs.me
Thu Oct 26 18:42:19 EDT 2017

Hello all,

I have completed an implementation for PEP 561, and believe it is time to share the PEP and implementation with python-dev

Python-ideas threads:

The live version is here: https://www.python.org/dev/peps/pep-0561/

As always, duplicated below.

Ethan Smith

PEP: 561 Title: Distributing and Packaging Type Information Author: Ethan Smith <ethan at ethanhs.me> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History:

Abstract

PEP 484 introduced type hinting to Python, with goals of making typing gradual and easy to adopt. Currently, typing information must be distributed manually. This PEP provides a standardized means to package and distribute type information and an ordering for type checkers to resolve modules and collect this information for type checking using existing packaging architecture.

Rationale

Currently, package authors wish to distribute code that has inline type information. However, there is no standard method to distribute packages with inline type annotations or syntax that can simultaneously be used at runtime and in type checking. Additionally, if one wished to ship typing information privately the only method would be via setting MYPYPATH or the equivalent to manually point to stubs. If the package can be released publicly, it can be added to typeshed [1]_. However, this does not scale and becomes a burden on the maintainers of typeshed. Additionally, it ties bugfixes to releases of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this section [2]_ the PEP recommends using shared/typehints/pythonX.Y/ for shipping stub files. However, manually adding a path to stub files for each third party library does not scale. The simplest approach people have taken is to add site-packages to their MYPYPATH, but this causes type checkers to fail on packages that are highly dynamic (e.g. sqlalchemy and Django).

Specification

There are several motivations and methods of supporting typing in a package. This PEP recognizes three (3) types of packages that may be created:

  1. The package maintainer would like to add type information inline.

  2. The package maintainer would like to add type information via stubs.

  3. A third party would like to share stub files for a package, but the maintainer does not want to include them in the source of the package.

This PEP aims to support these scenarios and make them simple to add to packaging and deployment.

The two major parts of this specification are the packaging specifications and the resolution order for resolving module type information. The packaging spec is based on and extends PEP 345 metadata. The type checking spec is meant to replace the shared/typehints/pythonX.Y/ spec of PEP 484 [2]_.

New third party stub libraries are encouraged to distribute stubs via the third party packaging proposed in this PEP in place of being added to typeshed. Typeshed will remain in use, but if maintainers are found, third party stubs in typeshed are encouraged to be split into their own package.

Packaging Type Information

In order to make packaging and distributing type information as simple and easy as possible, the distribution of type information, and typed Python code is done through existing packaging frameworks. This PEP adds a new item to the *.distinfo/METADATA file to contain metadata about a package's support for typing. The new item is optional, but must have a name of Typed and have a value of either inline or stubs, if present.

Metadata Examples::

Typed: inline
Typed: stubs

Stub Only Packages ''''''''''''''''''

For package maintainers wishing to ship stub files containing all of their type information, it is prefered that the *.pyi stubs are alongside the corresponding *.py files. However, the stubs may be put in a sub-folder of the Python sources, with the same name the *.py files are in. For example, the flyingcircus package would have its stubs in the folder flyingcircus/flyingcircus/. This path is chosen so that if stubs are not found in flyingcircus/ the type checker may treat the subdirectory as a normal package. The normal resolution order of checking *.pyi before *.py will be maintained.

Third Party Stub Packages '''''''''''''''''''''''''

Third parties seeking to distribute stub files are encouraged to contact the maintainer of the package about distribution alongside the package. If the maintainer does not wish to maintain or package stub files or type information inline, then a "third party stub package" should be created. The structure is similar, but slightly different from that of stub only packages. If the stubs are for the library flyingcircus then the package should be named flyingcircus-stubs and the stub files should be put in a sub-directory named flyingcircus. This allows the stubs to be checked as if they were in a regular package.

In addition, the third party stub package should indicate which version(s) of the runtime package are supported by indicating the runtime package's version(s) through the normal dependency data. For example, if there was a stub package flyingcircus-stubs, it can indicate the versions of the runtime flyingcircus package supported through install_requires in distutils based tools, or the equivalent in other packaging tools.

Type Checker Module Resolution Order

The following is the order that type checkers supporting this PEP should resolve modules containing type information:

  1. User code - the files the type checker is running on.

  2. Stubs or Python source manually put in the beginning of the path. Type checkers should provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages.

  3. Third party stub packages - these packages can supersede the installed untyped packages. They can be found at pkg-stubs for package pkg, however it is encouraged to check the package's metadata using packaging query APIs such as pkg_resources to assure that the package is meant for type checking, and is compatible with the installed version.

  4. Inline packages - finally, if there is nothing overriding the installed package, and it opts into type checking.

  5. Typeshed (if used) - Provides the stdlib types and several third party libraries

Type checkers that check a different Python version than the version they run on must find the type information in the site-packages/dist-packages of that Python version. This can be queried e.g. pythonX.Y -c 'import site; print(site.getsitepackages())'. It is also recommended that the type checker allow for the user to point to a particular Python binary, in case it is not in the path.

To check if a package has opted into type checking, type checkers are recommended to use the pkg_resources module to query the package metadata. If the typed package metadata has None as its value, the package has not opted into type checking, and the type checker should skip that package.

Implementation

A CPython branch with a modified distutils supporting the typed setup keyword lives here: [impl]. In addition, a sample package with inline types is available [typed_pkg], as well as a sample package [pkg_checker]_ which reads the metadata of installed packages and reports on their status as either not typed, inline typed, or a stub package.

Acknowledgements

This PEP would not have been possible without the ideas, feedback, and support of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, and Guido van Rossum.

Version History

References

.. [1] Typeshed (https://github.com/python/typeshed)

.. [2] PEP 484, Storing and Distributing Stub Files (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)

.. [impl] CPython sample implementation (https://github.com/ethanhs/cpython/tree/typeddist)

.. [typed_pkg] Sample typed package (https://github.com/ethanhs/sample-typed-package)

.. [pkg_checker] Sample package checker (https://github.com/ethanhs/check_typedpkg)

Copyright

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20171026/50fa3b9b/attachment.html>

More information about the Python-Dev mailing list