[Python-Dev] PEP 561 rework (original) (raw)

Ethan Smith ethan at ethanhs.me
Sun Nov 12 14:21:33 EST 2017

On Sun, Nov 12, 2017 at 9:53 AM, Jelle Zijlstra <jelle.zijlstra at gmail.com> wrote:

2017-11-12 3:40 GMT-08:00 Ethan Smith <ethan at ethanhs.me>: Hello,

I re-wrote my PEP to have typing opt-in be per-package rather than per-distribution. This greatly simplifies things, and thanks to the feedback and suggestions of Nick Coghlan, it is entirely compatible with older packaging tooling. The main idea is there are two types of packages: - types are packaged with runtime code (inline or stubs in the same package) - types are in a separate package (a third party or maintainer wants to ship type information, but not with runtime code). The PEP is live on python.org: https://www.python.org/dev/peps/pep-0561/ And as always, duplicated below. Cheers, Ethan Smith --------------------------------------------------- PEP: 561 Title: Distributing and Packaging Type Information Author: Ethan Smith <ethan at ethanhs.me> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017

Abstract ======== PEP 484 introduced type hinting to Python, with goals of making typing gradual and easy to adopt. Currently, typing information must be distributed manually. This PEP provides a standardized means to leverage existing tooling to package and distribute type information with minimal work and an ordering for type checkers to resolve modules and collect this information for type checking. Rationale ========= Currently, package authors wish to distribute code that has inline type information. Additionally, maintainers would like to distribute stub files to keep Python 2 compatibility while using newer annotation syntax. However, there is no standard method to distribute packages with type information. Also, if one wished to ship stub files privately the only method available would be via setting MYPYPATH or the equivalent to manually point to stubs. If the package can be released publicly, it can be added to typeshed [1]. However, this does not scale and becomes a burden on the maintainers of typeshed. In addition, it ties bug fixes in stubs to releases of the tool using typeshed. PEP 484 has a brief section on distributing typing information. In this section [2] the PEP recommends using shared/typehints/pythonX.Y/ for shipping stub files. However, manually adding a path to stub files for each third party library does not scale. The simplest approach people have taken is to add site-packages to their MYPYPATH, but this causes type checkers to fail on packages that are highly dynamic (e.g. sqlalchemy and Django). Definition of Terms =================== The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are to be interpreted as described in RFC 2119. "inline" - the types are part of the runtime code using PEP 526 and 3107 syntax. "stubs" - files containing only type information, empty of runtime code. "Distributions" are the packaged files which are used to publish and distribute a release. [3] "Module" a file containing Python runtime code or stubbed type information. "Package" a directory or directories that namespace Python modules. Specification ============= There are several motivations and methods of supporting typing in a package. This PEP recognizes three (3) types of packages that users of typing wish to create: 1. The package maintainer would like to add type information inline. 2. The package maintainer would like to add type information via stubs. 3. A third party or package maintainer would like to share stub files for a package, but the maintainer does not want to include them in the source of the package. This PEP aims to support these scenarios and make them simple to add to packaging and deployment. The two major parts of this specification are the packaging specifications and the resolution order for resolving module type information. The type checking spec is meant to replace the shared/typehints/pythonX.Y/ spec of PEP 484 [2]. New third party stub libraries SHOULD distribute stubs via the third party packaging methods proposed in this PEP in place of being added to typeshed. Typeshed will remain in use, but if maintainers are found, third party stubs in typeshed MAY be split into their own package. Packaging Type Information -------------------------- In order to make packaging and distributing type information as simple and easy as possible, packaging and distribution is done through existing frameworks. Package maintainers who wish to support type checking of their code MUST add a py.typed file to their package supporting typing. This marker is recursive, if a top-level package includes it, all sub-packages MUST support type checking as well. To have this file installed with the package, maintainers can use existing packaging options such as packagedata in distutils, shown below. Distutils option example:: ... packagedata = { 'pkg': ['py.typed'], }, ... For namespace packages, the py.typed file should be in the submodules of the namespace, to avoid conflicts and for clarity. Stub Only Packages '''''''''''''''''' For package maintainers wishing to ship stub files containing all of their type information, it is preferred that the *.pyi stubs are alongside the corresponding *.py files. However, the stubs can also be put in a separate package and distributed separately. Third parties can also find this method useful if they wish to distribute stub files. The name of the stub package MUST follow the scheme pkgstubs for type stubs for the package named pkg. The normal resolution order of checking *.pyi before *.py will be maintained. This is very minor, but what do you think of using "pkg-stubs" instead of "pkgstubs" (using a hyphen rather than an underscore)? This would make the name illegal to import as a normal Python package, which makes it clear that it's not a normal package. Also, there could be real packages named "stubs".

I suppose this makes sense. I checked PyPI and as of a few weeks ago there were no packages with the name pattern, but I like the idea of making it explicitly non-runtime importable. I cannot think of any reason not to do it, and the avoidance of confusion about the package being importable is a benefit. I will make the change with my next round of edits.

Third parties seeking to distribute stub files are encouraged to contact the

maintainer of the package about distribution alongside the package. If the maintainer does not wish to maintain or package stub files or type information inline, then a third party stub only package can be created.

In addition, stub only distributions SHOULD indicate which version(s) of the runtime package are supported by indicating the runtime distribution's version(s) through normal dependency data. For example, if there was a stub package flyingcircusstubs, it can indicate the versions of the runtime Flyingcircus distribution supported through installrequires in distutils based tools, or the equivalent in other packaging tools.

Type Checker Module Resolution Order ------------------------------------ The following is the order that type checkers supporting this PEP SHOULD resolve modules containing type information: 1. User code - the files the type checker is running on. 2. Stubs or Python source manually put in the beginning of the path. Type checkers SHOULD provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages. 3. Stub packages - these packages can supersede the installed packages. They can be found at pkgstubs for package pkg. 4. Inline packages - if there is nothing overriding the installed package, and it opts into type checking, inline types SHOULD be used. 5. Typeshed (if used) - Provides the stdlib types and several third party libraries. Type checkers that check a different Python version than the version they run on MUST find the type information in the site-packages/dist-packages of that Python version. This can be queried e.g. pythonX.Y -c 'import site; print(site.getsitepackages())'. It is also recommended that the type checker allow for the user to point to a particular Python binary, in case it is not in the path. Implementation ============== The proposed scheme of indicating support for typing is completely backwards compatible, and requires no modification to tooling. A sample package with inline types is available [typedpkg], as well as a sample package checker [pkgchecker] which reads the metadata of installed packages and reports on their status as either not typed, inline typed, or a stub package. Acknowledgements ================ This PEP would not have been possible without the ideas, feedback, and support of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Nathaniel Smith, and Guido van Rossum. Version History =============== * 2017-11-12 * Rewritten to use existing tooling only * No need to indicate kind of type information in metadata * Name of marker file changed from .typeinfo to py.typed * 2017-11-10 * Specification re-written to use package metadata instead of distribution metadata. * Removed stub only packages and merged into third party packages spec. * Removed suggestion for typecheckers to consider checking runtime versions * Implementations updated to reflect PEP changes. * 2017-10-26 * Added implementation references. * Added acknowledgements and version history. * 2017-10-06 * Rewritten to use .distinfo/METADATA over a distutils specific command. * Clarify versioning of third party stub packages. * 2017-09-11 * Added information about current solutions and typeshed. * Clarify rationale. References ========== .. [1] Typeshed (https://github.com/python/typeshed) .. [2] PEP 484, Storing and Distributing Stub Files (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files) .. [3] PEP 426 definitions (https://www.python.org/dev/peps/pep-0426/) .. [typedpkg] Sample typed package (https://github.com/ethanhs/sample-typed-package) .. [pkgchecker] Sample package checker (https://github.com/ethanhs/checktypedpkg) Copyright ========= This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle. zijlstra%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20171112/1fc86bef/attachment-0001.html>

More information about the Python-Dev mailing list