[Python-Dev] PEP 544: Protocols (original) (raw)

Ivan Levkivskyi levkivskyi at gmail.com
Mon Mar 20 08:00:06 EDT 2017

Hi all,

PEP 484 specifies semantics for type hints. These type hints are used by various tools, including static type checkers. However, PEP 484 only specifies the semantics for nominal subtyping (subtyping based on subclassing). Here we propose a specification for semantics of structural subtyping (static duck typing).

Previous discussions on this PEP happened at:

https://mail.python.org/pipermail/python-ideas/2015-September/thread.html#35859 https://github.com/python/typing/issues/11 https://github.com/python/peps/pull/224

-- Ivan

===========================================

PEP: 544 Title: Protocols Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Ivan Levkivskyi <levkivskyi at gmail.com>, Jukka Lehtosalo < jukka.lehtosalo at iki.fi>, Łukasz Langa <lukasz at langa.pl> Discussions-To: Python-Dev <python-dev at python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 05-Mar-2017 Python-Version: 3.7

Abstract

Type hints introduced in PEP 484 can be used to specify type metadata for static type checkers and other third party tools. However, PEP 484 only specifies the semantics of nominal subtyping. In this PEP we specify static and runtime semantics of protocol classes that will provide a support for structural subtyping (static duck typing).

.. _rationale:

Rationale and Goals

Currently, PEP 484 and the typing module [typing]_ define abstract base classes for several common Python protocols such as Iterable and Sized. The problem with them is that a class has to be explicitly marked to support them, which is unpythonic and unlike what one would normally do in idiomatic dynamically typed Python code. For example, this conforms to PEP 484::

from typing import Sized, Iterable, Iterator

class Bucket(Sized, Iterable[int]): ... def len(self) -> int: ... def iter(self) -> Iterator[int]: ...

The same problem appears with user-defined ABCs: they must be explicitly subclassed or registered. This is particularly difficult to do with library types as the type objects may be hidden deep in the implementation of the library. Also, extensive use of ABCs might impose additional runtime costs.

The intention of this PEP is to solve all these problems by allowing users to write the above code without explicit base classes in the class definition, allowing Bucket to be implicitly considered a subtype of both Sized and Iterable[int] by static type checkers using structural [wiki-structural]_ subtyping::

from typing import Iterator, Iterable

class Bucket: ... def len(self) -> int: ... def iter(self) -> Iterator[int]: ...

def collect(items: Iterable[int]) -> int: ... result: int = collect(Bucket()) # Passes type check

Note that ABCs in typing module already provide structural behavior at runtime, isinstance(Bucket(), Iterable) returns True. The main goal of this proposal is to support such behavior statically. The same functionality will be provided for user-defined protocols, as specified below. The above code with a protocol class matches common Python conventions much better. It is also automatically extensible and works with additional, unrelated classes that happen to implement the required protocol.

Nominal vs structural subtyping

Structural subtyping is natural for Python programmers since it matches the runtime semantics of duck typing: an object that has certain properties is treated independently of its actual runtime class. However, as discussed in PEP 483, both nominal and structural subtyping have their strengths and weaknesses. Therefore, in this PEP we do not propose to replace the nominal subtyping described by PEP 484 with structural subtyping completely. Instead, protocol classes as specified in this PEP complement normal classes, and users are free to choose where to apply a particular solution. See section on rejected_ ideas at the end of this PEP for additional motivation.

Non-goals

At runtime, protocol classes will be simple ABCs. There is no intent to provide sophisticated runtime instance and class checks against protocol classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are completely optional:

Existing Approaches to Structural Subtyping

Before describing the actual specification, we review and comment on existing approaches related to structural subtyping in Python and other languages:

invariants might be difficult to implement. However, the idea of marking an interface class with a special base class is reasonable and easy to implement both statically and at runtime.

especially in static context. However, in a runtime context, ABCs are good candidates for protocol classes and they are already used extensively in the typing module.

behavior of protocols. As discussed in rationale, we propose to add static support for such behavior. In addition, to allow users to achieve such runtime behavior for user defined protocols a special @runtime decorator will be provided, see detailed discussion below.

postpone this; see rejected_ ideas. In general, the idea of static protocol checking without runtime implications looks reasonable, and basically this proposal follows the same line.

.. _specification:

Specification

Terminology

We propose to use the term protocols for types supporting structural subtyping. The reason is that the term iterator protocol, for example, is widely understood in the community, and coming up with a new term for this concept in a statically typed context would just create confusion.

This has the drawback that the term protocol becomes overloaded with two subtly different meanings: the first is the traditional, well-known but slightly fuzzy concept of protocols such as iterator; the second is the more explicitly defined concept of protocols in statically typed code. The distinction is not important most of the time, and in other cases we propose to just add a qualifier such as protocol classes when referring to the static type concept.

If a class includes a protocol in its MRO, the class is called an explicit subclass of the protocol. If a class is a structural subtype of a protocol, it is said to implement the protocol and to be compatible with a protocol. If a class is compatible with a protocol but the protocol is not included in the MRO, the class is an implicit subtype of the protocol.

The attributes (variables and methods) of a protocol that are mandatory for other class in order to be considered a structural subtype are called protocol members.

.. _definition:

Defining a protocol

Protocols are defined by including a special new class typing.Protocol (an instance of abc.ABCMeta) in the base classes list, preferably at the end of the list. Here is a simple example::

from typing import Protocol

class SupportsClose(Protocol): def close(self) -> None: ...

Now if one defines a class Resource with a close() method that has a compatible signature, it would implicitly be a subtype of SupportsClose, since the structural subtyping is used for protocol types::

class Resource: ... def close(self) -> None: self.file.close() self.lock.release()

Apart from few restrictions explicitly mentioned below, protocol types can be used in every context where a normal types can::

def close_all(things: Iterable[SupportsClose]) -> None: for t in things: t.close()

f = open('foo.txt') r = Resource() close_all([f, r]) # OK! close_all([1]) # Error: 'int' has no 'close' method

Note that both the user-defined class Resource and the built-in IO type (the return type of open()) are considered subtypes of SupportsClose, because they provide a close() method with a compatible type signature.

Protocol members

All methods defined in the protocol class body are protocol members, both normal and decorated with @abstractmethod. If some or all parameters of protocol method are not annotated, then their types are assumed to be Any (see PEP 484). Bodies of protocol methods are type checked, except for methods decorated with @abstractmethod with trivial bodies. A trivial body can contain a docstring. Example::

from typing import Protocol from abc import abstractmethod

class Example(Protocol): def first(self) -> int: # This is a protocol member return 42

  @abstractmethod
  def second(self) -> int:    # Method without a default implementation
      """Some method."""

Note that although formally the implicit return type of a method with a trivial body is None, type checker will not warn about above example, such convention is similar to how methods are defined in stub files. Static methods, class methods, and properties are equally allowed in protocols.

To define a protocol variable, one must use PEP 526 variable annotations in the class body. Additional attributes only defined in the body of a method by assignment via self are not allowed. The rationale for this is that the protocol class implementation is often not shared by subtypes, so the interface should not depend on the default implementation. Examples::

from typing import Protocol, List

class Template(Protocol): name: str # This is a protocol member value: int = 0 # This one too (with default)

  def method(self) -> None:
      self.temp: List[int] = [] # Error in type checker

To distinguish between protocol class variables and protocol instance variables, the special ClassVar annotation should be used as specified by PEP 526.

Explicitly declaring implementation

To explicitly declare that a certain class implements the given protocols, they can be used as regular base classes. In this case a class could use default implementations of protocol members. typing.Sequence is a good example of a protocol with useful default methods.

Abstract methods with trivial bodies are recognized by type checkers as having no default implementation and can't be used via super() in explicit subclasses. The default implementations can not be used if the subtype relationship is implicit and only via structural subtyping -- the semantics of inheritance is not changed. Examples::

class PColor(Protocol):
    @abstractmethod
    def draw(self) -> str:
        ...
    def complex_method(self) -> int:
        # some complex code here

class NiceColor(PColor):
    def draw(self) -> str:
        return "deep blue"

class BadColor(PColor):
    def draw(self) -> str:
        return super().draw()  # Error, no default implementation

class ImplicitColor:   # Note no 'PColor' base here
    def draw(self) -> str:
        return "probably gray"
    def comlex_method(self) -> int:
        # class needs to implement this

nice: NiceColor
another: ImplicitColor

def represent(c: PColor) -> None:
    print(c.draw(), c.complex_method())

represent(nice) # OK
represent(another) # Also OK

Note that there is no conceptual difference between explicit and implicit subtypes, the main benefit of explicit subclassing is to get some protocol methods "for free". In addition, type checkers can statically verify that the class actually implements the protocol correctly::

class RGB(Protocol):
    rgb: Tuple[int, int, int]

    @abstractmethod
    def intensity(self) -> int:
        return 0

class Point(RGB):
    def __init__(self, red: int, green: int, blue: str) -> None:
        self.rgb = red, green, blue  # Error, 'blue' must be 'int'

    # Type checker might warn that 'intensity' is not defined

A class can explicitly inherit from multiple protocols and also form normal classes. In this case methods are resolved using normal MRO and a type checker verifies that all subtyping are correct. The semantics of @abstractmethod is not changed, all of them must be implemented by an explicit subclass before it could be instantiated.

Merging and extending protocols

The general philosophy is that protocols are mostly like regular ABCs, but a static type checker will handle them specially. Subclassing a protocol class would not turn the subclass into a protocol unless it also has typing.Protocol as an explicit base class. Without this base, the class is "downgraded" to a regular ABC that cannot be used with structural subtyping.

A subprotocol can be defined by having both one or more protocols as immediate base classes and also having typing.Protocol as an immediate base class::

from typing import Sized, Protocol

class SizedAndCloseable(Sized, Protocol): def close(self) -> None: ...

Now the protocol SizedAndCloseable is a protocol with two methods, __len__ and close. If one omits Protocol in the base class list, this would be a regular (non-protocol) class that must implement Sized. If Protocol is included in the base class list, all the other base classes must be protocols. A protocol can't extend a regular class.

Alternatively, one can implement SizedAndCloseable like this, assuming the existence of SupportsClose from the example in definition_ section::

from typing import Sized

class SupportsClose(...): ... # Like above

class SizedAndCloseable(Sized, SupportsClose, Protocol): pass

The two definitions of SizedAndClosable are equivalent. Subclass relationships between protocols are not meaningful when considering subtyping, since structural compatibility is the criterion, not the MRO.

Note that rules around explicit subclassing are different from regular ABCs, where abstractness is simply defined by having at least one abstract method being unimplemented. Protocol classes must be marked explicitly.

Generic and recursive protocols

Generic protocols are important. For example, SupportsAbs, Iterable and Iterator are generic protocols. They are defined similar to normal non-protocol generic types::

T = TypeVar('T', covariant=True)

class Iterable(Protocol[T]): @abstractmethod def iter(self) -> Iterator[T]: ...

Note that Protocol[T, S, ...] is allowed as a shorthand for Protocol, Generic[T, S, ...].

Recursive protocols are also supported. Forward references to the protocol class names can be given as strings as specified by PEP 484. Recursive protocols will be useful for representing self-referential data structures like trees in an abstract fashion::

class Traversable(Protocol): leaves: Iterable['Traversable']

Using Protocols

Subtyping relationships with other types

Protocols cannot be instantiated, so there are no values with protocol types. For variables and parameters with protocol types, subtyping relationships are subject to the following rules:

Generic protocol types follow the same rules of variance as non-protocol types. Protocol types can be used in all contexts where any other types can be used, such as in Union, ClassVar, type variables bounds, etc. Generic protocols follow the rules for generic abstract classes, except for using structural compatibility instead of compatibility defined by inheritance relationships.

Unions and intersections of protocols

Union of protocol classes behaves the same way as for non-protocol classes. For example::

from typing import Union, Optional, Protocol

class Exitable(Protocol): def exit(self) -> int: ... class Quitable(Protocol): def quit(self) -> Optional[int]: ...

def finish(task: Union[Exitable, Quitable]) -> int: ... class GoodJob: ... def quit(self) -> int: return 0 finish(GoodJob()) # OK

One can use multiple inheritance to define an intersection of protocols. Example::

from typing import Sequence, Hashable

class HashableFloats(Sequence[float], Hashable, Protocol): pass

def cached_func(args: HashableFloats) -> float: ... cached_func((1, 2, 3)) # OK, tuple is both hashable and sequence

If this will prove to be a widely used scenario, then a special intersection type construct may be added in future as specified by PEP 483, see rejected_ ideas for more details.

Type[] with protocols

Variables and parameters annotated with Type[Proto] accept only concrete (non-protocol) subtypes of Proto. The main reason for this is to allow instantiation of parameters with such type. For example::

class Proto(Protocol): @abstractmethod def meth(self) -> int: ... class Concrete: def meth(self) -> int: return 42

def fun(cls: Type[Proto]) -> int: return cls().meth() # OK fun(Proto) # Error fun(Concrete) # OK

The same rule applies to variables::

var: Type[Proto] var = Proto # Error var = Concrete # OK var().meth() # OK

Assigning an ABC or a protocol class to a variable is allowed if it is not explicitly typed, and such assignment creates a type alias. For normal (non-abstract) classes, the behavior of Type[] is not changed.

NewType() and type aliases

Protocols are essentially anonymous. To emphasize this point, static type checkers might refuse protocol classes inside NewType() to avoid an illusion that a distinct type is provided::

form typing import NewType , Protocol, Iterator

class Id(Protocol): code: int secrets: Iterator[bytes]

UserId = NewType('UserId', Id) # Error, can't provide distinct type

On the contrary, type aliases are fully supported, including generic type aliases::

from typing import TypeVar, Reversible, Iterable, Sized

T = TypeVar('T') class SizedIterable(Iterable[T], Sized, Protocol): pass CompatReversible = Union[Reversible[T], SizedIterable[T]]

.. _discussion:

@runtime decorator and narrowing types by isinstance()

The default semantics is that isinstance() and issubclass() fail for protocol types. This is in the spirit of duck typing -- protocols basically would be used to model duck typing statically, not explicitly at runtime.

However, it should be possible for protocol types to implement custom instance and class checks when this makes sense, similar to how Iterable and other ABCs in collections.abc and typing already do it, but this is limited to non-generic and unsubscripted generic protocols (Iterable is statically equivalent to Iterable[Any]`). The typingmodule will define a special@runtimeclass decorator that provides the same semantics for class and instance checks as forcollections.abc`` classes, essentially making them "runtime protocols"::

from typing import runtime, Protocol

@runtime class Closeable(Protocol): def close(self): ...

assert isinstance(open('some/file'), Closeable)

Static type checkers will understand isinstance(x, Proto) and issubclass(C, Proto) for protocols defined with this decorator (as they already do for Iterable etc.). Static type checkers will narrow types after such checks by the type erased Proto (i.e. with all variables having type Any and all methods having type Callable[..., Any]). Note that isinstance(x, Proto[int]) etc. will always fail in agreement with PEP 484. Examples::

from typing import Iterable, Iterator, Sequence

def process(items: Iterable[int]) -> None: if isinstance(items, Iterator): # 'items' have type 'Iterator[int]' here elif isinstance(items, Sequence[int]): # Error! Can't use 'isinstance()' with subscripted protocols

Note that instance checks are not 100% reliable statically, this is why this behavior is opt-in, see section on rejected_ ideas for examples.

Using Protocols in Python 2.7 - 3.5

Variable annotation syntax was added in Python 3.6, so that the syntax for defining protocol variables proposed in specification_ section can't be used in earlier versions. To define these in earlier versions of Python one can use properties::

class Foo(Protocol): @property def c(self) -> int: return 42 # Default value can be provided for property...

  @abstractproperty
  def d(self) -> int:   # ... or it can be abstract
      return 0

In Python 2.7 the function type comments should be used as per PEP 484. The typing module changes proposed in this PEP will be also backported to earlier versions via the backport currently available on PyPI.

Runtime Implementation of Protocol Classes

Implementation details

The runtime implementation could be done in pure Python without any effects on the core interpreter and standard library except in the typing module:

Changes in the typing module

The following classes in typing module will be protocols:

Most of these classes are small and conceptually simple. It is easy to see what are the methods these protocols implement, and immediately recognize the corresponding runtime protocol counterpart. Practically, few changes will be needed in typing since some of these classes already behave the necessary way at runtime. Most of these will need to be updated only in the corresponding typeshed stubs [typeshed]_.

All other concrete generic classes such as List, Set, IO, Deque, etc are sufficiently complex that it makes sense to keep them non-protocols (i.e. require code to be explicit about them). Also, it is too easy to leave some methods unimplemented by accident, and explicitly marking the subclass relationship allows type checkers to pinpoint the missing implementations.

Introspection

The existing class introspection machinery (dir, __annotations__ etc) can be used with protocols. In addition, all introspection tools implemented in the typing module will support protocols. Since all attributes need to be defined in the class body based on this proposal, protocol classes will have even better perspective for introspection than regular classes where attributes can be defined implicitly -- protocol attributes can't be initialized in ways that are not visible to introspection (using setattr(), assignment via self, etc.). Still, some things like types of attributes will not be visible at runtime in Python 3.5 and earlier, but this looks like a reasonable limitation.

There will be only limited support of isinstance() and issubclass() as discussed above (these will always fail with TypeError for subscripted generic protocols, since a reliable answer could not be given at runtime in this case). But together with other introspection tools this give a reasonable perspective for runtime type checking tools.

.. _rejected:

Rejected/Postponed Ideas

The ideas in this section were previously discussed in [several]_ [discussions]_ [elsewhere]_.

Make every class a protocol by default

Some languages such as Go make structural subtyping the only or the primary form of subtyping. We could achieve a similar result by making all classes protocols by default (or even always). However we believe that it is better to require classes to be explicitly marked as protocols, for the following reasons:

Support optional protocol members

We can come up with examples where it would be handy to be able to say that a method or data attribute does not need to be present in a class implementing a protocol, but if it is present, it must conform to a specific signature or type. One could use a hasattr() check to determine whether they can use the attribute on a particular instance.

Languages such as TypeScript have similar features and apparently they are pretty commonly used. The current realistic potential use cases for protocols in Python don't require these. In the interest of simplicity, we propose to not support optional methods or attributes. We can always revisit this later if there is an actual need.

Make protocols interoperable with other approaches

The protocols as described here are basically a minimal extension to the existing concept of ABCs. We argue that this is the way they should be understood, instead of as something that replaces Zope interfaces, for example. Attempting such interoperabilities will significantly complicate both the concept and the implementation.

On the other hand, Zope interfaces are conceptually a superset of protocols defined here, but using an incompatible syntax to define them, because before PEP 526 there was no straightforward way to annotate attributes. In the 3.6+ world, zope.interface might potentially adopt the Protocol syntax. In this case, type checkers could be taught to recognize interfaces as protocols and make simple structural checks with respect to them.

Use assignments to check explicitly that a class implements a protocol

In Go language the explicit checks for implementation are performed via dummy assignments [golang]_. Such a way is also possible with the current proposal. Example::

class A: def len(self) -> float: return ...

_: Sized = A() # Error: A.len doesn't conform to 'Sized' # (Incompatible return type 'float')

This approach moves the check away from the class definition and it almost requires a comment as otherwise the code probably would not make any sense to an average reader -- it looks like dead code. Besides, in the simplest form it requires one to construct an instance of A, which could be problematic if this requires accessing or allocating some resources such as files or sockets. We could work around the latter by using a cast, for example, but then the code would be ugly. Therefore we discourage the use of this pattern.

Support isinstance() checks by default

The problem with this is instance checks could be unreliable, except for situations where there is a common signature convention such as Iterable. For example::

class P(Protocol): def common_method_name(self, x: int) -> int: ...

class X: def common_method_name(self) -> None: ... # Note different signature

def do_stuff(o: Union[P, X]) -> int: if isinstance(o, P): return o.common_method_name(1) # oops, what if it's an X instance?

Another potentially problematic case is assignment of attributes after instantiation::

class P(Protocol): x: int

class C: def initialize(self) -> None: self.x = 0

c = C() isinstance(c1, P) # False c.initialize() isinstance(c, P) # True

def f(x: Union[P, int]) -> None: if isinstance(x, P): # static type of x is P here ... else: # type of x is "int" here? print(x + 1)

f(C()) # oops

We argue that requiring an explicit class decorator would be better, since one can then attach warnings about problems like this in the documentation. The user would be able to evaluate whether the benefits outweigh the potential for confusion for each protocol and explicitly opt in -- but the default behavior would be safer. Finally, it will be easy to make this behavior default if necessary, while it might be problematic to make it opt-in after being default.

Provide a special intersection type construct

There was an idea to allow Proto = All[Proto1, Proto2, ...] as a shorthand for::

class Proto(Proto1, Proto2, ..., Protocol): pass

However, it is not yet clear how popular/useful it will be and implementing this in type checkers for non-protocol classes could be difficult. Finally, it will be very easy to add this later if needed.

References

.. [typing] https://docs.python.org/3/library/typing.html

.. [wiki-structural] https://en.wikipedia.org/wiki/Structural_type_system

.. [zope-interfaces] https://zopeinterface.readthedocs.io/en/latest/

.. [abstract-classes] https://docs.python.org/3/library/abc.html

.. [collections-abc] https://docs.python.org/3/library/collections.abc.html

.. [typescript] https://www.typescriptlang.org/docs/handbook/interfaces.html

.. [golang] https://golang.org/doc/effective_go.html#interfaces_and_types

.. [typeshed] https://github.com/python/typeshed/

.. [mypy] http://github.com/python/mypy/

.. [several]

https://mail.python.org/pipermail/python-ideas/2015-September/thread.html#35859

.. [discussions] https://github.com/python/typing/issues/11

.. [elsewhere] https://github.com/python/peps/pull/224

Copyright

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20170320/44602e3c/attachment-0001.html>

More information about the Python-Dev mailing list