[Python-Dev] PEP 563: Postponed Evaluation of Annotations (original) (raw)

Steven D'Aprano steve at pearwood.info
Thu Nov 2 11:45:16 EDT 2017

On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:

PEP: 563 Title: Postponed Evaluation of Annotations

This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in _annotations_ in string form.

This means that now all annotations, not just forward references, are no longer validated at runtime and will allow arbitrary typos and errors:

def spam(n:itn): # now valid ...

Up to now, it has been only forward references that were vulnerable to that sort of thing. Of course running a type checker should pick those errors up, but the evaluation of annotations ensures that they are actually valid (not necessarily correct, but at least a valid name), even if you happen to not be running a type checker. That's useful.

Are we happy to live with that change?

Rationale and Goals ===================

PEP 3107 added support for arbitrary annotations on parts of a function definition. Just like default values, annotations are evaluated at function definition time. This creates a number of issues for the type hinting use case: * forward references: when a type hint contains names that have not been defined yet, that definition needs to be expressed as a string literal;

After all the discussion, I still don't see why this is an issue. Strings makes perfectly fine forward references. What is the problem that needs solving? Is this about people not wanting to type the leading and trailing ' around forward references?

* type hints are executed at module import time, which is not computationally free.

True; but is that really a performance bottleneck? If it is, that should be stated in the PEP, and state what typical performance improvement this change should give.

After all, if we're going to break people's code in order to improve performance, we should at least be sure that it improves performance :-)

Postponing the evaluation of annotations solves both problems.

Actually it doesn't. As your PEP says later:

This PEP is meant to solve the problem of forward references in type annotations. There are still cases outside of annotations where forward references will require usage of string literals. Those are listed in a later section of this document.

So the primary problem this PEP is designed to solve, isn't actually solved by this PEP.

(See Guido's comments, quoted later.)

Implementation ==============

In Python 4.0, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective _annotations_ dictionary. Static type checkers will see no difference in behavior,

Static checkers don't see annotations at all, since that's not available at edit/compile time. Static checkers see only the source code. The checker (and the human reader!) will no longer have the useful clue that something is a forward reference:

# before
class C:
    def method(self, other:'C'): 
        ...

since the quotes around C will be redundant and almost certainly left out. And if they aren't left out, then what are we to make of the annotation? Will the quotes be stripped out, or left in?

In other words, will method's annotations contain 'C' or "'C'"? That will make a difference when the type hint is eval'ed.

If an annotation was already a string, this string is preserved verbatim.

That's ambiguous. See above.

Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable (with the sole exception of class-level names resolved by typing.gettypehints()).

Even if you call get_type_hints from inside the function defining the local names?

def function(): A = something() def inner(x:A)->int: ... d = typing.get_type_hints(inner) return (d, inner)

I would expect that should work. Will it?

For code which uses annotations for other purposes, a regular eval(ann, globals, locals) call is enough to resolve the annotation.

Let's just hope nobody doing that has allowed any tainted strings to be stuffed into annotations.

* modules should use their own _dict_.

Which is better written as vars() with no argument, I believe. Or possibly globals().

If a function generates a class or a function with annotations that have to use local variables, it can populate the given generated object's _annotations_ dictionary directly, without relying on the compiler.

I don't understand this paragraph.

The biggest controversy on the issue was Guido van Rossum's concern that untokenizing annotation expressions back to their string form has no precedent in the Python programming language and feels like a hacky workaround. He said:

One thing that comes to mind is that it's a very random change to the language. It might be useful to have a more compact way to indicate deferred execution of expressions (using less syntax than lambda:). But why would the use case of type annotations be so all-important to change the language to do it there first (rather than proposing a more general solution), given that there's already a solution for this particular use case that requires very minimal syntax?

I agree with Guido's concern here. A more general solution would (hopefully!) be like a thunk, and might allow some interesting techniques unrelated to type checking. Just off the top of my head, say, late binding of default values (without the "if arg is None: arg = []" trick).

A few people voiced concerns that there are libraries using annotations for non-typing purposes. However, none of the named libraries would be invalidated by this PEP. They do require adapting to the new requirement to call eval() on the annotation with the correct globals and locals set.

Since this is likely to be a common task for such libraries, can we have a evaluate_annotations() function to do this, rather than have everyone reinvent the wheel?

def func(arg:int): ...

evaluate_annotations(func) assert func.annotations['arg'] is int

It could be a decorator, as well as modifying annotations in place.

I imagine something with a signature like this:

def evaluate_annotations( obj:Union[Function, Class], globals:Dict=None, locals:Dict=None )->Union[Function, Class]: """Evaluate the annotations of a function, or recursively a class and all its methods. Replace the annotations in place. Returns the modified argument, making this suitable as a decorator.

If globals is not given, it is taken from the function.__globals__ 
or class.__module__ if available. If locals is not given, it 
defaults to the current locals.
"""

-- Steve

More information about the Python-Dev mailing list