Issue 46195: Annotated and Optional get_type_hints buggy interaction (original) (raw)
Created on 2021-12-29 18:38 by med2277, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Messages (8)
Author: Mehdi2277 (med2277)
Date: 2021-12-29 18:38
This is two closely related issues with get_type_hints on an optional annotated member. I'm testing with Annotated/get_type_hints from typing extensions on 3.8 and assuming they're backport equivalent to current behavior.
The first issue is get_type_hints has inconsistent behavior depending on whether annotation comes from a function with a None default or an attribute with a None default.
class Foo: def init( self, x: Annotated[Optional[str], "doc string"] = None, ): ...
class Foo2: x: Annotated[Optional[str], "doc string"] = None
get_type_hints(Foo.init)
{'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}
get_type_hints(Foo2)
{'x': typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string']}
Attributes with a None default are not wrapped by get_type_hints, but function parameters. Which of the two behaviors is correct I don't know, but I'd expect the two to be equivalent annotations.
The second issue is for function arguments with a None default the optional wrapper happens even if the type inside annotated already has optional. Example,
from typing_extensions import Annotated, get_type_hints
class Foo: def init( self, x: Annotated[Optional[str], "doc string"] = None, ): ...
get_type_hints(Foo.init, include_extras=True)
{'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}
For Annotated types I would expect any type rules like wrapping to apply only to the first argument and not the entire annotation. I mainly ran into this for a runtime type introspection library (similar in spirit to pydantic).
As a note include_extras being True or False while it changes type is an issue in either case. With include_extras as False the Annotated goes away, but the type still gets double wrapped as an Optional.
Author: Guido van Rossum (gvanrossum) * 
Date: 2021-12-29 19:54
Could you try with 3.10 and the stdlib typing.Annotated please? There might be changes (in the past a default of None automatically caused an Optional to be added, but we changed our minds.-- --Guido (mobile)
Author: Nikita Sobolev (sobolevn) * 
Date: 2021-12-30 00:38
I can verify that this happens on 3.10 and main branches:
from typing import Annotated, Optional, get_type_hints
class Foo:
def __init__(self, x: Annotated[Optional[str], "d"] = None): ...
class Foo2:
x: Annotated[Optional[str], "d"] = None
print(get_type_hints(Foo.__init__, include_extras=False)) # ok
# {'x': typing.Optional[str]}
print(get_type_hints(Foo2, include_extras=False)) # ok
# {'x': typing.Optional[str]}
print(get_type_hints(Foo.__init__, include_extras=True)) # not ok?
# {'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}
print(get_type_hints(Foo2, include_extras=True)) # ok
# {'x': typing.Annotated[typing.Optional[str], 'd']}Notice that 3rd case does not look correct: {'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}
In my opinion it should be {'x': typing.Annotated[typing.Optional[str], 'd']}
I will look into it! :)
Author: Nikita Sobolev (sobolevn) *
Date: 2021-12-30 00:39
And on 3.9 as well.
Author: Nikita Sobolev (sobolevn) *
Date: 2021-12-30 10:38
As Guido said, the root cause of this problem is because None default automatically adds Optional to the resulting type.
So, what happens there:
- correct
valueis passed to_eval_type, correct resulttyping.Annotated[typing.Optional[str], 'd']is returned at this point - then
if name in defaults and defaults[name] is None:adds extraOptionalannotation on top ofAnnotated
in the past a default of None automatically caused an Optional to be added, but we changed our minds
Guido, are you talking about https://github.com/python/typing/issues/275 ?
Now all type-checkers (AFAIK) support something similar to --no-implicit-optional mode.
Having this in mind, I see different solutions to the current problem:
- Remove
Optionalinference withNonedefault. This is going to be a some-what breaking change. The only positive side of this is that we can really simplify our code (mainly because the other solution is to complicate our code even more). - Or we can change this place to explicitly check for
Annotatedtype and its internal type. This should be the easiest to write and backport. But, it still has some complexity to it. I think that this is a better solution: we don't break existing behavior, change is local and pretty trivial.
Also caused by this:
Author: Guido van Rossum (gvanrossum) *
Date: 2021-12-30 17:21
Yes, we changed PEP 484 in https://github.com/python/peps/pull/689.
So get_type_hints() should follow suit.
Author: Jelle Zijlstra (JelleZijlstra) *
Date: 2022-03-02 05:30
New changeset 20a1c8ee4bcb1c421b7cca1f3f5d6ad7ce30a9c9 by Nikita Sobolev in branch 'main':
bpo-46195: Do not add Optional in get_type_hints (GH-30304)
https://github.com/python/cpython/commit/20a1c8ee4bcb1c421b7cca1f3f5d6ad7ce30a9c9
Author: Jelle Zijlstra (JelleZijlstra) *
Date: 2022-03-02 05:31
This is now fixed in 3.11, but we'll leave 3.10 and 3.9 alone. Thanks for your bug report!
History
Date
User
Action
Args
2022-04-11 14:59:54
admin
set
github: 90353
2022-03-02 05:31:58
JelleZijlstra
set
status: open -> closed
resolution: fixed
messages: +
stage: patch review -> resolved
2022-03-02 05:30:27
JelleZijlstra
set
nosy: + JelleZijlstra
messages: +
2021-12-30 18:40:26
sobolevn
set
keywords: + patch
stage: patch review
pull_requests: + <pull%5Frequest28517>
2021-12-30 17:21:30
gvanrossum
set
messages: +
2021-12-30 10:38:57
sobolevn
set
messages: +
2021-12-30 00:39:09
sobolevn
set
messages: +
versions: + Python 3.9
2021-12-30 00:38:21
sobolevn
set
versions: + Python 3.10, Python 3.11, - Python 3.8
2021-12-30 00:38:12
sobolevn
set
nosy: + sobolevn
messages: +
2021-12-29 19:54:58
gvanrossum
set
messages: +
2021-12-29 19:10:03
med2277
set
type: behavior
2021-12-29 18:38:47
med2277
create