Self as TypeVar default (original) (raw)
This came up in AbstractContextManager.__enter__ not properly annotated · Issue #13950 · python/typeshed · GitHub.
In typeshed, contextlib.AbstractContextManager is an ABC (and a protocol) that is generic over the return type of __enter__. But in CPython it has a default implementation that returns Self. Ideally, we’d give the type variable for the return type of __enter__ a default of Self to reflect this, but both mypy and pyright currently don’t like that.
The Typing Spec is silent about what is allowed as argument for default. Should Self be allowed? What would happen when such a type var is used in contexts where Self is normally not allowed?
srittau (Sebastian Rittau) May 6, 2025, 1:07pm 2
There is a chapter Valid Locations for Self, though, which currently forbids Self outside class contexts. This might theoretically allow definitions such as class Foo[T=Self]: ..., but that would be inconsistent, and it’s not clear, whether this is really considered “inside a class context”.
jorenham (Joren Hammudoglu) May 6, 2025, 4:05pm 3
In the optype library I maintain, there are (generic) Protocols for each of the binary- and unary operator dunder methods, e.g. optype.CanNeg[R] for __neg__: () -> R, optype.CanISub[T, R] for __isub__: (T) -> R, and optype.CanEnter[R] for __enter__: () -> R.
For many of them, it’s common to return Self. But Self cannot be a typar default, and Self cannot be passed as type argument either. This means that these generic optype protocols are currently useless if you want to match against e.g. __neg__: () -> Self, __iadd__: (T) -> Self, or __enter__: () -> Self.
I worked around it by creating a second Protocol that uses Self instead of the type parameter R. So there’s a CanNegSelf for CanNeg[R], a CanISubSelf[T] for CanISub[T, R], a CanEnterSelf for CanEnter[R], etc.. The full list can be found in the optype docs.
Anyway, to make a long story short; I really would like to see this happen, as well 
ImogenBits (Imogen) May 6, 2025, 7:53pm 4
Would just allowing Self to be a type var default actually help this situation very much? If you are able to do that you’d presumably write something like this:
class SomeBase[T=Self]:
def __enter__(self) -> T:
return self
But then you’d get a type error in that function definition since self is an instance of Self, but not necessarily T. On the other hand, if you type the function return value as Self to make the default implementation type check, no subclasses can change it to some other type.
I think the problem here is that we don’t really need just a default value for a type variable here, but to couple that default value with the default implementation of the method. If a user wants to change the type var, they have to also provide their own implementation for the method. Or from a different perspective, the information doesn’t flow from the type var to the method but the other way around. Like, list[T] is basically letting you choose whatever T you want and then all its methods behave accordingly. But SomeBase[T] is saying that you must choose a T that corresponds to the enter method your subclass is using.
To get around that we’d have to also add some mechanism to mark default implementations as only being usable if the type var is chosen to match it. So e.g. something like this:
class SomeBase[T=Self]:
@only_default_types
def __enter__(self) -> T:
return self
would make it so that type checkers will type check the default implementation as though T is substituted with its default argument, and will error on any subclasses that change T (in a way that makes it non-assignable) without providing their own enter method.
Self shouldn’t be allowed as a default. This becomes more obvious if you expand out what Self is syntax sugar for, as that’s making a typevar act as a default. Where would the solution for that default be sourced? (note: it’s well known that typevars as bounds is equivalent to HKTs, be careful saying we can “just” do this for defaults)
jorenham (Joren Hammudoglu) May 6, 2025, 11:29pm 6
In case of generic functions, or some type alias to Callable or something, it indeed wouldn’t make any sense. And the same can be said about using Self itself. So by extension, a typar that defaults to Self can only be used in places where Self is valid. That limits it to user-defined generic types, but it will require some additional restrictions.
erictraut (Eric Traut) May 7, 2025, 12:33am 7
I agree with @mikeshardmind that Self shouldn’t be allowed as a default.
A default value for a type parameter must be valid if you omit the type argument, and you should be able to provide the default value explicitly if you want.
class Foo[T = int]: ...
f1: Foo # This is the same as `Foo[int]`
f2: Foo[int] # Explicitly supplying the default is OK too
Self cannot be used to explicitly specialize a generic class unless Self has meaning in that context.
class Bar1[T]: ...
b1: Bar1[Self] # Nonsensical because Self has no meaning here
class Bar2[T = Self]: ...
b2: Bar2 # Likewise nonsensical
I think @jorenham put his finger on the real problem with the current definition of AbstractContextManager. It’s attempting to be a generic protocol and an abstract class with a partial implementation. The class definition can’t accurately describe both of these. The default implementation of __enter__ is defined to return _T_co, but in reality it returns Self.
Consider the following:
class C1(AbstractContextManager[int, None]):
def __exit__(
self,
exc_type: Optional[type[BaseException]],
exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType],
) -> None:
return None
# C1 inherits an implementation of `__enter__` that does not
# return what the type definition claims it does.
reveal_type(C1().__enter__()) # int, but at runtime is `C1`
srittau (Sebastian Rittau) May 7, 2025, 9:06am 8
While it won’t help much with the actual implementation, it helps with the interface as represented in the type stubs – and therefore the users of AbstractContextManager. One # type: ignore in the implementation of AbstractContextManager (if the standard library gains type annotations at some point) saves potentially thousands of # type: ignores by users of AbstractContextManager. (And for other similar classes.)
srittau (Sebastian Rittau) May 7, 2025, 9:19am 9
I don’t see a conceptual problem with having other type vars be the default for a type var either. Other languages already allow this. Typescript example:
interface Foo<X, Y = X> {
a: Y;
}
const foo: Foo<number> = { a: 1 };
When using the legacy type var definition (typing.TypeVar), resolving Self would need to be deferred, but this is only awkward in the legacy case. (And maybe would not worth supporting there, even if it’s supported in the modern syntax.)
srittau (Sebastian Rittau) May 7, 2025, 9:25am 10
Sure, but that shouldn’t preclude it from being used in contexts where it makes sense.
Yes, it’s unfortunate that it exists, but it is a real-world problem that allowing Self would address. The fact is that since the introduction of protocols to typing, the existing ABCs were partially declared to be protocols. Changing this now is unrealistic.
jorenham (Joren Hammudoglu) May 7, 2025, 10:14am 11
You can also do this in Python at the moment
from typing import Protocol
class CanPosNeg[PosT, NegT = PosT](Protocol):
def __pos__(self) -> PosT: ...
def __neg__(self) -> NegT: ...
reveal_type(CanPosNeg[int])
# Type of "CanPosNeg[int]" is "type[CanPosNeg[int, int]]"
Maybe it could help make sense out of this, if we imagine that every user-defined generic class has a secret Self typar?
jorenham (Joren Hammudoglu) May 7, 2025, 10:37am 12
Possible workaround:
class _Selfie: ... # imposter `Self`
type _AbsConManDefault = AbsConMan[_Selfie]
class AbsConMan[EnterT = _Selfie, ExitT = None](Protocol):
@overload
def __enter__[_SelfT: _AbsConManDefault](self: _SelfT) -> _SelfT: ...
@overload
def __enter__(self) -> EnterT: ...
def __exit__(self, *args) -> ExitT: ...
def with_self(acm: AbsConMan):
with acm as x:
reveal_type(x) # AbsConMan[_Selfie, None]
def with_str(acm: AbsConMan[str]):
with acm as x:
reveal_type(x) # AbsConMan[str, None]