Polymorphic inference: basic support for variadic types by ilevkivskyi · Pull Request #15879 · python/mypy (original) (raw)

This is the fifth PR in the series started by #15287, and a last one for the foreseeable future. This completes polymorphic inference sufficiently for extensive experimentation, and enabling polymorphic fallback by default.

Remaining items for which I am going to open follow-up issues:

• Enable --new-type-inference by default (should be done before everything else in this list).
• Use polymorphic inference during unification.
• Use polymorphic inference as primary an only mechanism, rather than a fallback if basic inference fails in some way.
• Move apply_poly() logic from checkexpr.py to applytype.py (this one depends on everything above).
• Experiment with backtracking in the new solver.
• Experiment with universal quantification for types other that Callable (btw we already have a hacky support for capturing a generic function in an instance with ParamSpec).

Now some comments on the PR proper. First of all I decided to do some clean-up of TypeVarTuple support, but added only strictly necessary parts of the cleanup here. Everything else will be in follow up PR(s). The polymorphic inference/solver/application is practically trivial here, so here is my view on how I see large-scale structure of TypeVarTuple implementation:

• There should be no special-casing in applytype.py, so I deleted everything from there (as I did for ParamSpec) and complemented visit_callable_type() in expandtype.py. Basically, applytype.py should have three simple steps: validate substitutions (upper bounds, values, argument kinds etc.); call expand_type(); update callable type variables (currently we just reduce the number, but in future we may also add variables there, see TODO that I added).
• The only valid positions for a variadic item (a.k.a. UnpackType) are inside Instances, TupleTypes, and CallableTypes. I like how there is an agreement that for callables there should never be a prefix, and instead prefix should be represented with regular positional arguments. I think that ideally we should enforce this with an assert in CallableType constructor (similar to how I did this for ParamSpec).
• Completing expand_type() should be a priority (since it describes basic semantics of TypeVarLikeTypes). I think I made good progress in this direction. IIUC the only valid substitution for *Ts are TupleType.items, *tuple[X, ...], Any, and <nothing>, so it was not hard.
• I propose to only allow TupleType (mostly for semanal.py, see item below), plain TypeVarTupleType, and a homogeneous tuple instances inside UnpackType. Supporting unions of those is not specified by the PEP and support will likely be quite tricky to implement. Also I propose to even eagerly expand type aliases to tuples (since there is no point in supporting recursive types like A = Tuple[int, *A]).
• I propose to forcefully flatten nested TupleTypes, there should be no things like Tuple[X1, *Tuple[X2, *Ts, Y2], Y1] etc after semantic analysis. (Similarly to how we always flatten Parameters for ParamSpec, and how we flatten nested unions in UnionType constructor). Currently we do the flattening/normalization of tuples in expand_type() etc.
• I suspect build_constraints_for_unpack() may be broken, at least when it was used for tuples and callables it did something wrong in few cases I tested (and there are other symptoms I mentioned in a TODO). I therefore re-implemented logic for callables/tuples using a separate dedicated helper. I will investigate more later.

As I mentioned above I only implemented strictly minimal amount of the above plan to make my tests pass, but still wanted to write this out to see if there are any objections (or maybe I don't understand something). If there are no objections to this plan, I will continue it in separate PR(s). Btw, I like how with this plan we will have clear logical parallels between TypeVarTuple implementation and (recently updated) ParamSpec implementation.