[WIP] PEP 484: Describe a way of annotating decorated declarations by sixolet · Pull Request #242 · python/peps (original) (raw)

Hey! Thanks for your work on this!

I have kind of an interesting case that's coming up because of mypy tripping on decorators. Manual annotations could help cases like this, but it might be broader issue. This seems like the best place to comment on it though -- please redirect me if I should ask elsewhere.

I've built a stackable decorator (kinda like a monoid) for doing some algebraic-ish function composition with additional safety checks along the way. It's pretty neat, if I do say so myself. Here's a minimal repro (you can extrapolate how this can be super useful!). It's relatively complicated, so I've broken it down step by step afterwards.

from future import annotations from typing import Generic, Union, Callable, TypeVar

T = TypeVar("T")

class BuildCombinator(Generic[T]): def init(self, x: Callable[[T], int]) -> None: self.x = x

@staticmethod
def build(x: Callable[[T], int]) -> BuildCombinator[T]:
    return BuildCombinator[T](x)

def __call__(
    self, func: Union[BaseCombinator[T], BuildCombinator[T,]]
) -> Union[BaseCombinator[T], BuildCombinator[T]]:
    if isinstance(func, BaseCombinator):
        bc : BaseCombinator[T] = func
        return BaseCombinator[T](lambda t: bc.n(t) * self.x(t), bc.f)
    elif isinstance(func, BuildCombinator):
        blc: BuildCombinator[T] = func
        return BuildCombinator[T](lambda t: blc.x(t) * self.x(t))
    else:
        assert False
def do_thing(self, obj:T) -> str:
    return str(self.x(obj))

class BaseCombinator(Generic[T]): def init(self, n: Callable[[T], int], f: Callable[[T, str], str]) -> None: self.n = n self.f = f

@staticmethod
def base(x: Callable[[T, str], str]) -> BaseCombinator[T]:
    return BaseCombinator[T](lambda t: 1, x)

def do_thing(self, obj:T) -> str:
    return self.f(obj, str(self.n(obj)))

class A: @BuildCombinator.build def mul_3(self) -> int: return 3

@BaseCombinator.base
def one(self, s: str) -> str:
    return s

@mul_3
@BaseCombinator.base
def three(self, s: str) -> str:
    return s

@mul_3
@mul_3
@BaseCombinator.base
def nine(self, s: str) -> str:
    return s

@BuildCombinator.build
def mul_5(self) -> int:
    return 5

@mul_3
@mul_5
@BuildCombinator.build
def mul_15(self) -> int:
    return 1

@BuildCombinator.build
def mul_7(self) -> int:
    return 7

@mul_15
@BaseCombinator.base
def broken(self, s: str) -> str:
    return s

@mul_5
@mul_3
@BaseCombinator.base
def works(self, s: str) -> str:
    return s

if name == "main": a = A() print(a.one.do_thing(a), 1) print(a.three.do_thing(a), 3) print(a.nine.do_thing(a), 9) print(a.broken.do_thing(a), 15) print(a.works.do_thing(a), 15)

This prints out what you'd expect. But mypy complains with:

example.py:77: error: Untyped decorator makes function "broken" untyped
example.py:77: error: "BaseCombinator[A]" not callable
Found 2 errors in 1 file (checked 1 source file)

Both of the compositions should be yielding the exact same compositions, but the types don't propagate.

Interestingly when I use typing.get_type_hints on a.broken.do_thing and a.works.do_thing, I get {'obj': ~T, 'return': <class 'str'>} for both. So the types should be there for calling!

That's kinda complex, so I can quickly walk through what the code is doing. Essentially I have a base case function template, that is like this:

class A:
  def x(self) -> int: return 0

Anything that matches that template can be decorated like so:

class A:
  @build
  def x(self) -> int: return 0

build transforms x into a a "layer" of computation that operates on "bases", e.g.:

class A:
  @build
  def x(self) -> int: return 0
  @build
  def y(self) -> int: return 1
  @x
  @base 
   def base_example(self, s:str) -> str: return s

  @y 
  @x 
  @base 
   def base_example2(self, s:str) -> str: return s 

So far, so good. Code compiles and works and type checks, beautifully might I add. I'm a very happy user! But it gets a little bit harry and I get some mysterious type error.

I have a few different "base" combinators that have different semantics, e.g. you could imagine one that uses the @ build combinators to pass arguments, or one that uses them to log debugging info. The important thing is that these base cases are typed, and are aware of the type of the class A that they are inside of.

class BuildCombinator(Generic[T]):
    @staticmethod
    def  build(Callable[[T], int]) -> BuildCombinator[T]: ....
    def __call__(self, func: BaseCombinator[T]) -> BaseCombinator[T]: ...
class BaseCombinator(Generic[T]):
     @staticmethod
    def  base(Callable[[T, str], ]) -> BaseCominator[T]: ....

Now, the catch is that the BuildCombinator itself is allowed to combine on itself recursively. Once it's "in the monad" you can keep on stacking new effects. I've used multiplication because prime factoring is helpful for thinking about it, but the effects can be whatever.

class BuildCombinator(Generic[T]):
    @staticmethod
    def  build(Callable[[T], int]) -> BuildCombinator[T]: ....
    def __call__(self, func: Union[BaseCombinator[T], BuildCombinator[T,]]) -> Union[BaseCombinator[T], BuildCombinator[T]]:
        if isinstance(func, BaseCombinator): return BaseCombinator[T](...)
        if isinstance(fund, BuildCombinator): return BuildCombinator[T](...)
class BaseCombinator(Generic[T]):
     @staticmethod
    def  base(Callable[[T, str], ]) -> BaseCominator[T]: ....

This is where the issues come in. When you stack directly on a function from a base case it's fine. E.g.,

@mul_7
@mul_3
@base 
def twentyone(self, s: str) -> str: return s

but when you try to stack on a build and then apply it to a base, the type checker gets sad.

@mul_7
@mul_3
@build 
def mul_21(self): return 1

@mul_21
@base 
def twentyone(self, s: str) -> str: return s

But the program still seems to be actually correct, it's just the type checker.