Parametrised Callable allowed only when used as type alias

[joelberkeley]

I've found a case where a Callable is not allowed when used directly, but is allowed if used as an alias

from typing import TypeVar, Callable

T = TypeVar('T')

MkFoo = Callable[[T], list[T]]

def foo(mkfoo: MkFoo) -> None:
    _ = mkfoo(0)

works, but replacing MkFoo by its definition

def foo(mkfoo: Callable[[T], list[T]]) -> None:
    _ = mkfoo(0)

produces

main.py:13: error: Argument 1 has incompatible type "int"; expected "T"

[srittau]

The problem is that T is not constrained in any way. According to the type hints, the function provided as argument could accept str:

def mk_str(s: str) -> list[str]:
    return [s]

foo(mk_str)

But this obviously wouldn't work with the implementation. The solution is to constrain the type var:

T = TypeVar("T", bound=int)

[joelberkeley]

Thanks for the quick reply.

I don't follow your mk_str example. Is that valid syntax?

T is not constrained in any way

Perhaps, but only in how mypy interprets it. When I say

def foo(mkfoo: Callable[[T], list[T]]) -> None:
    ...

I could either mean T to be bound at the level of foo, or at the level of mkfoo. It's ambiguous, at least in theory (mypy may have rules about generics that don't make it ambiguous). Perhaps it's reasonable to require an alias as a solution, but it's certainly not obvious. I ended up going via a Protocol with __call__ before I found this solution. I wonder if there's some way to make things less surprising for users.

[JelleZijlstra]

@srittau I think you edited @joelberkeley's post instead of replying? Looks a bit confusing.

Type variable scoping in cases like this indeed isn't well specified, and different type checkers interpret it differently.

[joelberkeley]

Thanks @JelleZijlstra . My posts still look fine to me. Github's being a bit awkward atm, I'm seeing " The content you are editing has changed. Please copy your edits and refresh the page. " might explain what you're seeing.

Interesting to hear about scoping specification. In Idris, they use forall t . t -> List t, and where you put the forall in the functions signature determines the intended scoping

[srittau]

Posting my reply here and restoring the original comment.

I don't follow your mk_str example. Is that valid syntax?

Sorry, fixed the example.

T is not constrained in any way

Perhaps, but only in how mypy interprets it. When I say

def foo(mkfoo: Callable[[T], list[T]]) -> None:
    ...

I could either mean T to be bound at the level of foo, or at the level of mkfoo. It's ambiguous, at least in theory (mypy may have rules about generics that don't make it ambiguous). Perhaps it's reasonable to require an alias as a solution, but it's certainly not obvious. I ended up going via a Protocol with __call__ before I found this solution. I wonder if there's some way to make things less surprising for users.

I agree that the effect of binding a type variable to an alias should be described in our documentation, which is unfortunately severely lacking at the moment.

[erictraut]

The reason the type alias doesn't produce errors in this case is because you are using a generic type alias (MkFoo) that accepts a single type argument, but you're not supplying a type argument, so a type checker will assume Any. That means you're effectively defining foo as:

def foo(mkfoo: Callable[[Any], list[Any]]) -> None: ...

That's probably not what you intended.

If you provide a type argument T, you will see that mypy produces the same error in both cases.

def foo(mkfoo: MkFoo[T]) -> None:
    _ = mkfoo(0)

@srittau, I don't understand why a TypeVar bound is required in this case. Regardless of whether T is scoped to foo or mkfoo, it can be solved. Pyright has no problem solving it without a bound.

[joelberkeley]

you are using a generic type alias (MkFoo) that accepts a single type argument, but you're not supplying a type argument, so a type checker will assume Any

🤦🏻‍♂️ of course, yes. So I guess a protocol is necessary (at least for mypy)

class MkFoo(Protocol):
    def __call__(self, x: T) -> list[T]:
        ...

which explicitly puts the T in the scope of mkfoo.

I don't understand how it would work if T is scoped to foo. Does T essentially become object then?

[srittau]

@erictraut I don't understand how this can be solved with an unconstrained T, assuming T is used as the context for foo() here:

T = TypeVar("T")

MkFoo: TypeAlias = Callable[[T], list[T]]

def foo(mkfoo: MkFoo[T]) -> None:
    mkfoo(0)

foo(lambda s: [s[0:4]])  # would crash at runtime

[erictraut]

It can't be solved if you pass a lambda because a lambda's parameter types must be inferred from context, and there isn't sufficient context available without a bound. If you pass mk_str as in your example above, I think there is sufficient type information to solve for T because mk_str is fully annotated.

[joelberkeley]

I think what @srittau is saying is that scoping T to foo would require the implementation to work for all T, which it clearly doesn't. That said, I'm not thinking very clearly atm so I'll leave this for tomorrow.

[erictraut]

Ah yes, that makes sense.