TypeVars for implicit lambdas can be tricked based on a single Callable

[max-muoto]

from collections.abc import Callable
from typing import TypeVar

T = TypeVar("T", bool, float)

class MyClass:
    def map(self, func: Callable[[float], float] | Callable[[bytes], bytes]): ...

def foo(x: T) -> T: ...

MyClass().map(lambda x: foo(x))

Here, despite T being limited to either bool | float, we're possibly to pass in bytes. It seems only one of the unioned callables (regardless of the order) needs to match to trick Pyright into accepting this.

Pyright Playground

Note: I think this has been an issue for a while, not a regression with the improved union callable

Versoin: 1.1.363

### Tasks

[max-muoto]

@erictraut Unrelated, and can create a separate issue for this, but Pylance uses the first callable for the type, even though I believe it's actually the union:

[erictraut]

I don't see any bug here. Pyright doesn't report any errors here, and I think that's correct. Mypy likewise doesn't report any errors.

Let's remove the TypeVar from the equation because I think it's needlessly complicating things. Instead, let's assume that foo is defined as simply: def foo(x: float) -> float. That would make lambda x: foo(x) compatible with Callable[[float], float] which makes it compatible with the func parameter type in the MyClass.map method.

Incidentally, the constrained TypeVar you've defined looks suspect to me. One of the constraints (bool) is a subtype of the other constraint (float). The typing spec is not clear on what should happen in this case. This issue is on a long list of issues that I'd like to have clarified in the typing spec. My preference is to make it illegal to specify overlapping constraints like this, but we'll need to see what the consensus of the community is. In any case, I recommend against defining constrained TypeVars with overlapping constraint types.

[max-muoto]

I don't see any bug here. Pyright doesn't report any errors here, and I think that's correct. Mypy likewise doesn't report any errors.

Let's remove the TypeVar from the equation because I think it's needlessly complicating things. Instead, let's assume that foo is defined as simply: def foo(x: float) -> float. That would make lambda x: foo(x) compatible with Callable[[float], float] which makes it compatible with the func parameter type in the MyClass.map method.

Incidentally, the constrained TypeVar you've defined looks suspect to me. One of the constraints (bool) is a subtype of the other constraint (float). The typing spec is not clear on what should happen in this case. This issue is on a long list of issues that I'd like to have clarified in the typing spec. My preference is to make it illegal to specify overlapping constraints like this, but we'll need to see what the consensus of the community is. In any case, I recommend against defining constrained TypeVars with overlapping constraint types.

Thanks for the clarification! Without any subtypes, or generics, I assumed this should cause issues though, no?

from collections.abc import Callable

class Container: ...


class MyClass:
    def map(self, func: Callable[[str], str] | Callable[[Container], Container]): ...


def foo(x: Container) -> Container: ...


MyClass().map(lambda x: foo(x))

Playground Link

[erictraut]

No, why would that generate an error? The type of the lambda expression in this example is (x: Decimal) -> Decimal which is compatible with the annotated type of the func parameter. There's no type violation here.

[max-muoto]

No, why would that generate an error? The type of the lambda expression in this example is (x: Decimal) -> Decimal which is compatible with the annotated type of the func parameter. There's no type violation here.

Sorry - as I modified it a bit. I assumed the type of x in the lambda would be str | Container which would be incompatible with Container, it seems I am wrong in this assumption though. Thanks again here!

[max-muoto]

For my personal understanding, is there a reason this example no longer works in 1.1.363 though? https://pyright-play.net/?pyrightVersion=1.1.363&pythonVersion=3.12&strict=true&code=GYJw9gtgBAxmA28CmMAuBLMA7AzgOgEMAjGKdCABzBFSgGEDFjkAoUSKVATwvSwHMylarQCCWLiykx4BHDigBZLnVnyAXCyjaoAEyTAoEAhQAUOJPGAAaKMACuWGOvqNZRZAG1POVCAC6tr4BUAA%2BrkweSN7iXIFQwf4AlC54aVIs%2BobAYGCmAB4ufKhJUAC0AHwJfqnpLMqqcjimSXjGZrIQRLoEUIV2uQVJSUA

[max-muoto]

I had a case like this, in my codebase, where previously the Any would get matched to the int in foo.

[erictraut]

That code didn't type check in previous versions of pyright either. You can use the version control in the pyright playground to confirm.

Evaluation of lambdas requires bidirectional type inference. In cases where the "expected type" consists of a union, pyright attempts to filter the subtypes of the union to remove any non-callalble types. If there are multiple callable types, it picks one of them to use for bidirectional type inference. To make this deterministic, it uses an internal sorting mechanism to sort the candidates. The sorting is somewhat arbitrary, since there is no well-defined order for types; the goal is to simply make it deterministic so union ordering doesn't change type checking behaviors. Perhaps you're recalling some other union of callables where the callable with the Any parameter happened to be sorted first. In any case, I wouldn't recommend creating a union of Callable[[str], str] and Callable[[Any], str]. Since the former is a subtype of the latter, you should simply use Callable[[Any], str]. Creating a union serves only to add complexity and confuse the type checker.

[max-muoto]

That code didn't type check in previous versions of pyright either. You can use the version control in the pyright playground to confirm.

Evaluation of lambdas requires bidirectional type inference. In cases where the "expected type" consists of a union, pyright attempts to filter the subtypes of the union to remove any non-callalble types. If there are multiple callable types, it picks one of them to use for bidirectional type inference. To make this deterministic, it uses an internal sorting mechanism to sort the candidates. The sorting is somewhat arbitrary, since there is no well-defined order for types; the goal is to simply make it deterministic so union ordering doesn't change type checking behaviors. Perhaps you're recalling some other union of callables where the callable with the Any parameter happened to be sorted first. In any case, I wouldn't recommend creating a union of Callable[[str], str] and Callable[[Any], str]. Since the former is a subtype of the latter, you should simply use Callable[[Any], str]. Creating a union serves only to add complexity and confuse the type checker.

Did you try on 1.1.360, or is there something wrong here?

[erictraut]

Did you try on 1.1.360

No, I went back only two versions. It looks like something changed between 1.1.360 and 1.1.361. I don't consider it a bug though, for the reasons I explained above.

[max-muoto]

Did you try on 1.1.360

No, I went back only two versions. It looks like something changed between 1.1.360 and 1.1.361. I don't consider it a bug though, for the reasons I explained above.

-Sounds good! It's from a third party library so will just put up a PR tweaking it based on this, thanks.