Type checking for *args and **kwargs when passing them to another function

[t1m013y]

Problem:

I am writing a python application, and I made a function that creates an object and passes its additional *args and **kwargs to object's constructor and takes few its own arguments. I want to make *args and **kwargs typed exactly as inner function's arguments.

Description and examples:

A simple example with two functions:

def inner(a: int, b: int) -> None:
    ...


def wrapper(arg1: str, *args, **kwargs) -> None:
    inner(*args, **kwargs)  # It will automatically detect passing arguments to another function

In my case, inner is object constructor, I gave an example with functions to make it more simple.

Type checkers should see wrapper's signature like this:

def wrapper(arg1: str, a: int, b: int):
    ...

It may be enabled by default when passing *args and **kwargs to another function is detected or it may be enabled by adding a decorator to the function.

Detailed description:

Already suggested this to mypy, detailed description of this feature can be found here: python/mypy#19302.

[erictraut]

Here's how that would typically be done.

from typing import Callable, Concatenate

def inner(a: int, b: int) -> None: ...

def create_wrapper[**P, R](fn: Callable[P, R]) -> Callable[Concatenate[str, P], R]:
    def wrapper(arg1: str, *args: P.args, **kwargs: P.kwargs) -> R:
        return fn(*args, **kwargs)

    return wrapper

wrapper = create_wrapper(inner)

[t1m013y]

Here's how that would typically be done.

from typing import Callable, Concatenate

def inner(a: int, b: int) -> None: ...

def create_wrapper[**P, R](fn: Callable[P, R]) -> Callable[Concatenate[str, *P], R]: def wrapper(arg1: str, *args: P.args, **kwargs: P.kwargs) -> R: return fn(*args, **kwargs)

return wrapper

wrapper = create_wrapper(inner)

It's too difficult, need to create a single-use decorator with complex typing. My feature is about an easy way to do this (automatically detected passing arguments to another function or by just adding a decorator to a wrapper).

[t1m013y]

Here's how that would typically be done.

from typing import Callable, Concatenate

def inner(a: int, b: int) -> None: ...

def create_wrapper[**P, R](fn: Callable[P, R]) -> Callable[Concatenate[str, P], R]: def wrapper(arg1: str, *args: P.args, **kwargs: P.kwargs) -> R: return fn(*args, **kwargs)

return wrapper

wrapper = create_wrapper(inner)

Please see python/mypy#19302 for detailed information. It would be much more easy, flexible and readable if this feature existed in the form in which I suggested it (auto-detect or adding a decorator).

[wyattscarpenter]

I agree completely. Specifying these types is far too difficult for end users and the typechecker should deduce them, instead. It's a common pattern that should be supported automatically.

[t1m013y]

I agree completely. Specifying these types is far too difficult for end users and the typechecker should deduce them, instead. It's a common pattern that should be supported automatically.

Additionally, decorators were initially created to modify behavior of many functions, not for single-use like the way it's typically done for passing *args and **kwargs.

[wyattscarpenter]

I wonder if a not-single-use decorator to do this could be defined with the current typing rules. Something like

def inner(a: int, b: int) -> None:
    ...

@pass_through(inner)
def wrapper(arg1: str, *args, **kwargs) -> None:
    inner(*args, **kwargs)

I took at stab at this but wasn't really able to get anywhere, despite summiting the trippy levels of recursive indirection involved. I don't think the current typing spec has suitable machinery to manipulate type signatures in this way.

[wyattscarpenter]

If anyone comes here looking for a workaround, I did figure out this similar technique, using our old friend TypedDict and PEP 692 – Using TypedDict for more precise **kwargs typing. Doesn't accomplish quite the desired thing, but it does single-source the type signature of the kwargs. In a TypedDict.

from typing import TypedDict, Unpack, reveal_type

InnerKwargs = TypedDict('InnerKwargs', {'x': int, 'y': int})

def inner(**kwargs: Unpack[InnerKwargs]) -> None:
    ...

def wrapper(arg1: str, **kwargs: Unpack[InnerKwargs]) -> None:
    inner(**kwargs)

reveal_type(wrapper) # Type of "wrapper" is "(arg1: str, **kwargs: **InnerKwargs) -> None"

wrapper("high") #Complains about missing arguments unless you mark the Point2D as total=False
wrapper("high", x=1, y=2, z=4) #Complains about No parameter z even if closed=False

Of course, one of the biggest reasons to want to pass through kwargs (also also args) is because the inner function is something you don't control, as described in https://github.com/python/typing/discussions/1501 — this solution doesn't fix that, since you'd have to change the type signature of the inner function to use a TypedDict kwargs (or, you'd have to duplicate its arguments in your kwargs TypedDict, anyway).

[t1m013y]

I wonder if a not-single-use decorator to do this could be defined with the current typing rules. Something like

def inner(a: int, b: int) -> None: ...

@pass_through(inner) def wrapper(arg1: str, *args, **kwargs) -> None: inner(*args, **kwargs)

I took at stab at this but wasn't really able to get anywhere, despite summiting the trippy levels of recursive indirection involved. I don't think the current typing spec has suitable machinery to manipulate type signatures in this way.

Yes, I have already described the similar decorator in python/mypy#19302, but with few additional options (this is the decorator factory):

def pass_args(func: collections.abc.Callable | None, pass_args: bool = False, pass_kwargs: bool = False):
    ...

# Usage:
@pass_args(inner, pass_args=True, pass_kwargs=True)
def wrapper(arg1: int, *args, **kwargs):
   ...
   inner(*args, **kwargs)
   ...

If anyone comes here looking for a workaround, I did figure out this similar technique, using our old friend TypedDict and PEP 692 – Using TypedDict for more precise **kwargs typing. Doesn't accomplish quite the desired thing, but it does single-source the type signature of the kwargs. In a TypedDict.

from typing import TypedDict, Unpack, reveal_type

InnerKwargs = TypedDict('InnerKwargs', {'x': int, 'y': int})

def inner(**kwargs: Unpack[InnerKwargs]) -> None: ...

def wrapper(arg1: str, **kwargs: Unpack[InnerKwargs]) -> None: inner(**kwargs)

reveal_type(wrapper) # Type of "wrapper" is "(arg1: str, **kwargs: **InnerKwargs) -> None"

wrapper("high") #Complains about missing arguments unless you mark the Point2D as total=False wrapper("high", x=1, y=2, z=4) #Complains about No parameter z even if closed=False

Of course, one of the biggest reasons to want to pass through kwargs (also also args) is because the inner function is something you don't control, as described in #1501 — this solution doesn't fix that, since you'd have to change the type signature of the inner function to use a TypedDict kwargs (or, you'd have to duplicate its arguments in your kwargs TypedDict, anyway).

I originally needed this for the library function, so it isn't the solution.

[t1m013y]

In the original message in python/mypy#19302 I mentioned the decorator factory which modifies argument passing behavior syntax like this:

def pass_args(func: collections.abc.Callable | None, pass_args: bool = False, pass_kwargs: bool = False):
    ...

Now I think it would be more practical to make pass_args and pass_kwargs True by default.

[t1m013y]

Also I think it would be useful to add an option to pass func argument to the decorator both function itself and as a type (would be useful for typing another decorators) (see python/mypy#19302)