dramatiq
dramatiq copied to clipboard
Feature request: pass arguments automatically and individually in pipelines
Feature request
A form of pipeline that automatically passes its first input argument on to the next actor, along with any additional arguments as individual args.
@dramatiq.actor()
def actor_1(list_of_numbers: List[Int]):
other_numbers = [5, 6, 7]
return other_numbers # -> Notice how we don't return 'list_of_numbers'
# -> Arguments are received in order, no need to access indices in a tuple
@dramatiq.actor()
def actor_2(list_of_numbers: List[Int], other_numbers: List[int]):
return list_of_numbers.extends(other_numbers)
pipeline([actor_1.message([1,2,3]), actor_2.message()]
If my understanding is correct, I would have to do the following to pass multiple args in dramatiq and maintain type hints:
@dramatiq.actor()
def actor_1(list_of_numbers: List[Int]):
other_numbers = [5, 6, 7]
return list_of_numbers, other_numbers
@dramatiq.actor()
def actor_2(data): # -> No introspection or typing for function arguments :(
list_of_numbers: List[int] = data[0]
other_numbers: List[int] = data[1]
new_list = list_of_numbers.extends(other_numbers)
return list_of_numbers, new_list # -> Easy to confuse the order
pipeline([actor_1.message([1,2,3]), actor_2.message()]
Benefits
- We don't have to keep passing 'list_of_numbers' manually through all the pipeline stages
- Feels a lot cleaner than manually accessing indices of a returned tuple
- Code introspection can tell us how many arguments an actor expects as well as the type of them.
Issues
Breaking change since it fundamentally alters how arguments are passed between actors in a pipeline
Ideal solution
Create a new composition type 'streamlined_pipeline' with this behavior or allow a keyword argument to be passed to 'pipeline' that modifies the behavior, i.e:
pipeline(messages, pass_args=True)