redpanda-data
create temporal_workflow output
This output will trigger a Temporal workflow execution using Benthos messages to build the workflow execution request.
To see this in action, check out this proof of concept repo which contains a copy of this PR's code and instructions on how to run it:
https://github.com/disintegrator/benthos-temporal-poc
This looks like a great idea @disintegrator
rehydration on failed jobs in temporal will perhaps require sone intervention / handler at the benthos level .
If you get a chance let me know
wondeful stuff though as this makes benthos into such a more useful thing and is exactly what I needed for a project that I am working on for scientists.
Hey @gedw99, I'm fairly new to Temporal and not quite sure what you mean by rehydration. Can you explain the concept to me please?
From my experience writing this plugin, I think it'll be fairly straightforward to add more processors and outputs that can send signals, query workflows and so on...
Hey @gedw99, I'm fairly new to Temporal and not quite sure what you mean by rehydration. Can you explain the concept to me please?
From my experience writing this plugin, I think it'll be fairly straightforward to add more processors and outputs that can send signals, query workflows and so on...
Yeah sorry about the jargon. Temporal used to use this word in their docs but no longer do.
Basically when an activity ( not a larger workflow ) fails. It needs to decide what to do. The classic thing to do it retry ( with back off accumulating), and it needs to get all the values ( the outfits of the activities ) that it created along the way ( it’s part of a larger thing remember ) and then reuse them.
The logic is based on the fact that a workflow has many a to ours and any part of that chain can fail right. Who knows which part. So Temporal records the values that each activity created as well ( but I could not find it ) the inputs. So when it restarts it Troyes to not run everything again but only the failed activity. But its state is determined by state created by previous activities in that instance of a workflow. Christ this is complicated. So it’s tries to be smart if “re-hrdrate” the state and then can kick off the failed activity with its correct state .
That’s why it has a db . It checkpoints the state so that on failed it can pick up where it left off.
this is the denominator of Temporal compared to other workflow systems.
This is why I was asking if benthos needs to compensate or handle failures on an activity.
i guess we will find out when kicking the tires a bit.
Also you probably know that it can do side effects ( like an activity that sends an email ). If the workflow fails and then retries you definitely want this as you don’t want to send a 2nd email :) . Retries are explicitly configured , so it’s knows not to do it again .
I know this is not a complete answer . We will have to just be aware of these types of race conditions when testing this.
Magazin if this gets together it’s a really powerful feature.
I doubt I will get a chance to try this as holiday season and deadlines are kicking my arse …
@bhataprameya I’m on holiday this week and will get on to it when I’m back.
Hey, I'm not currently a Temporal user and I'm a little reluctant to move ahead with this PR. It'd be poor form to dump code I can't maintain on the project. I'm closing this PR at this time.
If you're interested in a Benthos <> Temporal integration, you can take the code from this PR and use it to build a custom Benthos binary. I actually built a custom binary with this plugin's code while working on the PR - you can check it out here. We often do this when we want to incubate some plugin before contributing it back to the project.