epirust
epirust copied to clipboard
Engines pick up stale simulation request
Steps to reproduce:
- Run a simulation in multi-engine mode with, say, 2 engines.
- Let the simulation finish. Orchestrator and engines will terminate.
- Start another multi-engine simulation with 3 or 4 engines (> 2)
- The new engines will pick up the old simulation request and complain that no work was found for engines3 and 4.
Cause: Since we're adding new consumers (engines), Kafka delivers the older messages to the engines.
Possible fix: Let the orchestrator delete the simulation requests after the simulation finishes.
Small-term fix: add fromOffset: latest in the consumer config.
Calling this small term-fix as it is now creating multiple records of consumer vs offset on the kafka-server for each run, which is not the desired behaviour.