database-stream-processor icon indicating copy to clipboard operation
database-stream-processor copied to clipboard
verified publisher icon vmware

Force pipeline termination from pipeline manager

Open ryzhyk opened this issue 2 years ago • 2 comments

The runner handles the /shutdown command by first calling controller.stop() and then sending a termination message to the actix dispatch loop. Both operations can be slow or even get stuck in some cases. We will need a way to enforce eventual termination from the pipeline manager by killing the process/pod after a timeout.

@lalithsuresh

ryzhyk avatar May 04 '23 07:05 ryzhyk

@ryzhyk I get the operations being slow, but do we know when/why they get stuck? Because that would be a bug.

lalithsuresh avatar May 04 '23 14:05 lalithsuresh

  • controller.stop() waits for any in-progress iteration of the circuit to complete. Which should normally be fast, but can take a while if there's a perf issue in the circuit or we're processing a very large input batch. Another possible reason is buggy connectors that refuse to shutdown instantly or at all.
  • actix is not supposed to take long, but I think I've seen it take a while to shutdown. Not sure what's up with that.

ryzhyk avatar May 04 '23 15:05 ryzhyk