temporal icon indicating copy to clipboard operation
temporal copied to clipboard

History service: use a separate context when IO

Open wxing1292 opened this issue 3 years ago • 1 comments

Is your feature request related to a problem? Please describe. History shard will override the context to ensure minimal timeout, however, this checking / overriding logic does not prevent caller from cancelling the call, potentially causing shard instability (assert shard whenever see non shard ownership lost error)

thanks @nagl-temporal for finding this potential issue

wxing1292 avatar Aug 03 '22 17:08 wxing1292

To repro: Insert a short (I used 10ms) sleep in UpdateWorkflowExecution to increase the chances that context cancellation occurs during the persistence operation. start a reasonably chatty (parallelism=4) maru workload against temporal running locally. Ctrl-c your bench-go worker - you should see rangeID updates in the log from reliable-shard. You may have to do this a few times to get unlucky (or increase the sleep).

It's weird that a badly behaved client can trigger rangeID updates.

nagl-temporal avatar Aug 03 '22 18:08 nagl-temporal