FluidFramework icon indicating copy to clipboard operation
FluidFramework copied to clipboard

Offline: Provide incoming batch's starting CSN to the PendingStateManager

Open markfields opened this issue 1 year ago • 0 comments

Description

Fixes AB#8544

On submit, the PendingStateManager stores the batch's starting CSN along with each message. We need the corresponding information for incoming messages too, for two reasons:

  1. We can assert that they match as we process incoming acks for local messages
  2. We need it to compute batch ID, when checking for the same batch coming from a forked container.

Today, two things happening in RemoteMessageProcessor keep the correct info from flowing to the PSM:

  1. Grouped Batches overwrite clientSequenceNumber with the index into the batch
  2. Messages from a compressed batches are processed one-by-one, so the context of the start of the batch is not available when later messages are sent to PSM.

Here's the fix - the RemoteMessageProcessor has the full context and can return the batch's starting CSN with each message, irrespective of grouped batching or compression.

Reviewer Guidance

These changes are mostly internal to OpDecompressor, plus the signature changes to plumb the CSN down to PSM. To prove correctness, we add an error log in PSM if the batch start CSNs don't match.

Future consideration

A more comprehensive approach which I like, but is more disruptive, would be to have the OpDecompressor act like the OpSplitter, and hold onto the batch messages until the last one arrives, and only then release the batch to be returned for processing. This would make it more obvious that batches are processed all at once, and when processing the batch other parts of the system could trivially get at the start of the batch (for CSN and in the future the batch ID).

markfields avatar Jun 28 '24 23:06 markfields