relay icon indicating copy to clipboard operation
relay copied to clipboard

feat(spooler): Add cross-stacks spooling

Open iambriccardo opened this issue 1 year ago • 0 comments

This PR changes the EnvelopeBuffer in the following ways:

The buffer now holds only a priority queue of project key pairs from which we want to pop envelopes. The buffer has an EnvelopeProvider that contains a map of project key pairs to an array of envelopes (treated as a stack). Spooling occurs when the total count of in-memory envelopes across stacks is reached. Spooling now happens when x envelopes are reached, and exactly x envelopes will be spooled. Unspooling works differently in that we now load only one envelope from disk if needed. During a pop, we load from disk and immediately return; during a peek, we load from disk and add to the in-memory buffer. These changes aim to address the issues with the spooler in our production tests.

Re-apply of: https://github.com/getsentry/relay/pull/4107 Closes: https://github.com/getsentry/team-ingest/issues/549

#skip-changelog

iambriccardo avatar Oct 10 '24 06:10 iambriccardo