Regression: System.IndexOutOfRangeException since upgrading EventStore library

[Skyppid]

Describe the bug After upgrading from 23.2.1 to 23.3.5 we suddenly experienced thousands of errors on our service. Basically every message that is attempted to be written to EventStore failed to do so resulting in a lot of data loss.

I tried figuring out what the cause would be since the code wasn't touched at all. Sentry shows that the issue stems from library internal methods so likely something broke with that.

Since EventStore unfortunately does not provide any symbols I cannot provide more information. Maybe you want to offer these for future releases to make things easier for all of us?

To Reproduce Steps to reproduce the behavior:

• Upgrade the library from 23.2.1 to 23.3.5

This is all the code we use to store the event. Worked like a charm ever since we developed our service months ago. Only broke after the package upgrade:

public async Task StoreEventAsync(ChangeEventMessage @event, CancellationToken cancellationToken)
{
    await client.AppendToStreamAsync(@event.SourceEntityId.ToString(), StreamState.Any,
        [new EventData(Uuid.NewUuid(), nameof(ChangeEventMessage), JsonSerializer.SerializeToUtf8Bytes(@event, jsonOptions.Value.JsonSerializerOptions))], null, null, null, cancellationToken);
}

Expected behavior No exception caused by the event store library itself

Actual behavior IndexOutOfRangeException in a context that does not make sense outside the library.

Config/Logs/Screenshots See screenshot above

EventStore details

• EventStore server version: 24.2.0

• Operating system: Linux

• EventStore client version (if applicable): 23.3.5

Additional context I cannot provide an isolated project to reproduce this issue as of now. Interestingly this error occurs on both staging and production clusters but I cannot reproduce it locally after importing a backup of the production event store. Chances are that this might be an issue because on clusters it's running on Linux while I am developing on Windows.

Maybe you can identify the possible cause by looking at the diffs, something must have changed during these versions...

EDIT: I can verify after downgrading to 23.2.1 it's working again. Not the first time EventStore library upgrades caused issues. Please consider working on your reliability. This should really not happen for enterprise level software.