replicator icon indicating copy to clipboard operation
replicator copied to clipboard

Reader Error when trying to replicate to backup db

Open tomfriedhof opened this issue 6 months ago • 0 comments

Describe the bug I tried to replicate a new database after an initial application boot. The application sets up several streams on boot to populate general configuration for the application. After everything is initialized, and after I have verified the application is working as expected, I run the replicator to mirror the db to another EventStore instance. The replicator gets to 95% complete and then produces a read error:

{
  "@t": "2025-07-07T06:29:50.3107288Z",
  "@m": "Reader error",
  "@i": "450d20cc",
  "@l": "Error",
  "@x": "System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.\n   at EventStore.Replicator.Esdb.Grpc.Internals.StreamMetadataJsonConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /app/src/EventStore.Replicator.Esdb.Grpc/Internals/StreamMetadataJsonConverter.cs:line 96\n   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)\n   at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.MapMetadata(ResolvedEvent evt, Int32 sequence, Activity activity) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 142\n   at EventStore.Replicator.Esdb.Grpc.GrpcEventReader.ReadEvents(Position fromPosition, Func`2 next, CancellationToken cancellationToken) in /app/src/EventStore.Replicator.Esdb.Grpc/GrpcEventReader.cs:line 93\n   at EventStore.Replicator.Read.ReaderPipe.<>c__DisplayClass1_0.<<-ctor>g__Reader|1>d.MoveNext() in /app/src/EventStore.Replicator/Read/ReaderPipe.cs:line 41",
  "SourceContext": "EventStore.Replicator.Read.ReaderPipe"
}

To Reproduce Steps to reproduce the behavior:

  1. Boot up the application, and allow all setup streams to be appended to log. Verify application is working.
  2. Run the Replicator to copy db events to another instance
  3. Unable to finish replication due to a read error

Config/Logs/Screenshots The logs from the replicator prior to the error look like this. It seems to be something with reading the stream metadata:

repl-replicator|{
  "@t": "2025-07-07T06:29:55.3112993Z",
  "@m": "gRPC: Read event with id \"f54f1c07-2d93-48fa-91be-5a24ded58389\" of type \"$ProjectionCheckpoint\" from \"$projections-tenantA--ContactAttachmentIndex-checkpoint\" at \"C:555279090/P:555279090\"",
  "@i": "4138b59c",
  "@l": "Debug",
  "@tr": "ec5c6baab66310cbd0ab33a62bf45e3c",
  "@sp": "8a756f934965c9c5",
  "Id": "f54f1c07-2d93-48fa-91be-5a24ded58389",
  "Type": "$ProjectionCheckpoint",
  "Stream": "$projections-tenantA--ContactAttachmentIndex-checkpoint",
  "Position": "C:555279090/P:555279090",
  "SourceContext": "EventStore.Replicator.Esdb.Grpc.GrpcEventReader"
}
repl-replicator|{
  "@t": "2025-07-07T06:29:55.3114730Z",
  "@m": "gRPC: Read event with id \"e545277b-6539-4bda-af97-611283c863d5\" of type \"$metadata\" from \"$$tenantB/$tenantB/UserNotification\" at \"C:555279316/P:555279316\"",
  "@i": "4138b59c",
  "@l": "Debug",
  "@tr": "22a6c7db1fc30648fd1ca2c45aadd87e",
  "@sp": "d99737bb3477b12c",
  "Id": "e545277b-6539-4bda-af97-611283c863d5",
  "Type": "$metadata",
  "Stream": "$$tenantB/$tenantB/UserNotification",
  "Position": "C:555279316/P:555279316",
  "SourceContext": "EventStore.Replicator.Esdb.Grpc.GrpcEventReader"
}

EventStore details

  • EventStore server version: eventstore:latest docker image

  • Operating system: Linux

tomfriedhof avatar Jul 07 '25 17:07 tomfriedhof