ExcludeSystemEvents should not filter out deleted streams

[riccardone]

Describe the bug SubscribeToAll with ExcludeSystemEvents filter I expect the deleted events to be still delivered

To Reproduce Steps to reproduce the behavior:

1. SubscribeToAll(FromAll.After(_lastPosition), cancellationToken: ctsToken, resolveLinkTos: false, filterOptions: new SubscriptionFilterOptions(EventTypeFilter.ExcludeSystemEvents()))
2. await foreach (var message in subscription.Messages)
3. deleted events are not delivered

Expected behavior in a catchupfrom all subscription using new SubscriptionFilterOptions(EventTypeFilter.ExcludeSystemEvents())) the deleted events should not be filtered. We should still deliver deleted events, as these are not system events and should not be filtered out until they are physically removed by a scavenge operation

Actual behavior in a catchupfrom all subscription using new SubscriptionFilterOptions(EventTypeFilter.ExcludeSystemEvents())) the deleted events are filtered

EventStore details

• KurrentDb server version: 25.0.0.1907

• Operating system: windows

• EventStore client version (if applicable): KurrentDB.Client 1.0.0

[w1am]

Hey @riccardone,

Thanks for creating the issue. That’s already the case. ExcludeSystemEvents only excludes events that begin with "$". I suspect that your start position is after the deleted events, which is why you're not seeing them.

[riccardone]

I guess there is something not working as expected as I can't get any deleted event at the breakpoint. You can run the same or similar code to reproduce. The position doesn't seems to be relevant as the issue happen after the liveprocessing has started and code is waiting for new events. I wonder if there could be an issue that is not considering that the deleted stream start with $$ but the event in it start with $ and therefore gets excluded

[w1am]

I couldn't reproduce it locally. Can you try it without the filter to see if the events you wanted to see don't have a $ at the beginning?

[riccardone]

deleted events they have $metadata as event type and therefore they are excluded by the ExcludeSystemEvents method using this regex
when a stream is deleted, in EventStore a stream with the same name but prefixed with $$ is created. The event in it has the word '$metadata' as EventType and therefore the simple regex above is blocking it. So it's a matter of decide to keep it as is but misleading the developer (as the deleted stream/event are NOT SystemEvents) and make it clear in the documentation. The developer wanting to deliver delete stream/events need to not use ExcludeSystemEvents and instead write a slightly more complicated regex and logic to filter out these particular event when the stream start with $$. The alternative is doing this fix in the ExcludeSystemEvents method.

Not sure why you can't reproduce it. Just subscribe fromall/start, write an event, set a breakpoint in the subscription, delete the stream to confirm the breakpoint is not hit

Can you try it without the filter to see if the events you wanted to see don't have a $ at the beginning? yes the deleted events are delivered without using ExcludeSystemEvents. These events have only one '$' special char as described above while the stream name has '$$'

[YoEight]

Hey @riccardone,

I understand there’s some confusion about what happens when deleting a stream. In your case, it’s a soft delete. Internally, this updates the stream’s metadata by setting its truncateBefore property to the stream’s revision at the time of the soft deletion.

The only time you’ll see a stream deleted event ($streamDeleted) is during a hard delete, also called a tombstone. This event is a system event that gets written when the stream is permanently deleted.

From our conversation, I don’t see a bug—just a misunderstanding of what the documentation describes. I’d appreciate hearing more of your thoughts so we can improve the documentation and clarify how stream deletion works.

[riccardone]

Ok, as far as I understand the deletion of a stream (soft or hard) will cause some events in eventstore, right? do you consider these events system events?

The ExcludeSystemEvents regex to do what it says on the tin should be at least: EventTypeFilter.RegularExpression(@"^(\$metadata|[^\$].*)")) this will let the hard or soft delete stream/events to be delivered to the subscription

[YoEight]

The ExcludeSystemEvents regex to do what it says on the tin should be at least: EventTypeFilter.RegularExpression(@"^($metadata|[^$].*)")) this will let the hard or soft delete stream/events to be delivered to the subscription

No, because everything you’re referring to is system-related. I don’t understand why you don’t simply provide a filter that matches your actual requirements instead. The ExcludeSystemEvents option is working as intended.

[riccardone]

in case you consider deleted events as user events, the ExcludeSystemEvents is not actually working as intended as it filters them out. In order to do what I need I had to use the eventtype filter I suggested before combined with a streamid filter allowing streams starting with double $$ (but not $ or $$$). It would have be nicer to have a combined filter built in that does that. That's why I started this issue. If that is not something possible then at least document that ExcludeSystemEvents specifying that it filter out also deleted events (so a developer like me doing replicas does not spend time debugging it). Thans

[alexeyzimarev]

Any event with the event type starting with the dollar sign $ (one, two, three - doesn't matter) is a system event. The ExcludeSystemEvents filter would filter them out. However, deleted events are just those events you put there and then deleted by truncating the stream where those events were appended to.

• Deletion event is actually the stream metadata event that goes to the stream metadata stream (sorry a bit awkward but it's how it is). And those are system events.
• Deleted events are those that should be scavenged and the server doesn't return them after the stream got deleted or truncated. Those stay in the log and get returned when you read from $all until you scavenge them.

So, you will be able to read deleted events from $all until they get scavenged. Maybe you can clarify what do you actually trying to do.

[alexeyzimarev]

In addition, ExcludeSystemEvents filter works as intended as it filters out all events where the event type starts with $, $$, $$$ and so on are system events.

[riccardone]

Replicate events from one origin to one destination and be able to create a replica/copy of origin. Using ExcludeSystemEvents and subscribe from $all the deleted events get filtered out.

To workaround the issue I had to use a regex like:

new SubscriptionFilterOptions(EventTypeFilter.RegularExpression(@"^(\$metadata|[^\$].*)"))

combined with a streamid filter like:

if (eventStreamId.StartsWith("$$$") || (eventStreamId.StartsWith('$') && !eventStreamId.StartsWith("$$")))
    return false;

Again, the use case is to be able to replicate all user data (including stream deletions that have not yet been scavenged) from one instance to another. I have achieved this by combining an event type filter with a stream ID filter. Using the ExcludeSystemEvents option is not appropriate for this use case, as it simply filters out anything starting with $

[alexeyzimarev]

We have a replicator readymade for this, have you tried it?

[riccardone]

I didn't know you have done a Replicator. We tried it and it works even if it is a bit heavier than my Linker. We will run it for a while and compare between the two tools to see which one to use. Thanks