SEMICeu
Latest state + time retention
The idea of this retention policy is that it combines the benefits of snapshots with retention. This way, if replication of state is the concern of the client, a log that is as small as possible can be offered. This saves resources for both publisher and subscriber.
The idea of the algorithm is explained here: https://docs.confluent.io/kafka/design/log_compaction.html
What Sander wants to do is say something like: I want to keep 1 hour of data, for which I also will only keep the latest version. Today it’s only possible to say you will keep all latest versions until the end of time, and you will keep all data for the last 1 hour.
We need a new design for retention policies to allow for the first use case.
