DiceDB
Stable support of persistency
[!NOTE] This issue is meant to be resolved after stable implementation of multiple threading architecture as per community’s decision (ref)
currently dumping data to aof file during graceful shutdown which will make shutdown and restart process slower in case data load is high
so we should add support to take snapshot of data in aof file in particular interval to ensure data durability and to allow for recovery in case of system failures.
key points of why redis also takes snapshot:
- In case of a server crash or power failure, Redis can recover its data from the most recent snapshot.
- snapshots serve as point-in-time backups.
- can be used to transfer data between instances or for creating replicas.
- in some configurations, Redis can use snapshots to free up memory by saving data to disk.
in redis config it looks like this:
save 900 1 # Save after 900 seconds (15 minutes) if at least 1 key changed
save 300 10 # Save after 300 seconds (5 minutes) if at least 10 keys changed
save 60 10000 # Save after 60 seconds if at least 10,000 keys changed
in our config it will look like this:
# dice.toml
[[Server.SnapshotInterval]]
Interval = 900000000000
KeysChanged = 1
[[Server.SnapshotInterval]]
Interval = 300000000000
KeysChanged = 10
[[Server.SnapshotInterval]]
Interval = 60000000000
KeysChanged = 10000
@vinitparekh17 looks interesting. And as you noted we can take this up after the stable implementation of multithreading.
Key points to remember for us is how do we proceed with the snapshots across shards? Shall we maintain separate file per shard or combine in a single file? For both of these approaches we need to come with a optimised logic and if we want to support both options the configuration file parameters should be in place.
Great way to start thinking about it. Thanks!
Also we should take @arpitbbhayani @JyotinderSingh thoughts into consideration regarding persistence support based on the priorities as well.
