rust-ceramic icon indicating copy to clipboard operation
rust-ceramic copied to clipboard

feat: add process to track and run data migrations

Open dav1do opened this issue 1 year ago • 2 comments

The upcoming changes to normalize some of event payload requires running a data migration a la nosql style. This provides tables to track the daemon version and the state of migrations. They can be run by calling the daemon with the --migrate-data flag. If there are required migrations, the daemon will fail to start.

dav1do avatar Jun 14 '24 00:06 dav1do

So I realized the migrations to extract metadata isn't as urgent as I thought (I fixed IOD to not have bugs without it, and it might be more efficient tbh - will have the cleaned up PR today, still need to split it/make it easier to review). I do think we want to do it before GA, but I don't think I have the new table structures schema quite right, so it might make sense to pause this for a couple days. This doesn't really do much, and it doesn't hurt to merge this and store version info, but we may want to track started_at or other data (so we can see if you downgrade in history). And we may want to support running migrations in the background (slowly read events and extract metadata but maybe it's better to just rip through everything for now), which would mean the flag might not be quite right.

Anyway, @stbrody, curious to hear your thoughts/discuss briefly before you spend any more time on this (and I really appreciate how much you've already spent).

dav1do avatar Jun 17 '24 14:06 dav1do