Fast Sync Mode

[bitjson]

Chaingraph currently supports only a single mode of syncing which we'll call "maintenance mode". In maintenance mode the agent maintains:

• a list of block header hashes according to each connected node (BlockTree), and
• the SyncState for that node (information on which blocks have not yet been saved to the database)

The agent requests new blocks one-by-one, filling up the BlockBuffer to ensure blocks are ready to send to the DB whenever the DB is ready to accept another block. As new block headers come in, they're simply added to the tree and requested when block buffer space becomes available.

Maintenance mode is the typical mode of operation in production – the database catches up to all connected nodes, saving new transactions and blocks as they come in and keeping indexes updated. So the performance of maintenance mode ultimately determines the maximum throughput of a particular Chaingraph instance. (And maintenance mode performance is primarily influenced by triggers and indexes, both the built-in set and any custom triggers/indexes added by users.)

Right now, there's a sort of "initial sync mode" which uses the same code paths as maintenance mode, but simply disables a few non-critical indexes and only building them after the sync is complete. This is a common, easy-to-implement strategy for quickly populating a Postgres database, but with a little more effort, we could build a true "fast sync mode" using COPY commands.

Fast sync mode should be enabled by default – it would begin without any indexes or foreign key constraints, relying on the agent to keep track of progress and internal IDs. It should stream data into each table using pg-copy-streams, and ideally it should be able to recover from crashes or DB connection issues. (Maybe initially the recovery would simply be to identify that a fast sync has already been attempted and create all indexes, then continue the sync in maintenance mode.) It's probably also worth experimenting with setting all tables to UNLOGGED initially (should be optional since some users will be replicating the Chaingraph database during sync).

It's also important that it remains easy to test maintenance mode performance on new infrastructure, so at a minimum, we need a disable-able CHAINGRAPH_ENABLE_FAST_SYNC environment variable. It may even be valuable to end the fast sync with a few GB left to sync, build indexes, then finish the last portion of the sync in maintenance mode to estimate the instance's maximum throughput (on the current hardware). Maybe just an environment variable which allows operators to select a place to stop the sync, e.g. CHAINGRAPH_FAST_SYNC_END_HEIGHT=bchn:700000,tbchn:50000 would pretend those are the best block heights until all indexes are built. (Also might need a CHAINGRAPH_FAST_SYNC_ONLY which refuses to enter maintenance mode and just kills the agent when the fast sync is complete – then operators can apply any custom triggers/indexes before disabling CHAINGRAPH_FAST_SYNC_ONLY and verifying the maximum performance of their particular setup.)

Looking at maintenance mode sync times on a machine with plenty of CPU, memory (64GB), and a large NVMe disk, blockThroughput currently peaks (at ~9MB/s) when the block buffer fills enough to utilize all pgPool clients (CPU count), then slowly falls (to ~2MB/s) over the course of the sync (presumably because primary key indexes grow larger, fall out of memory, and become more expensive to update). Running iotop during the sync also indicates that disk read is consistently ~10x disk write, so I imagine a fast sync mode could really bring down the total time to setup Chaingraph (currently ~9 hours on the above machine).