foundationdb icon indicating copy to clipboard operation
foundationdb copied to clipboard

Add support for bulk data loading that bypasses transaction logs

Open etschannen opened this issue 7 years ago • 1 comments

This feature will improve write bandwidth when initially loading data into a database. During the load, the range will not be readable. The design needs to consider how to get back to a consistent view of the world if a client dies during loading.

Proposed design:

  • verify the range is empty, lock reads to the range, writes to the range that come from the tlogs are discarded
  • disable data movement for the range
  • directly send mutations to the server responsible for the range
  • if one of the servers dies while loading mutations, clear the loaded data from the other servers and unlock the range
  • unlocked the range
  • re-enable data movement for the range

etschannen avatar Jan 04 '19 18:01 etschannen

Fast restore will likely need this bulk loading feature to achieve superior performance on large amount (tens of terabytes) of range data in backup.

Built on top of the fast restore framework, this can be done as follows:

  1. Similar to how fast restore sample backup to divide keyspace for appliers, let loaders sample all range files and controller divide the keyspace equally to storage servers;
  2. Disable DD and let restore controller sets key-range for storage servers, which is done similar to how DD sets shards for SS;
  3. Let loaders parse range files and directly send kv pairs to SSes. <-- If we have 200 SSes for double replication and each SS can provide 10MB/s write throughput, we got 100*10MB/s write throughput.

Implementation can be done in two phases:

  1. Change the fast restore to load range files and write kv to FDB; <-- this does not involve changing DD;
  2. Implement the step 2) and 3) above.

xumengpanda avatar Aug 01 '20 02:08 xumengpanda