horaedb icon indicating copy to clipboard operation
horaedb copied to clipboard

Integration test for compaction offload

Open LeslieKid opened this issue 1 year ago • 0 comments

Describe This Problem

The subproblem for #1545 . Currently, the compaction offload lacks integration tests. Impl these tests is crucial for ensuring the reliability and correctness of the compaction offload. Secondly, adding support for a manual compact operation in SQL would provide more flexibility to users, allowing them to trigger table compaction on demand.

Proposal

The procedure to test the compaction module (in sql):

  1. Create a table.
  2. Insert data into table.
  3. Flush in-memory data into sst.
  4. Update data to simulate conditions requiring compaction.
  5. Flush.
  6. Trigger compaction. (manual compaction)
  7. ~~Show compaction status.(optional)~~ (delete for easiness)
  8. Verify the result.

The problem is that horaedb has not support manual compaction in sqlness yet. I want to impl compact as a pre-command, similar to flush command. The reason is:

  • Consistency with existing command. flush forces in-memory data to be written to SST files. Similarly, compact could be triggered to merge and optimize these SST files, aligning the two operations under the same command paradigm.

Additional Context

No response

LeslieKid avatar Sep 20 '24 04:09 LeslieKid