Support WAL Compression
The write-ahead logging consumes a significant amount of I/O resources during data writing, sometimes even 2-7 times the amount of TsFile I/O. Currently, the write-ahead logging in the system is uncompressed, this is a main reason for the large volume of WAL. Therefore, compressing the write-ahead logging can save I/O resources and enhance system performance. This PR primarily introduces support for compressing write-ahead logging, featuring the following characteristics:
-
The basic unit for compression and decompression is a WAL Buffer, which may contain several WAL Entries, with a default size of 16MB. This design achieves a good compression ratio while minimizing modifications to upper-layer applications.
-
To maintain compatibility with previous versions, we have replaced the original Magic String in the WAL file with a Version String to distinguish between different versions of WAL. If there are subsequent new versions of WAL, we can continue to add new Version Strings.
-
Multiple compression methods are supported. Currently, the choice of compression method is not open to users, but theoretically, our design supports various compression methods, and we may open this option in the future.
Changes in system resource consumption and system performance caused by this feature will be supplemented in subsequent comments.
Codecov Report
Attention: Patch coverage is 82.53968% with 55 lines in your changes missing coverage. Please review.
Project coverage is 41.80%. Comparing base (
89af73d) to head (d0b76f3).
Additional details and impacted files
@@ Coverage Diff @@
## master #12476 +/- ##
============================================
+ Coverage 41.75% 41.80% +0.04%
Complexity 71 71
============================================
Files 3293 3294 +1
Lines 207571 207852 +281
Branches 24664 24708 +44
============================================
+ Hits 86673 86888 +215
- Misses 120898 120964 +66
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
