iotdb icon indicating copy to clipboard operation
iotdb copied to clipboard

Support WAL Compression

Open THUMarkLau opened this issue 1 year ago • 1 comments

The write-ahead logging consumes a significant amount of I/O resources during data writing, sometimes even 2-7 times the amount of TsFile I/O. Currently, the write-ahead logging in the system is uncompressed, this is a main reason for the large volume of WAL. Therefore, compressing the write-ahead logging can save I/O resources and enhance system performance. This PR primarily introduces support for compressing write-ahead logging, featuring the following characteristics:

  1. The basic unit for compression and decompression is a WAL Buffer, which may contain several WAL Entries, with a default size of 16MB. This design achieves a good compression ratio while minimizing modifications to upper-layer applications.

  2. To maintain compatibility with previous versions, we have replaced the original Magic String in the WAL file with a Version String to distinguish between different versions of WAL. If there are subsequent new versions of WAL, we can continue to add new Version Strings.

  3. Multiple compression methods are supported. Currently, the choice of compression method is not open to users, but theoretically, our design supports various compression methods, and we may open this option in the future.

Changes in system resource consumption and system performance caused by this feature will be supplemented in subsequent comments.

THUMarkLau avatar May 07 '24 08:05 THUMarkLau

Codecov Report

Attention: Patch coverage is 82.53968% with 55 lines in your changes missing coverage. Please review.

Project coverage is 41.80%. Comparing base (89af73d) to head (d0b76f3).

Files Patch % Lines
...torageengine/dataregion/wal/io/WALInputStream.java 79.59% 40 Missing :warning:
...eengine/dataregion/wal/utils/WALEntryPosition.java 50.00% 7 Missing :warning:
...java/org/apache/iotdb/db/conf/IoTDBDescriptor.java 50.00% 4 Missing :warning:
...wal/allocation/AbstractNodeAllocationStrategy.java 0.00% 3 Missing :warning:
...b/storageengine/dataregion/wal/io/WALMetaData.java 88.88% 1 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #12476      +/-   ##
============================================
+ Coverage     41.75%   41.80%   +0.04%     
  Complexity       71       71              
============================================
  Files          3293     3294       +1     
  Lines        207571   207852     +281     
  Branches      24664    24708      +44     
============================================
+ Hits          86673    86888     +215     
- Misses       120898   120964      +66     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar May 25 '24 12:05 codecov[bot]

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

sonarqubecloud[bot] avatar Jun 17 '24 07:06 sonarqubecloud[bot]