iotdb icon indicating copy to clipboard operation
iotdb copied to clipboard

fix tests in BatchedAlignedSeriesReadChunkCompactionTest

Open kabo87777 opened this issue 1 month ago • 0 comments

Fix flaky tests in BatchedAlignedSeriesReadChunkCompactionTest

Description

This PR fixes 12 flaky tests in BatchedAlignedSeriesReadChunkCompactionTest that were failing intermittently due to non-deterministic iteration order of collections.

Root Cause

The getPaths() method in AbstractCompactionTest.java had two sources of non-determinism:

  1. HashSet usage: The method used HashSet<IFullPath> which does not guarantee iteration order
  2. Unsorted schemaMap.values(): The measurementSchemas list was created directly from schemaMap.values() without sorting, and since schemaMap is backed by a ConcurrentHashMap, the iteration order is non-deterministic

This caused AlignedFullPath objects to be created with measurements in different orders across test runs, leading to comparison failures when validating compaction results.

Fix

  1. Changed HashSet to LinkedHashSet to preserve insertion order
  2. Sorted measurementSchemas by measurement name before creating the list

Changes

File: iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/AbstractCompactionTest.java

// Before (non-deterministic):
Set<IFullPath> paths = new HashSet<>();
// ...
List<IMeasurementSchema> measurementSchemas = new ArrayList<>(schemaMap.values());

// After (deterministic):
Set<IFullPath> paths = new LinkedHashSet<>();
// ...
List<IMeasurementSchema> measurementSchemas =
    schemaMap.values().stream()
        .sorted(Comparator.comparing(IMeasurementSchema::getMeasurementName))
        .collect(Collectors.toList());

Tests Fixed

  • BatchedAlignedSeriesReadChunkCompactionTest#testCompactionWithDifferentCompressionTypeOrEncoding
  • BatchedAlignedSeriesReadChunkCompactionTest#testCompactionWithEmptyBatch
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByFlushPage
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByWritePoint
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithAllDeletedColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithAllDeletedPageByFlushPage
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNotExistColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNullColumn
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNullColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithPartialDeletedColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithPartialDeletedPageByWritePoint

Verification

Verified using NonDex with 3 different random seeds - all tests pass consistently.

kabo87777 avatar Dec 09 '25 07:12 kabo87777