pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Handle exception in realtime decoder gracefully

Open KKcorps opened this issue 3 years ago • 1 comments

This PR continues the effort from #9163

We already added continueOnError flag in previous PRs to handle errors gracefully. This PR extends the same configs to handle decoder exceptions as well. A new config flag has also been added called skipPartialRecords through which user can decided whether to consume these partial records or not after continueOnError.

KKcorps avatar Oct 02 '22 13:10 KKcorps

Codecov Report

Merging #9511 (3158047) into master (91fa86b) will decrease coverage by 0.03%. The diff coverage is 13.33%.

@@             Coverage Diff              @@
##             master    #9511      +/-   ##
============================================
- Coverage     28.13%   28.09%   -0.04%     
  Complexity       53       53              
============================================
  Files          1936     1936              
  Lines        103927   103961      +34     
  Branches      15770    15778       +8     
============================================
- Hits          29235    29211      -24     
- Misses        71832    71901      +69     
+ Partials       2860     2849      -11     
Flag Coverage Δ
integration1 25.35% <5.00%> (-0.06%) :arrow_down:
integration2 24.43% <13.33%> (-0.26%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../local/recordtransformer/CompositeTransformer.java 0.00% <ø> (ø)
...cal/recordtransformer/SanitizationTransformer.java 0.00% <0.00%> (ø)
...gment/local/segment/creator/TransformPipeline.java 0.00% <0.00%> (ø)
...t/creator/impl/SegmentIndexCreationDriverImpl.java 0.00% <0.00%> (ø)
...ache/pinot/segment/local/utils/IngestionUtils.java 0.00% <0.00%> (ø)
...ot/spi/config/table/ingestion/IngestionConfig.java 0.00% <0.00%> (ø)
.../core/segment/processing/mapper/SegmentMapper.java 87.87% <25.00%> (-1.19%) :arrow_down:
...manager/realtime/HLRealtimeSegmentDataManager.java 78.08% <50.00%> (-2.20%) :arrow_down:
...manager/realtime/LLRealtimeSegmentDataManager.java 61.64% <54.54%> (+2.19%) :arrow_up:
...ntroller/helix/core/minion/TaskMetricsEmitter.java 34.88% <0.00%> (-51.17%) :arrow_down:
... and 42 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov-commenter avatar Oct 10 '22 04:10 codecov-commenter