replicator icon indicating copy to clipboard operation
replicator copied to clipboard

[DRAFT] kafka: adding best effort and consistent mode.

Open sravotto opened this issue 1 year ago • 1 comments

This change adds transactional support to the Kafka connector for changefeeds.

This change uses the conveyor package to deliver mutation to the target database in any of the supported modes of operations.

The conveyor tracks resolved timestamps across all the partitions with a topic, and ensures that the checkpoint used by cdc-sink is advanced based on the minimal resolved timestamp received on all the partitions.

A second commit, refactors the configuration for better readability.


This change is Reviewable

sravotto avatar May 02 '24 20:05 sravotto

Codecov Report

Attention: Patch coverage is 76.34409% with 22 lines in your changes are missing coverage. Please review.

Project coverage is 75.78%. Comparing base (4dcfcdd) to head (c6c4efb).

Files Patch % Lines
internal/source/kafka/consumer.go 68.62% 10 Missing and 6 partials :warning:
internal/source/kafka/conn.go 57.14% 3 Missing :warning:
internal/source/kafka/config.go 91.30% 1 Missing and 1 partial :warning:
internal/source/kafka/provider.go 90.90% 0 Missing and 1 partial :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #843      +/-   ##
==========================================
- Coverage   75.91%   75.78%   -0.13%     
==========================================
  Files         226      226              
  Lines       10938    10962      +24     
==========================================
+ Hits         8304     8308       +4     
- Misses       1902     1917      +15     
- Partials      732      737       +5     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar May 02 '24 20:05 codecov-commenter