materialize icon indicating copy to clipboard operation
materialize copied to clipboard

[Epic] Add Support for `ENVELOPE DEBEZIUM UPSERT` in Platform

Open nmeagan11 opened this issue 3 years ago • 2 comments

Initiative and Theme

Materialize is Friendly; Materialize works with your existing pipelines

Problem

Subset of https://github.com/MaterializeInc/materialize/issues/13044. Supporting Debezium upsert envelopes is necessary for users that already have Kafka + Debezium. This is our second highest priority envelope to support in Platform, behind ENVELOPE DEBEZIUM.

Success Criteria

Users can successfully execute a CREATE SOURCE ... FROM KAFKA ... ENVELOPE DEBEZIUM UPSERT statement.

QA Sign-off

  • [ ] Add ENVELOPE DEBEZIUM UPSERT to Zippy, Platform Checks and the Feature Benchmark

Time Horizon

Large

Blockers

  • Resolve open question in https://github.com/MaterializeInc/materialize/issues/13121#issuecomment-1165140138
  • https://github.com/MaterializeInc/materialize/issues/11772
  • Determine if we need to do https://github.com/MaterializeInc/materialize/issues/13309

nmeagan11 avatar Jun 27 '22 17:06 nmeagan11

If we opt to abandon #13309 (i.e., we continue to not perform deduplication with ENVELOPE DEBEZIUM UPSERT) then I think supporting ENVELOPE DEBEZIUM UPSERT requires no additional work over ENVELOPE UPSERT, which is quite appealing.

benesch avatar Jul 11 '22 17:07 benesch

The code for this has now landed, with the caveat that errors are un-retractable until https://github.com/MaterializeInc/materialize/issues/14084 lands.

Leaving the issue open pending QA signoff as the code is still only very lightly tested.

umanwizard avatar Aug 09 '22 16:08 umanwizard

ENVELOPE DEBEZIUM UPSERT is now present in both Zippy and Platform Checks, so this is good to go.

philip-stoev avatar Sep 12 '22 15:09 philip-stoev