redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

[CORE-2841] Transforms: Start consuming from an arbitrary offset (numeric from start/end or timestamp)

Open oleiman opened this issue 1 year ago • 3 comments

This PR wires up the ability to configure the start offset of a transform at deploy time. This can be either a unix timestamp (ms since epoch) or an offset delta (+oo from start offset of -oo from end).

Includes rpk experience.

Backports Required

  • [x] none - not a bug fix
  • [ ] none - this is a backport
  • [ ] none - issue does not exist in previous branches
  • [ ] none - papercut/not impactful enough to backport
  • [ ] v24.1.x
  • [ ] v23.3.x
  • [ ] v23.2.x

Release Notes

Improvements

  • Adds the ability to start transform processing from an arbitrary offset on the input topic.

oleiman avatar Jun 24 '24 21:06 oleiman

/ci-repeat 1

oleiman avatar Jun 27 '24 04:06 oleiman

force push contents:

  • introduce kafka::offset_delta
  • reject offset "-0" in rpk
  • minor CR changes

oleiman avatar Jun 28 '24 18:06 oleiman

force push empty diff (signoff)

oleiman avatar Jun 28 '24 18:06 oleiman

force push rebase dev to avoid conflicts on feature flag

oleiman avatar Jul 01 '24 18:07 oleiman

force push feature flag for the new position types, minor CR changes

oleiman avatar Jul 01 '24 19:07 oleiman

empty diff for commit signoff

oleiman avatar Jul 01 '24 19:07 oleiman

force push contents:

  • serde shenanigans
  • adjust feature gate to guard new offset alternatives only

oleiman avatar Jul 02 '24 02:07 oleiman

force push make legacy_transform_offset_options struct write-only.

oleiman avatar Jul 02 '24 02:07 oleiman

force push a dt test case

oleiman avatar Jul 02 '24 17:07 oleiman

force push to address some CR comments on the rpk side

oleiman avatar Jul 02 '24 21:07 oleiman

CI Failure is https://github.com/redpanda-data/redpanda/issues/20574 (known, unrelated)

oleiman avatar Jul 03 '24 15:07 oleiman