parallel-consumer icon indicating copy to clipboard operation
parallel-consumer copied to clipboard

Need ability to seek to the beginning of a partition

Open fowlerp-qlik opened this issue 3 years ago • 4 comments

We need the option to read from the beginning of a partition. Something along the lines of https://stackoverflow.com/questions/49723182/kafka-consumer-seektobeginning but the parallel consumer does not expose seekToBeginning() or assignment(). Is it possible to expose these or provide an overall wrapper for seekToBeginning()?

Blocked by:

  • #346

fowlerp-qlik avatar Feb 11 '22 17:02 fowlerp-qlik

This should be achievable from the following workaround:

  • shutdown PC
  • use the raw consumer to seek the raw consumer
  • rebuild pc with the same consumer
  • pc will start receiving records from wherever underlying consumer starts from

astubbs avatar May 17 '22 16:05 astubbs

@astubbs I can't figure out how to implement this workaround. Please provide a working example.

rrva avatar May 22 '22 17:05 rrva

FYI this is being implemented here: https://github.com/astubbs/parallel-consumer/tree/features/consumer-interface

astubbs avatar Jul 05 '22 13:07 astubbs

You can try out the consumer interface branch now if you like - it works, and has been used in some experimental tests in other branches.

  • #346

astubbs avatar Jul 12 '22 12:07 astubbs

Closing Issue

johnbyrnejb avatar Jul 07 '23 14:07 johnbyrnejb