DataflowTemplates icon indicating copy to clipboard operation
DataflowTemplates copied to clipboard

add PubSubToPubSub idattribute options to remove message duplication

Open case-k-git opened this issue 4 years ago • 2 comments

use idattribute method to remove message duplication. Dataflow remove message duplication automatically based on created pub/sub message id at least once policy. but message gonna duplicate if publisher send same message several time. by using idattribute. we can remove message duplication based on idattributekey(message id ) set by publisher

https://beam.apache.org/releases/javadoc/2.27.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html

case-k-git avatar Jun 01 '21 13:06 case-k-git

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Apr 16 '22 10:04 stale[bot]

Hi @case-k-git -- thanks for your contribution here. Unfortunately we ran behind on reviews, and the structure of the project changed a bit. Is there still interest in pursuing this template? We can work together on a plan to resolve conflicts + review the code.

bvolpato avatar Apr 10 '23 13:04 bvolpato