Nicholas

icon indicating a website link https://nickefy.medium.com/

icon location I contribute to the future by crunching data and writing about it.

Results 2 comments of Nicholas

[Question] Multiple writeStreams, Merge, and idempotent writes

Hi, can I confirm if its safe to set the idempotency spark config in each of the foreachbatch itself? ``` def x_batch(x_df: DataFrame, batch_id: int) -> None: spark = x_df.sparkSession...

too many s3 list when hoodie.metadata.enable=true

hi, I'm facing the same issues when using flink to stream hudi into S3 as well. There are too many list requests being made to s3 and it eventually causes...