[SUPPORT] Need configuration to fail the multistreamer job in case atleast one of the table ingestion fails

Open ROOBALJINDAL opened this issue 1 year ago • 1 comments

We are using AWS Serverless EMR 6.15 and airflow to submit hudi multistreamer jobs. Even if some tables fails to ingest, job is always shown as successful. We need a way to fail the job if atleast 1 of the table ingestion fails. We should know if all the tables were successfully ingested or job ended with the failures. I think its a common business usecase and should be handled if possible

Need a configuration to turn this feature on/off. Let me know if any other information is required. Thanks

Jul 01 '24 06:07 ROOBALJINDAL

@ROOBALJINDAL There is no such configuration available today with HoodieMultiTableStreamer.

Created JIRA - https://issues.apache.org/jira/browse/HUDI-7944

Feel free to contribute here if you are interested.

Jul 01 '24 11:07 ad1happy2go