Ankit Kothari

@danny0405 any update on when will this be released. I do see this PR https://github.com/apache/hudi/pull/11124 which enables metadata by default.

@danny0405 I tired using 1.0-beta2 and get s3a atomic writes not supported exception here is the stack trace ``` "message":"JobMaster for job 7c57d7e79fef591c171e8e487112be09 failed.","name":"org.apache.flink.util.FlinkException","cause":{"commonElementCount":14,"localizedMessage":"Could not start the JobMaster.","message":"Could not start...

@ad1happy2go that seems to have done the trick. Thanks However, we did see another exception post above change ``` at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:556) at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:199) at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:142) at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)...

@ad1happy2go @danny0405 any pointers ?

Thanks for responding @soumilshah1995 The index change did help a bit but we are still seeing a significant cost towards s3 bucket listing api calls

@KnightChess unfortunately no.

Hey, since this is just a POC, its less than $100. However, the storage cost is half than the listing cost so want to understand if theres an incorrect config...

Sure. Please find the schema (due to security reasons, I am not allowed to share the actual schema but this is the closest representation without giving much details) The work...

@Gatsby-Lee thanks for the suggestions. I have disabled metadata, clustering and cleaner services atm. Will let it run for a day or so and update with the changes if any....

Update: The listing cost did drop by 50% post disabling metadata, clustering and clean-up. FYI, this cost is in 10s of $ per day. The storage cost is also in...