hudi icon indicating copy to clipboard operation
hudi copied to clipboard
verified publisher icon apache

[SUPPORT] Spark multi writer failed ! ! !

Open eric9204 opened this issue 3 years ago • 5 comments

Describe the problem you faced

when i use config 1 ,The exception A occurred in one of the two spark jobs writing the same table; when i use config 2 ,The exception B occurred in one of the two spark jobs writing the same table.

there was not clustering config .

To Reproduce

config 1: hoodie.write.concurrency.mode=optimistic_concurrency_control hoodie.cleaner.policy.failed.writes=LAZY hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider hoodie.write.lock.zookeeper.url=10.1.2.12 hoodie.write.lock.zookeeper.port=2181 hoodie.write.lock.zookeeper.lock_key=ss_bucket_lock hoodie.write.lock.zookeeper.base_path=/ss_bucket_lock

config 2 : hoodie.write.concurrency.mode=optimistic_concurrency_control hoodie.cleaner.policy.failed.writes=LAZY hoodie.write.lock.provider=org.apache.hudi.hive.HiveMetastoreBasedLockProvider hoodie.write.lock.hivemetastore.database=default hoodie.write.lock.hivemetastore.table=occ_lock_ss_bucket_dsj hoodie.write.lock.hivemetastore.uris=thrift://host-10-4-6-15:9083

Expected behavior

Two spark jobs could write the same hudi table normally at the same time !

Environment Description

  • Hudi version : Hudi-0.11.1
  • Spark version : spark-3.1.x
  • Hive version : None
  • Hadoop version : Hadoop-3.3.0
  • Storage (HDFS/S3/GCS..) : Hdfs
  • Running on Docker? (yes/no) : No

**Stacktrace ，Exception A **

22/08/05 15:01:15 INFO CommitUtils: Creating  metadata for INSERT numWriteStats:1numReplaceFileIds:0
22/08/05 15:01:15 INFO TransactionManager: Transaction starting for Option{val=[==>20220805150110559__commit__INFLIGHT]} with latest completed transaction instant Option{val=[20220805150050505__commit__COMPLETED]}
22/08/05 15:01:15 INFO ZookeeperBasedLockProvider: ACQUIRING lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO ZookeeperBasedLockProvider: ACQUIRED lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO TransactionManager: Transaction started for Option{val=[==>20220805150110559__commit__INFLIGHT]} with latest completed transaction instant Option{val=[20220805150050505__commit__COMPLETED]}
22/08/05 15:01:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /tmp/hudi/ss_bucket_dsj
22/08/05 15:01:18 INFO HoodieTableConfig: Loading table properties from /tmp/hudi/ss_bucket_dsj/.hoodie/hoodie.properties
22/08/05 15:01:18 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from /tmp/hudi/ss_bucket_dsj
22/08/05 15:01:18 INFO HoodieTableMetaClient: Loading Active commit timeline for /tmp/hudi/ss_bucket_dsj
22/08/05 15:01:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/08/05 15:01:18 INFO FileSystemViewManager: Creating in-memory based Table View
22/08/05 15:01:18 INFO HoodieBucketIndex: use bucket index, numBuckets=1
22/08/05 15:01:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO SimpleConcurrentFileWritesConflictResolutionStrategy: Found conflicting writes between first operation = {actionType=commit, instantTime=20220805150110559, actionState=INFLIGHT'}, second operation = {actionType=commit, instantTime=20220805150100333, actionState=COMPLETED'} , intersecting file ids [00000000-4447-4513-8068-326e01720c62-0]
22/08/05 15:01:18 INFO TransactionUtils: Conflict encountered between current instant = {actionType=commit, instantTime=20220805150110559, actionState=INFLIGHT'} and instant = {actionType=commit, instantTime=20220805150100333, actionState=COMPLETED'}, attempting to resolve it...
22/08/05 15:01:18 INFO TransactionManager: Transaction ending with transaction owner Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO ZookeeperBasedLockProvider: RELEASING lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO ZookeeperBasedLockProvider: RELEASED lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO TransactionManager: Transaction ended with transaction owner Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO MapPartitionsRDD: Removing RDD 167 from persistence list
22/08/05 15:01:18 INFO MapPartitionsRDD: Removing RDD 159 from persistence list
22/08/05 15:01:18 INFO BlockManager: Removing RDD 167
22/08/05 15:01:18 ERROR HoodieStreamingSink: Micro batch id=32 threw following exception: 
org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
        at org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102)
        at org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
        at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
        at org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79)
        at org.apache.hudi.client.SparkRDDWriteClient.preCommit(SparkRDDWriteClient.java:473)
        at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:233)
        at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:122)
        at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:651)
        at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$2(HoodieStreamingSink.scala:91)
        at scala.util.Try$.apply(Try.scala:213)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$1(HoodieStreamingSink.scala:90)
        at org.apache.hudi.HoodieStreamingSink.retry(HoodieStreamingSink.scala:166)
        at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:89)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:586)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$15(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Caused by: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
        ... 38 more
22/08/05 15:01:18 INFO HoodieStreamingSink: Ignore the exception and move on streaming as per hoodie.datasource.write.streaming.ignore.failed.batch configuration
22/08/05 15:01:18 INFO HoodieStreamingSink: Micro batch id=32 succeeded
22/08/05 15:01:18 INFO BlockManager: Removing RDD 159
22/08/05 15:01:18 INFO CheckpointFileManager: Writing atomically to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32 using temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp
22/08/05 15:01:18 INFO CheckpointFileManager: Renamed temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32

**Stacktrace for Exception B **

22/08/05 14:35:08 INFO TransactionManager: Transaction starting for Option{val=[==>20220805143450492__commit__INFLIGHT]} with latest completed transaction instant Option{val=[20220805143430140__commit__COMPLETED]}
22/08/05 14:35:08 INFO HiveMetastoreBasedLockProvider: ACQUIRING lock at database default and table occ_lock_ss_bucket_dsj
22/08/05 14:35:08 INFO LockManager: Retrying to acquire lock...
22/08/05 14:35:18 INFO HiveMetastoreBasedLockProvider: ACQUIRING lock at database default and table occ_lock_ss_bucket_dsj
22/08/05 14:35:18 INFO MapPartitionsRDD: Removing RDD 6 from persistence list
22/08/05 14:35:18 INFO BlockManager: Removing RDD 6
22/08/05 14:35:18 INFO MapPartitionsRDD: Removing RDD 14 from persistence list
22/08/05 14:35:18 INFO BlockManager: Removing RDD 14
22/08/05 14:35:18 ERROR HoodieStreamingSink: Micro batch id=133 threw following exception: 
java.lang.IllegalArgumentException: ALREADY_ACQUIRED
        at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
        at org.apache.hudi.hive.HiveMetastoreBasedLockProvider.acquireLock(HiveMetastoreBasedLockProvider.java:136)
        at org.apache.hudi.hive.HiveMetastoreBasedLockProvider.tryLock(HiveMetastoreBasedLockProvider.java:112)
        at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:67)
        at org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:53)
        at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:230)
        at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:122)
        at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:651)
        at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$2(HoodieStreamingSink.scala:91)
        at scala.util.Try$.apply(Try.scala:213)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$1(HoodieStreamingSink.scala:90)
        at org.apache.hudi.HoodieStreamingSink.retry(HoodieStreamingSink.scala:166)
        at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:89)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:586)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$15(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
22/08/05 14:35:18 INFO HoodieStreamingSink: Ignore the exception and move on streaming as per hoodie.datasource.write.streaming.ignore.failed.batch configuration
22/08/05 14:35:18 INFO HoodieStreamingSink: Micro batch id=133 succeeded
22/08/05 14:35:18 ERROR MicroBatchExecution: Query [id = d0ebeccf-4600-4aba-b097-420cd8356448, runId = 233f22f7-66d9-47a8-9093-0381f157428a] terminated with error
java.lang.AssertionError: assertion failed: Concurrent update to the commit log. Multiple streaming jobs detected for 133
        at scala.Predef$.assert(Predef.scala:223)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:602)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.withProgressLocked(MicroBatchExecution.scala:613)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:598)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException: assertion failed: Concurrent update to the commit log. Multiple streaming jobs detected for 133
=== Streaming Query ===
Identifier: [id = d0ebeccf-4600-4aba-b097-420cd8356448, runId = 233f22f7-66d9-47a8-9093-0381f157428a]
Current Committed Offsets: {KafkaV2[Subscribe[user_hive4]]: {"user_hive4":{"2":49415,"1":47279,"0":49423}}}
Current Available Offsets: {KafkaV2[Subscribe[user_hive4]]: {"user_hive4":{"2":49752,"1":47604,"0":49760}}}

Current State: ACTIVE
Thread State: RUNNABLE

Logical Plan:
Project [id#23, cast(score#24 as double) AS score#31, sex#25, to_utc_timestamp(cast(ts#26 as timestamp), Asia/Shanghai) AS ts#32]
+- SubqueryAlias s
   +- Project [split(kafka_value#21, ,, -1)[0] AS id#23, split(kafka_value#21, ,, -1)[1] AS score#24, split(kafka_value#21, ,, -1)[2] AS sex#25, split(kafka_value#21, ,, -1)[3] AS ts#26]
      +- SubqueryAlias t
         +- Project [cast(value#8 as string) AS kafka_value#21]
            +- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@6a155037, KafkaV2[Subscribe[user_hive4]]

        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:354)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Caused by: java.lang.AssertionError: assertion failed: Concurrent update to the commit log. Multiple streaming jobs detected for 133
        at scala.Predef$.assert(Predef.scala:223)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:602)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.withProgressLocked(MicroBatchExecution.scala:613)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:598)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        ... 1 more

eric9204 avatar Aug 05 '22 04:08 eric9204

I also met the second issue, this should already be fixed in master branch , you can try it out

fengjian428 avatar Aug 05 '22 14:08 fengjian428

I also met the second issue, this should already be fixed in master branch , you can try it out

@fengjian428 Thank you for your reply，I'll try the master branch..

eric9204 avatar Aug 08 '22 02:08 eric9204

I also met the second issue, this should already be fixed in master branch , you can try it out

@fengjian428 Thank you for your reply，I'll try the master branch..

The second issue was indeed fixed , thank you !

eric9204 avatar Aug 08 '22 06:08 eric9204

@eric9204 For the first failure, can you share the full write configs? Was there any pending/inflight compaction and clustering? Can you share the .hoodie directory under the base path of the table? I think the lock provider is running fine but conflict resolution failed. The timeline will help us build the sequence of events and see why the resolution failed.

codope avatar Aug 10 '22 11:08 codope

@eric9204 For the first failure, can you share the full write configs? Was there any pending/inflight compaction and clustering? Can you share the .hoodie directory under the base path of the table? I think the lock provider is running fine but conflict resolution failed. The timeline will help us build the sequence of events and see why the resolution failed.

2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/.aux
2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/.heartbeat
2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/.schema
2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/.temp
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback
    0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback.requested
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback
    0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback.requested
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback
    0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback
    0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback
    0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback
    0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback.requested
1.5 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback
    0 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback.inflight
1.3 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback.requested
1.5 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback
    0 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback.inflight
1.3 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback.requested
2.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.commit
    0 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.commit.requested
3.0 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.inflight
1.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean.inflight
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean.requested
2.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.commit
    0 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.commit.requested
3.0 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.inflight
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean.inflight
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean.requested
2.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.commit
    0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.commit.requested
3.0 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean.inflight
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean.requested
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback
    0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback.inflight
1.3 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback.requested
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback
    0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback.inflight
1.3 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback.requested
2.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.commit
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean.inflight
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean.requested
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback.inflight
1.3 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback.requested
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645810.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645810.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback.inflight
1.3 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback.requested
2.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.commit
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean.inflight
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.commit
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.inflight
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135734515.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135734515.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean.requested
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135753793.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135753793.inflight
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.inflight
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean.inflight
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean.requested
2.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.commit
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.commit.requested
3.0 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.inflight
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean.inflight
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean.requested
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback.inflight
1.3 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback.requested
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback.inflight
1.3 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback.requested
2.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.commit
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.commit.requested
3.0 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean.requested
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.commit
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean.requested
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.commit
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean.requested
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140215991.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140215991.inflight
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.commit
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.inflight
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean
1.6 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean.inflight
1.6 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean.requested
2.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.commit
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.inflight
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140238784.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140238784.inflight
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback.requested
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140308986.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140308986.rollback.requested
    0 2022-08-08 13:41 /tmp/hudi/ss_bucket_dsj/.hoodie/archived
  781 2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/hoodie.properties

Snipaste_2022-08-11_14-00-41

@codope The hudi table type is cow,and there was not any config about compaction or clustering.

eric9204 avatar Aug 11 '22 02:08 eric9204

@eric9204 : just so you are aware of how conflict resolution happens w/in hudi.

If two writers are writing to same file, hudi will fail one of them and will succeed the other. But if two writers are writing to two different files, both should succeed (assuming both are concurrent writers).

For eg, lets say writer1 ingests 100 records in commit C1.

and at time t10, wirter1 updates the same 100 records and ingests to hudi. wirter2 updates the same 100 records and ingests to hudi. Hudi may not be in a position to choose the winner. and so will succeed one and will fail the other. In this situations, you will see ConflictResolutionFailed error messages.

But lets say writer1 ingests 100 records in commit C1 and writer2 ingests diff set of 100 records to a diff partition in commit C2.

at time t10, wirter1 updates records as part of C1 and ingests to hudi. writer2 updates the records as part of C2 and ingests to hudi. Both write will succeed if it happens concurrently. bcoz, there is no overlap.

Hope that clarifies why one of the writer failed in your case. you can get some additions details here.

nsivabalan avatar Aug 30 '22 23:08 nsivabalan

@nsivabalan I know what you mean, there is another problem.

If one of writers fail, this hoodie commit transaction was failed at hudi side, but this batch was successful at spark side. So the next micro batch will consume record from last successful commit of offset. the data of last successful micro batch which is failed at hoodie side actually may be lost 。

22/08/05 15:01:18 INFO HoodieStreamingSink: Ignore the exception and move on streaming as per hoodie.datasource.write.streaming.ignore.failed.batch configuration
22/08/05 15:01:18 INFO HoodieStreamingSink: Micro batch id=32 succeeded
22/08/05 15:01:18 INFO BlockManager: Removing RDD 159
22/08/05 15:01:18 INFO CheckpointFileManager: Writing atomically to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32 using temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp
22/08/05 15:01:18 INFO CheckpointFileManager: Renamed temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32

Could the community add a retry strategy to make it successful instead of just discarding it.

eric9204 avatar Aug 31 '22 02:08 eric9204

yes. we have a config which you can flip to ensure spark streaming fails if hudi write fails. hoodie.datasource.write.streaming.ignore.failed.batch

default value for this config is set to true until 2 weeks back. we have flipped the default to false. if you can set this config value to false, atleast you won' see any data loss. if there are any valid errors, your pipeline should fail.

nsivabalan avatar Aug 31 '22 18:08 nsivabalan

@nsivabalan thanks! I've retested with hoodie.datasource.write.streaming.ignore.failed.batch=false ,the spark micro-batch indeed fail when hudi commit fail.

So, should I close this issue?

eric9204 avatar Sep 01 '22 02:09 eric9204

yes, thanks!

nsivabalan avatar Sep 01 '22 06:09 nsivabalan