hudi
hudi copied to clipboard
apache
File not found while using metadata table for insert_overwrite table
We are incrementally writing to a hudi table with insert_overwrite operations. Recently, We enabled Hudi metadata table for these tables. However after few days we started to encounter the FileNotFoundException issue while reading these tables from athena (with metadata listing enabled). Upon further investigation, we observed that the metadata contained older files references that were cleaned up by the cleaner and are no longer available.
Steps to reproduce the behavior:
- create a simple df and write to a hudi table incrementally with these properties
hoodie.datasource.meta.sync.enable=true
hoodie.meta.sync.client.tool.class=org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool
hoodie.write.markers.type=DIRECT
**hoodie.metadata.enable=true
hoodie.datasource.write.operation=insert_overwrite**
hoodie.datasource.write.partitionpath.field=cs_load_hr
hoodie.datasource.hive_sync.partition_fields=cs_load_hr
partition.assignment.strategy=org.apache.kafka.clients.consumer.RangeAssignor
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator
hoodie.deltastreamer.keygen.timebased.timestamp.type=DATE_STRING
hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd/HH
hoodie.deltastreamer.source.hoodieincr.partition.extractor.class=org.apache.hudi.hive.SlashEncodedHourPartitionValueExtractor
hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.SlashEncodedHourPartitionValueExtractor
hoodie.parquet.compression.codec=snappy
hoodie.table.services.enabled=true
hoodie.rollback.using.markers=false
hoodie.commits.archival.batch=30
hoodie.archive.delete.parallelism=500
hoodie.index.type=SIMPLE
hoodie.clean.allow.multiple=false
hoodie.clean.async=true
hoodie.clean.automatic=true
hoodie.cleaner.policy=KEEP_LATEST_COMMITS
hoodie.cleaner.commits.retained=3
hoodie.cleaner.parallelism=500
hoodie.cleaner.incremental.mode=true
hoodie.clean.max.commits=8
hoodie.archive.async=true
hoodie.archive.automatic=true
hoodie.archive.merge.enable=true
hoodie.archive.merge.files.batch.size=60
hoodie.keep.max.commits=10
hoodie.keep.min.commits=5
df.write.format("org.apache.hudi").options(hudiOptions).mode(SaveMode.Append).save(hudiOutputTablePath)
- after few incremental writes, some of the base files should be updated & metadata does not get updated properly, it continues to persist old files pointer as well.
- if you try reading the table using spark or athena, you will get FileNotFoundException keep in mind to enable metadata while reading. upon disabling the metadata listing on the read side, there is no error and reads work fine.
- Note : We have observed this issue only for insert_overwrite operations. Upsert operation table's metadata gets updated correctly.
Expected behavior
It is expected that the hoodie metadata gets updated correctly.
Environment Description
-
Hudi version : 0.13.1
-
Spark version : 3.2.1
-
Hive version : NA
-
Hadoop version :
-
Storage (HDFS/S3/GCS..) : S3
-
Running on Docker? (yes/no) :
Additional context
The timeline also contains replaceCommits for corrupted tables. (which are not present in case of upsert table)
$ aws s3 ls s3://tmp-data/investments_ctr_tbl/.hoodie/
PRE .aux/
PRE archived/
PRE metadata/
2023-12-08 13:32:17 0 .aux_$folder$
2023-12-08 13:32:17 0 .schema_$folder$
2023-12-08 13:32:17 0 .temp_$folder$
2023-12-14 22:17:18 4678 20231214221641350.clean
2023-12-14 22:17:11 3227 20231214221641350.clean.inflight
2023-12-14 22:17:10 3227 20231214221641350.clean.requested
2023-12-22 21:50:54 4439 20231222214849300.clean
2023-12-22 21:50:45 4337 20231222214849300.clean.inflight
2023-12-22 21:50:45 4337 20231222214849300.clean.requested
2023-12-30 21:51:16 4439 20231230214431936.clean
2023-12-30 21:51:07 4337 20231230214431936.clean.inflight
2023-12-30 21:51:07 4337 20231230214431936.clean.requested
2024-01-07 21:53:30 4439 20240107215204594.clean
2024-01-07 21:53:23 4337 20240107215204594.clean.inflight
2024-01-07 21:53:22 4337 20240107215204594.clean.requested
2024-01-15 21:55:00 4439 20240115215112126.clean
2024-01-15 21:54:52 4337 20240115215112126.clean.inflight
2024-01-15 21:54:52 4337 20240115215112126.clean.requested
2024-01-23 21:46:53 4439 20240123214442067.clean
2024-01-23 21:46:45 4337 20240123214442067.clean.inflight
2024-01-23 21:46:45 4337 20240123214442067.clean.requested
2024-01-27 21:42:46 4107 20240127214137934.replacecommit
2024-01-27 21:42:22 2221 20240127214137934.replacecommit.inflight
2024-01-27 21:41:42 2522 20240127214137934.replacecommit.requested
2024-01-28 21:47:38 4107 20240128214645441.replacecommit
2024-01-28 21:47:15 2221 20240128214645441.replacecommit.inflight
2024-01-28 21:46:50 2522 20240128214645441.replacecommit.requested
2024-01-29 21:48:52 4107 20240129214748785.replacecommit
2024-01-29 21:48:23 2221 20240129214748785.replacecommit.inflight
2024-01-29 21:47:55 2522 20240129214748785.replacecommit.requested
2024-01-30 21:47:51 4107 20240130214652235.replacecommit
2024-01-30 21:47:26 2221 20240130214652235.replacecommit.inflight
2024-01-30 21:46:58 2522 20240130214652235.replacecommit.requested
2024-01-31 21:46:06 4107 20240131214409159.replacecommit
2024-01-31 21:45:47 2221 20240131214409159.replacecommit.inflight
2024-01-31 21:44:16 2522 20240131214409159.replacecommit.requested
2024-01-31 21:45:57 4439 20240131214417071.clean
2024-01-31 21:45:49 4337 20240131214417071.clean.inflight
2024-01-31 21:45:49 4337 20240131214417071.clean.requested
2024-02-01 21:46:04 4107 20240201214350334.replacecommit
2024-02-01 21:45:35 2221 20240201214350334.replacecommit.inflight
2024-02-01 21:43:59 2522 20240201214350334.replacecommit.requested
2024-02-02 21:48:21 4107 20240202214718803.replacecommit
2024-02-02 21:47:56 2221 20240202214718803.replacecommit.inflight
2024-02-02 21:47:26 2522 20240202214718803.replacecommit.requested
2024-02-03 21:52:09 4107 20240203215112346.replacecommit
2024-02-03 21:51:46 2221 20240203215112346.replacecommit.inflight
2024-02-03 21:51:18 2522 20240203215112346.replacecommit.requested
2024-02-04 21:49:18 4107 20240204214759805.replacecommit
2024-02-04 21:48:40 2221 20240204214759805.replacecommit.inflight
2024-02-04 21:48:06 2522 20240204214759805.replacecommit.requested
2024-02-05 14:13:44 13208 20240205141148519.replacecommit
2024-02-05 14:13:21 8512 20240205141148519.replacecommit.inflight
2024-02-05 14:11:52 2522 20240205141148519.replacecommit.requested
2023-12-08 13:32:17 0 archived_$folder$
2023-12-08 13:32:25 1130 hoodie.properties
2023-12-08 13:32:21 0 metadata_$folder$
Also, here is the output of the command metadata validate-files ran from the hudi-cli on the corrupted table
2024-02-02 11:28:43,123 INFO metadata.BaseTableMetadata: Listed file in partition from metadata: partition=2023/12/17/00, #files=2
%d [%thread] %-5level %logger - %msg%n%d [%thread] %-5level %logger - %msg%n????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? Partition ? File Name ? Is Present in FS ? Is Present in Metadata ? FS size ? Metadata size ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/29/00 ? 1940e6fe-febf-409b-9096-a75d01b288f7-0_0-4-1446_20231229214316193.parquet ? false ? true ? 0 ? 450079 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/29/00 ? 27107d76-bfbc-45eb-8eed-15752437efa6-0_1-5-1444_20231230214425196.parquet ? false ? true ? 0 ? 450105 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2024/01/02/00 ? 297612f5-4681-47e0-bac2-d4293ac50827-0_1-4-1414_20240102214855026.parquet ? false ? true ? 0 ? 449931 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/25/00 ? 82adb6c9-89b1-4f44-97f1-1ef235e44a4a-0_1-4-1402_20231225214603565.parquet ? false ? true ? 0 ? 449930 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/25/00 ? 3a19cf04-eca6-4e24-bd2e-75e0262d1eee-0_2-6-1406_20231226214637699.parquet ? false ? true ? 0 ? 449920 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/31/00 ? 890aa505-a02d-4adc-b4fd-62a39c28be08-0_2-4-1412_20240101214642664.parquet ? false ? true ? 0 ? 450162 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/31/00 ? e980b191-f4d0-426f-ac87-b1b773c695bf-0_2-6-1405_20231231214814370.parquet ? false ? true ? 0 ? 450084 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2024/01/01/00 ? b3a014b1-1e84-48f0-bec9-b1f3113f7d0f-0_1-4-1411_20240101214642664.parquet ? false ? true ? 0 ? 450462 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2024/01/01/00 ? 8839f2f0-8c8a-425e-b397-8151c5d2c0ae-0_0-4-1413_20240102214855026.parquet ? false ? true ? 0 ? 450512 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/26/00 ? df627807-1c95-4fa8-b253-c8317e0d79d8-0_0-4-1440_20231227220120993.parquet ? false ? true ? 0 ? 449428 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/26/00 ? 711a7fb3-b9e4-4ae8-8f07-1f162ad56453-0_0-6-1404_20231226214637699.parquet ? false ? true ? 0 ? 449332 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/21/00 ? ef8c236e-e47b-4b14-82b9-88b35a2b4b65-0_2-7-1492_20231222214842148.parquet ? false ? true ? 0 ? 450542 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/21/00 ? f0bc5273-c44a-4ff3-9b2a-c5538368d016-0_2-4-1394_20231221214633206.parquet ? false ? true ? 0 ? 450545 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/22/00 ? 274eeefe-7e73-4aae-bdd3-3d1ff25fdf96-0_0-7-1490_20231222214842148.parquet ? false ? true ? 0 ? 450676 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/22/00 ? 59f01016-e8f7-46d1-a692-7a09c540c62e-0_1-6-1468_20231223214749872.parquet ? false ? true ? 0 ? 450381 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/19/00 ? d125bb1b-e853-48e2-b24d-2324a104ac53-0_0-4-1392_20231220214948676.parquet ? false ? true ? 0 ? 450057 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/19/00 ? 3d269000-52a2-4a73-9b6b-210c8825dd43-0_0-4-1380_20231219214031893.parquet ? false ? true ? 0 ? 450073 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/20/00 ? f042aece-42c0-41c0-ac92-51d06c092be8-0_1-4-1394_20231220214948676.parquet ? false ? true ? 0 ? 450911 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/20/00 ? c23b43d5-bfab-4234-83cc-e1c757dffefc-0_1-4-1393_20231221214633206.parquet ? false ? true ? 0 ? 450961 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/27/00 ? bc2cf246-4092-49c8-81ed-eedbf84f22e4-0_1-4-1441_20231227220120993.parquet ? false ? true ? 0 ? 451679 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/27/00 ? 8a685c8a-6ff4-4964-985c-6d78d76f41a4-0_2-4-1445_20231228214506235.parquet ? false ? true ? 0 ? 451823 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/23/00 ? b57e0b51-f42a-4134-a649-1ea955f48e79-0_0-4-1407_20231224214916554.parquet ? false ? true ? 0 ? 449843 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/23/00 ? 66a52bca-6db0-4274-a20e-ecdaac67bf78-0_0-6-1467_20231223214749872.parquet ? false ? true ? 0 ? 449815 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/18/00 ? bcce485f-70ae-4a80-b36c-16e9c9596e07-0_0-4-1368_20231218215310500.parquet ? false ? true ? 0 ? 450011 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/18/00 ? 8d75a76e-ff9b-4724-b485-70170b83ca7b-0_1-4-1381_20231219214031893.parquet ? false ? true ? 0 ? 450147 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/28/00 ? 652f19b9-fa60-40f8-829d-2b540b177b6b-0_1-4-1447_20231229214316193.parquet ? false ? true ? 0 ? 450805 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/28/00 ? d40de711-9bba-4bc1-af48-f2566421c9b4-0_0-4-1443_20231228214506235.parquet ? false ? true ? 0 ? 450801 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/24/00 ? 820b6a82-b2f6-440b-9777-9dd3e2ede1d1-0_2-4-1403_20231225214603565.parquet ? false ? true ? 0 ? 449584 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/24/00 ? 2d0559ef-f850-4955-a410-16abdeda943f-0_2-4-1409_20231224214916554.parquet ? false ? true ? 0 ? 449493 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/30/00 ? cc519a61-6344-438a-9ba6-61127a11c420-0_0-5-1443_20231230214425196.parquet ? false ? true ? 0 ? 450030 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/30/00 ? 669940a4-2c3a-4d22-ac69-4efaf7b292fa-0_1-6-1406_20231231214814370.parquet ? false ? true ? 0 ? 450059 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
? 2023/12/17/00 ? 15568146-a344-445e-b47a-266b9f15a674-0_2-4-1370_20231218215310500.parquet ? false ? true ? 0 ? 450303 ?
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????```
**Stacktrace**
query error while reading the table :
HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://tmp-data/investments_ctr_tbl/2023/12/17/00/15568146-a344-445e-b47a-266b9f15a674-0_2-4-1370_20231218215310500.parquet (offset=0, length=450303): io.trino.hdfs.s3.TrinoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey)
Do let us know if any other info is required regarding this issue.
@Shubham21k Can you try with 0.14.1 once. Also, async services doesn't work with Datasource writer.
I tried to reproduce this but unable to do it. Can you check in case you can enhance the same and reproduce it.
Code here - https://gist.github.com/ad1happy2go/364e66c4fa84229110f28994cc4a277f/edit
@Shubham21k What queries you are trying on this data? Does select * works? For point in time queries, this error is expected in case the commit is not archive but cleaned.
@ad1happy2go
- all the select queries are failing due to this error. As suggested, will try and check if this gets fixed in 0.14.1
- 'async services doesn't work with Datasource writer' - can you elaborate more as we have observed the cleaner & archival taking place with these configurations.
not able to open the code link for reproducing the error shared by you
Is there a work around for this issue. We are facing a similar issue as well.
@Shubham21k Code link here - https://gist.github.com/ad1happy2go/364e66c4fa84229110f28994cc4a277f
Async services are meant to run with streaming workloads like Hudi Streamer, so that table services can run asynchronously and doesn't block the ingestion of next micro batch. Having it with Data source writers (batch writers) doesn't make any sense and inline table services will be kicked in.
@shravanak Which Hudi version you are using? Are you also using insert_overwrite. Can you elaborate.
We are using insert write mode with hudi 0.14.0 I think the file or partition it is referring to missing might be before we upgraded to 0.14.0 which was on 0.12.2
@shravanak That may be the cause probably. Did you faced this issue with other tables also?
@shravanak Are you still facing this issue? Let us know in case you need help here.
hey @ad1happy2go : if this turns out to be MDT data consistency issue, do keep me posted. thanks.
