jharringtonCoupons comments

Results 2 comments of


                                            jharringtonCoupons

How to read existing hoodie written data from `S3` using `AWS Glue DynamicFrame` class. Failing with error with below error: An error occurred while calling o84.getDynamicFrame. s3://xxxx/.hoodie/202212312312.commit is not a Parquet file. expected magic number at tail

Can confirm having the same issue (see full stack trace from CloudWatch below). Is there a viable workaround? ```{ "Event": "GlueETLJobExceptionEvent", "Timestamp": 1663789368616, "Failure Reason": "Traceback (most recent call last):\n...

How to read existing hoodie written data from `S3` using `AWS Glue DynamicFrame` class. Failing with error with below error: An error occurred while calling o84.getDynamicFrame. s3://xxxx/.hoodie/202212312312.commit is not a Parquet file. expected magic number at tail

Below are two workable solutions to get around this problem. They both involve adding `additional_options` to the S3 connectionType. - **Exclude folder:** `additional_options={"exclusions": "[\"s3://bucket/folder/.hoodie/*\"]"}` - **Exclude file(s):** `additional_options={"exclusions": "[\"**.commit\",\"**.inflight\",\"**.properties\"]"}` @gtwuser...