sqlmesh
sqlmesh copied to clipboard
`table_diff` on Athena fails for Iceberg tables
Based on issue a user reported in Slack: https://tobiko-data.slack.com/archives/C044BRE5W4S/p1744125717966919?thread_ts=1742833178.418699&cid=C044BRE5W4S
The table_diff creates a temporary table in order to hold the diff results.
In the case of Athena, this temporary table will be created as Hive by default. However, if the source table is Iceberg, the temporary table also needs to be created as Iceberg because some data types are not compatible between the two.
Timestamp fields are a good example:
pyathena.error.OperationalError: NOT_SUPPORTED: Unsupported Hive type: timestamp(6) with time zone. You may need to manually clean the data at location 's3://athena/sqlmesh/tables/1543f306-59e7-4310-b2b0-f75e0558761a' before retrying. Athena will not delete data in your account.