dataform
dataform copied to clipboard
Can't change partition spec for BigQuery tables
bigquery error: Cannot replace a table with a different partitioning spec. Instead, DROP the table, and then recreate it.
Current workaround is to manually delete the table from within BigQuery.
Options:
- Always drop and recreate tables
- Drop and recreate tables only when full-refresh is defined
- Drop and recreate tables when the partition spec (or cluster by spec when added) exists on the table definition, or on the existing table in the warehouse.
This will significantly increase the number of statements we need to execute for BigQuery, so I propose we do option 3. It should be possible to get the current partition by spec when we compute warehouse state, and use this in the SQL generation logic.