dataform icon indicating copy to clipboard operation
dataform copied to clipboard

Can't change partition spec for BigQuery tables

Open lewish opened this issue 5 years ago • 0 comments

bigquery error: Cannot replace a table with a different partitioning spec. Instead, DROP the table, and then recreate it.

Current workaround is to manually delete the table from within BigQuery.

Options:

  • Always drop and recreate tables
  • Drop and recreate tables only when full-refresh is defined
  • Drop and recreate tables when the partition spec (or cluster by spec when added) exists on the table definition, or on the existing table in the warehouse.

This will significantly increase the number of statements we need to execute for BigQuery, so I propose we do option 3. It should be possible to get the current partition by spec when we compute warehouse state, and use this in the SQL generation logic.

lewish avatar Feb 03 '20 15:02 lewish