dbt-clickhouse icon indicating copy to clipboard operation
dbt-clickhouse copied to clipboard

append_new_columns doesn't allow you to add new columns to a model

Open hillbilly991 opened this issue 1 year ago • 0 comments

Describe the bug

on_schema_change is set to append_new_columns but when I add new columns, I get a NUMBER_OF_COLUMNS_DOESNT_MATCH error:

Database Error in model xxxxxxxxx (xxxxxxxxx)
--
  | HTTPDriver for xxxxxxxxx received ClickHouse error code 20
  | Code: 20. DB::Exception: Number of columns doesn't match (source: 15 and result: 14). (NUMBER_OF_COLUMNS_DOESNT_MATCH) (version 24.10.1.11366 (official build))
  | compiled code at xxxxxxxxxxx

I fixed this by going into Clickhouse directly and writing an ALTER TABLE query to add the new column. Then when I re materialised the asset, it worked

Steps to reproduce

  1. create an incremental model with on_schema_change = append_new_columns
  2. materialise it
  3. add a new column
  4. materialise it again and you should see the above error

Expected behaviour

When on_schema_change is set to append_new_columns and you add a new column to an already materialised model then it should allow you to add those columns

hillbilly991 avatar Mar 21 '25 14:03 hillbilly991