dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

`dbt run` fails when the `location_root` is updated

Open dcaribou opened this issue 3 years ago • 0 comments

Describe the bug

dbt run fails for models materialized as tables when the location_root is updated.

Steps To Reproduce

  1. Define model_a as materialized as a table
  2. Define location_root for model (i.e s3://my-bucket/location1)
  3. Run model
  4. Update model's location_root to a different path (i.e s3://my-bucket/location2)
  5. Run model

Error is thrown

The location of the existing table hive_metastore.db.model_a is s3://my-bucket/location1/modela. It doesn't match the specified location s3://my-bucket/location2/modela.

Expected behavior

The storage location is updated without an error. This is in-fact the behaviour when the model is defined with an incremental materialization and run with the full-refresh flag. The incremental materialization is working because a drop table if exists command runs before the create or replace table as, which is not the case for the table materialization

System information

Core:
  - installed: 1.1.1
  - latest:    1.1.1 - Up to date!

Plugins:
  - databricks: 1.1.0 - Update available!
  - spark:      1.1.0 - Up to date!

The operating system you're using: macOS Monterrey - 12.4

The output of python --version: Python 3.9.12

dcaribou avatar Jul 25 '22 14:07 dcaribou