data-diff
data-diff copied to clipboard
Support Postgres materialized views
Describe the bug
I want to compare two materialized views built using DBT, but when I run data-diff --dbt --select <my model>, I get "New model or no access to prod table." The reason is that is that in Postgres, materialized views are not stored in the information_schema.columns schema, which is what is queried by data-diff to find the column:
SELECT column_name, data_type, datetime_precision,
-- see comment for DEFAULT_NUMERIC_PRECISION
CASE
WHEN data_type = 'numeric'
THEN coalesce(numeric_precision, 131072 + 16383)
ELSE numeric_precision
END AS numeric_precision,
CASE
WHEN data_type = 'numeric'
THEN coalesce(numeric_scale, 16383)
ELSE numeric_scale
END AS numeric_scale
FROM information_schema.columns
WHERE table_name = '<my model>' AND table_schema = 'intermediate_prod'
Instead, we'd need to query pg_matviews, as noted in this StackOverflow answer.
I figure we could solve this issue by unioning the first query with another one that queries pg_matviews.
Describe the environment I'm running data-diff v0.11.0 with dbt-core and dbt-postgres 1.7.7 on Postgres 15.