dataform-co
Override database config is not applied to Auto generated assertions
About this https://docs.dataform.co/guides/assertions#auto-generated-assertions .
I am using dataform run --vars option to embed a new database(GCP project_id) into each sqlx files to
override the default_database in my CI/CD.
Not from code base but from behavior base, I found that override is not applied to auto generated assertions.
Is it an expected behavior or just a bug?
For example:
dataform.json
{
"warehouse": "bigquery",
"defaultSchema": "dataform",
"assertionSchema": "dataform_assertions",
"defaultDatabase": "my-dev-project"
}
some_query.sqlx
config {
type: "table",
database: dataform.projectConfig.vars.embed_my_prd_database,
assertions: {
uniqueKey: ["user_id"],
nonNull: ["user_id", "customer_id"],
rowConditions: [
'signup_date is null or signup_date > "2019-01-01"',
'email like "%@%.%"'
]
}
}
select ...
If you run dataform run --actions=some_query --vars=embed_my_prd_database=my-prd-project
expect: success
but I got followings because I was using SA credential of my-prd-project not of my-dev-project
Assertion failed: dataform_assertions.some_query_assertions_uniqueKey_0
>
> create or replace view `my-dev-project.dataform_assertions.some_query_assertions_uniqueKey_0` as
A more elegant solution that we have planned for this is CLI flags for dataform.json parameters.
Config option was not the fix, https://github.com/dataform-co/dataform/pull/1602 demonstrates however confirms that this was not an issue with Core's dataform.json interpretation.
Will create a new PR to add these to the tests, and fix the underlying cause.
I'm a bit stumped by this, but I've narrowed down the issue and it is clear it only happens in the CLI.
Adding more tests for Dataform Core (https://github.com/dataform-co/dataform/pull/1603) demonstrates that:
- Variables are correctly loaded from the project config override.
- Variables are correctly loaded as the database for tables.
- Variables are correctly loaded as the database for auto assertions.
Adding more tests for Dataform CLI (https://github.com/dataform-co/dataform/pull/1604) demonstrates that:
- Variables are correctly loaded from the project config override.
- Variables are correctly loaded as the database for tables.
- However for auto assertions, the variable is not applied.
Also running into this. We set the database in the config block of each .sqlx and assertions in the config block don't work
