dataform
dataform copied to clipboard
Support custom merge where clauses for Snowflake
Merge performance can be increased significantly when using clustered tables.
To do this you need to be able to specify additional on clauses in the merge statement, however there is currently no way to do that: https://github.com/dataform-co/dataform/blob/master/core/adapters/snowflake.ts#L104
Proposal
Add a new option for incremental merge statements, that allows adding arbitrary on clauses, e.g:
config {
type: "incremental",
mergeClause: "source.clustercolumn = destination.clustercolumn"
}
This can probably deprecate the BigQuery updatePartitionFilter option which is a less flexible option of the above.
see DBT discussion for comparison: https://github.com/fishtown-analytics/dbt/issues/2139