dataform icon indicating copy to clipboard operation
dataform copied to clipboard

Support custom merge where clauses for Snowflake

Open lewish opened this issue 5 years ago • 1 comments

Merge performance can be increased significantly when using clustered tables.

To do this you need to be able to specify additional on clauses in the merge statement, however there is currently no way to do that: https://github.com/dataform-co/dataform/blob/master/core/adapters/snowflake.ts#L104

Proposal

Add a new option for incremental merge statements, that allows adding arbitrary on clauses, e.g:

config {
  type: "incremental",
  mergeClause: "source.clustercolumn = destination.clustercolumn"
}

This can probably deprecate the BigQuery updatePartitionFilter option which is a less flexible option of the above.

lewish avatar Oct 07 '20 11:10 lewish

see DBT discussion for comparison: https://github.com/fishtown-analytics/dbt/issues/2139

rumbin avatar Oct 08 '20 19:10 rumbin