dataform icon indicating copy to clipboard operation
dataform copied to clipboard

How to add `refresh rate` in config for materialized views in dataform ?

Open Rstar1998 opened this issue 3 years ago • 4 comments

Also in addition to this , is there any precise documentation where we can lookup what all key values we can put in config for tables , view etc

Rstar1998 avatar Oct 31 '22 17:10 Rstar1998

We had discussed having that when support for materialized views was first added (https://github.com/dataform-co/dataform/pull/1302) but I'm not sure if it was ever implemented. @BenBirt ?

astorrs avatar Oct 31 '22 18:10 astorrs

No, we haven't added that yet unfortunately.

Probably the best place to see what options are available is: https://docs.dataform.co/reference#ITableConfig

BenBirt avatar Nov 01 '22 09:11 BenBirt

Otherwise known as refresh frequency https://cloud.google.com/bigquery/docs/materialized-views-create#manage_staleness_and_refresh_frequency

Ekrekr avatar Feb 13 '23 08:02 Ekrekr

It is possible to achieve this using using bigquery.additionalOptions since #1335 example config:

config { 
    type: "view",
    materialized: true,
    bigquery: {
        additionalOptions: {
            enable_refresh: "true",
            refresh_interval_minutes: "30"
        }
    }
}

would produce sql with OPTIONS(enable_refresh=true, refresh_interval_minutes=30) One important thing is these are not type/validty checked in anyway, so you would not get any errors until execution (i.e. it would compile) Here's our documentation and available options for materialized views

canberkkoparal avatar Feb 15 '23 13:02 canberkkoparal