datashare-toolkit
datashare-toolkit copied to clipboard
Destination tables not respecting nullable/required modes specified in schema.json
While interim tables reflect the column modes specified in a supplied schema.json, these modes do not make their way to the final table definition.
To Reproduce Steps to reproduce the behavior:
- deploy function to your project
- copy last_sale config files from examples to
<bucket>/bqds - copy last_sale.csv to bucket as
marketdata.last_sale.csv - despite
last_sale.schema.jsonhaving columns specified asREQUIRED, the ultimate destination table (marketdata.last_sale) shows all columns as beingNULLABLE
Desired behavior is for all columns being mapped directly to the destination table to inherit the mode spec from their schema definition.
Thanks for the report @michaelwsherman
Changes are raised in https://github.com/GoogleCloudPlatform/bq-datashare-toolkit/pull/53, however there appears to be a bug with WriteDisposition where it's truncating column mode and descriptions when using the WRITE_TRUNCATE option.