databricks
Bundle variables missing features
Hey team, I'm trying to improve Dx for my team by adopting this extension. So far, the team really likes it. I have found, however, a few gaps regarding handling of bundle variables:
- The databricks cli supports env vars with the prefix
BUNDLE_VAR_(a-la terraform) which we've used as part of our CI/CD pipeline. We fetch some values from SSM parameters on AWS. The extension, however, ignores these.
Another related papercut is that if some variables are missing values (precisely because they haven't been read from the env), then the variables view becomes empty (the bundle is presumably in an invalid state?). It would be really nice to have UI indication of what variables aren't set properly rather than just failing entirely.
- While looking to refactor and simplify, I found that the yaml spec supports complex variable types. This, again, isn't working in the UI, which expects the value to be of string type.
Hey, thanks for the feedback.
Complex vars are indeed bugged out, we will need to fix it.
Regarding the missing values for variables - right now if you run databricks bundle validate on a bundle that has variables without any default value, the CLI will show this error:
Error: no value assigned to required variable hello. Assignment can be done using "--var", by setting the BUNDLE_VAR_hello environment variable, or in .databricks/bundle/<target>/variable-overrides.json file
The extension right now relies on the .databricks/bundle/<target>/variable-overrides.json file, so if you set the right vars there, it should work as expected.
If you don't set the values for variables that are used in your resources, the extension won't be able to render the "resource explorer" panel, but the variables panel should still work:
Blanking out the Resource Explorer in this case isn't very friendly. It gives the impression that the extension is buggy and broken without providing UI to explain why it's empty.
It would be great if the extension could use those CLI error messages to add some UI (red colour, warning icon.. ) for the variable missing a value. Also display something in the empty panel: at least an error message saying the bundle isn't valid.
I'm aware of that json file option. It's just an annoying papercut since now I need to maintain two ways to output the values. Hence, my feature request is to add support for env vars and close this gap with the CLI 🙂