databricks
Include overrides for targets
https://github.com/databricks/cli/pull/856 introduced target-level overrides for sync. It would also be useful to do the same for include, as this would make the databricks.yml much easier to read/manage and provide some level of reusability.
Thanks for raising this. Can you elaborate on how you would want to factor your configuration to warrant different configuration files per target? Most (if not all) specialization should be doable per target already.
An example would be a job that is unique to a given environment (e.g. a client-specific feature, or a test suite).
Given a baseline config:
# databricks.yml
bundle:
name: example
include:
- "jobs.yml"
targets:
dev:
workspace:
host: abc
prod:
workspace:
host: xyz
And baseline jobs, included by baseline config:
# jobs.yml
resources:
jobs:
data_load:
name: Daily Load
# etc.
The requirement is to add a job, specifically to the dev target. This can be done by inling the required job:
# databricks.yml
bundle:
name: example
include:
- "jobs.yml"
targets:
dev:
workspace:
host: abc
resources:
jobs:
dev_specific:
name: Specific dev job
# etc.
prod:
workspace:
host: xyz
However, including it would be preferred, as this may grow larger over time or be shared in other specific envs:
# databricks.yml
bundle:
name: example
include:
- "jobs.yml"
targets:
dev:
workspace:
host: abc
include:
- dev_jobs.yml
prod:
workspace:
host: xyz
#dev_jobs.yml
resources:
jobs:
dev_specific:
name: Specific dev job
# etc.
The example here is using jobs, but it could be anything really.
