databricks
Define all-purpose job clusters
Is there a way to define all-purpose job clusters? Looking at default_python_job, it seems that the clusters are defined per job. I don't see any way to reference the cluster from another job (at least using ${resources.jobs.job-name.job_clusters} doesn't work. Just using the cluster key doesn't work either.
@ryanhubbell were you able to find a workaround for this? If so, could you please share.
I was not able to do it using Databricks Asset Bundles. I instead defined it via Terraform. In fact, I'm moving all of our stuff over to just using Terraform. It supports a lot more than DAB does, and it doesn't use yaml (not my favorite).
It's possible to create all purpose clusters starting form 0.229.0 version. Here's an example of how to achieve that
https://github.com/databricks/bundle-examples/tree/main/knowledge_base/development_cluster
