databricks
development mode and compute id
Hey, when we set development mode and compute_id to override all of the job's compute to given all purpose cluster which is great for fast testing. But problem is that whenever we deploy our whl artifact which tasks depends on, and every time we deploy we install whl file on global cluster level library, so each deployment requires Uninstall and Restart of cluster in order to install whl for new deployment. Why it cant install it in separate context? It happens even if u provide in library pypi for example pandas 2.0.0, installs it on global level, so in next deployment if u wanna try 2.0.1, u cant, its gona say 2.0.0 already exists. It kills purpose of development mode and compute_id..
This is indeed a behaviour right now which is caused by limitation on cluster side itself. We're looking at ways to improve this behaviour for all purpose clusters. I'll keep this thread updated
