cli icon indicating copy to clipboard operation
cli copied to clipboard

DAB unable to validate container local `file:///path-to-jar`

Open sshpuntoff opened this issue 2 months ago • 2 comments

Describe the issue

The CLI treats file:// URIs in job libraries as local paths and attempts to validate they exist in the bundle directory, causing deployment to fail even though these URIs reference files on the cluster's runtime filesystem.

Configuration

resources:
  jobs:
    my_job:
      tasks:
        - task_key: my_task
          libraries:
            - jar: file:///opt/spark/jars/my-library.jar

Steps to reproduce the behavior

  1. Create a bundle with a job that references a file:// URI in libraries
  2. Run databricks bundle deploy
  3. See error: file doesn't exist file:///opt/spark/jars/my-library.jar

Expected Behavior

The CLI should pass file:// URIs through to the Jobs API without validation or upload. These URIs reference files already present on the cluster's filesystem (via init scripts, container images, or pre-installed dependencies), and the Jobs API supports this pattern.

Actual Behavior

The CLI validates file:// paths as if they were local files in the bundle directory and fails deployment with "file doesn't exist" error.

OS and CLI version

macOS, CLI version [run databricks --version to fill this in]

Is this a regression?

Unknown - this may never have worked.

Debug Logs

Error: file doesn't exist file:///opt/spark/jars/my-library.jar
  at resources.jobs.my_job.tasks[0].libraries[0].jar

Note: The issue appears to be in bundle/libraries/local_path.go where file:// URIs are incorrectly treated as local paths.

sshpuntoff avatar Nov 07 '25 05:11 sshpuntoff

Suggested cli change, that allows validation to behave as expected. https://github.com/databricks/cli/pull/3892

sshpuntoff avatar Nov 07 '25 05:11 sshpuntoff

@pietern Just wanted to follow up here. I am open to adding a feature flag, etc. It is unclear if this current behavior is "Critical Path" or something we can change. Thanks!

sshpuntoff avatar Nov 13 '25 17:11 sshpuntoff

I see the code was merged! Marking this as resolved!

sshpuntoff avatar Dec 02 '25 15:12 sshpuntoff