cli icon indicating copy to clipboard operation
cli copied to clipboard
verified publisher icon databricks

DAB deployments containing AI/BI dashboard resources fail when using the Databricks Web Terminal

Open liamperritt opened this issue 10 months ago • 0 comments

Describe the issue

Since the Public Preview of Git integration for AI/BI dashboards was enabled, deployments of DABs that contain AI/BI dashboard resources now fail when using the Databricks Web Terminal with the following error message:

Error: exit status 1

Error: Invalid function argument

  on bundle.tf.json line 19, in resource.databricks_dashboard.my_dashboard:
  19:         "serialized_dashboard": "${file(\"/Workspace/Users/[email protected]/my-dab-project/src/my_dashboard_source.lvdash.json\")}",
    ├────────────────
    │ while calling file(path)

Invalid value for "path" parameter: no file exists at
"/Workspace/Users/[email protected]/my-dab-project/src/my_dashboard_source.lvdash.json";
this function works only with files that are distributed as part of the
configuration source code, so if this file will be created by a resource in
this configuration you must instead obtain this result from an attribute of
that resource.

Prior to the enablement of the Public Preview, this worked fine.

This issue does not occur for other workspace source file types referenced by DAB resources, such as notebooks (even when the .py or .sql file extension is included in the referenced file path), however, it no longer work for dashboard source files (with the .lvdash.json file extension), as those files are no longer cloned to the workspace Git Folder as Files, but as Dashboards.

I can get around the issue by setting the source_linked_deployment preset to false in my databricks.yml file, however this would mean that I can no longer take advantage of the "source linked deployment" feature, which is one of the main benefits of deploying DABs via the Web Terminal, as it lets the user deploy all resources once and then interactively test changes to source code files without needing to redeploy the bundle.

Configuration

./resources/my_dashboard.yml:

---
resources:
  dashboards:
    my_dashboard:
      display_name: My Dashboard
      file_path: ../src/my_dashboard_source.lvdash.json
      warehouse_id: ${var.warehouse_id}

./src/my_dashboard_source.lvdash.json:

{
    "pages": [
        {
            "name": "23efa2e4",
            "displayName": "My Dashboard"
        }
    ]
}

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

  1. Ensure that the Public Preview of Git integration for AI/BI dashboards is enabled on your Databricks workspace
  2. Clone a Git Folder in your workspace that contains a bundle with an AI/BI dashboard resource (as outlined in the above configuration)
  3. Create and start a Single User-mode compute cluster that is running DBR >= 15.4 LTS
  4. Open the Web Terminal
  5. Change directory into your Git folder (e.g. by running cd /Workspace/Users/[email protected]/my-dab-project)
  6. Run databricks bundle deploy
  7. See error

Expected Behavior

Bundle resources, including the dashboard, should be successfully deployed.

Actual Behavior

Deploy command fails with the afore-mentioned Invalid value for "path" parameter error.

OS and CLI version

OS: DBR 15.4 LTS CLI: v0.248.0

Is this a regression?

It's not a regression of the CLI, but a regression of the Databricks workspace behaviour when interacting with the Databricks CLI, introduced by the Public Preview of Git integration for AI/BI dashboards.

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

...
00:39:25 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/[REDACTED]/my-dab-project/src/my_dashboard_source.lvdash.json&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "modified_at": 1745368742107,
<   "object_id": 1399047106086511,
<   "object_type": "DASHBOARD",
<   "path": "/Workspace/Users/[REDACTED]/my-dab-project/src/my_dashboard_source... (14 more bytes)",
<   "resource_id": "01f01fdb1afa17f29dcf542d1cc5a6cd"
< } pid=2372 sdk=true
00:39:25 Debug: Apply pid=2372 mutator=PythonWrapperWarning
00:39:25 Debug: Apply pid=2372 mutator=artifacts.Prepare
00:39:25 Info: No local tasks in databricks.yml config, skipping auto detect pid=2372 mutator=artifacts.Prepare
00:39:25 Debug: Apply pid=2372 mutator=apps.Validate
00:39:25 Debug: Apply pid=2372 mutator=ValidateTargetMode
00:39:25 Debug: Apply pid=2372 mutator=ValidateSharedRootPermissions
00:39:25 Debug: Apply pid=2372 mutator=metadata.AnnotateJobs
00:39:25 Debug: Apply pid=2372 mutator=metadata.AnnotatePipelines
00:39:25 Debug: Apply pid=2372 mutator=terraform.Initialize
00:39:25 Debug: DATABRICKS_TF_EXEC_PATH is not defined pid=2372 mutator=terraform.Initialize
00:39:28 Debug: Using Terraform at /Workspace/Users/[REDACTED]/my-dab-project/.databricks/bundle/dev/bin/terraform pid=2372 mutator=terraform.Initialize
00:39:28 Debug: DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=2372 mutator=terraform.Initialize
00:39:28 Debug: Environment variables for Terraform: DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_CLI_PATH, DATABRICKS_AUTH_TYPE, HOME, PATH, DATABRICKS_USER_AGENT_EXTRA pid=2372 mutator=terraform.Initialize
00:39:28 Debug: Apply pid=2372 mutator=scripts.postinit
00:39:28 Debug: No script defined for postinit, skipping pid=2372 mutator=scripts.postinit
00:39:28 Debug: Apply pid=2372 mutator=fast_validate(readonly)
00:39:28 Debug: ApplyParallel pid=2372 mutator=fast_validate(readonly) mutator=validate:job_cluster_key_defined
00:39:28 Debug: ApplyParallel pid=2372 mutator=fast_validate(readonly) mutator=validate:job_task_cluster_spec
00:39:28 Debug: ApplyParallel pid=2372 mutator=fast_validate(readonly) mutator=validate:SingleNodeCluster
00:39:28 Debug: ApplyParallel pid=2372 mutator=fast_validate(readonly) mutator=validate:artifact_paths
00:39:28 Info: Phase: build pid=2372
00:39:28 Debug: Apply pid=2372 mutator=scripts.prebuild
00:39:28 Debug: No script defined for prebuild, skipping pid=2372 mutator=scripts.prebuild
00:39:28 Debug: Apply pid=2372 mutator=artifacts.Build
00:39:28 Debug: Apply pid=2372 mutator=scripts.postbuild
00:39:28 Debug: No script defined for postbuild, skipping pid=2372 mutator=scripts.postbuild
00:39:28 Debug: Apply pid=2372 mutator=ResolveVariableReferences
00:39:28 Debug: Apply pid=2372 mutator=ResolveVariableReferences(resources)
00:39:28 Info: Phase: deploy pid=2372
00:39:28 Debug: Apply pid=2372 mutator=scripts.predeploy
00:39:28 Debug: No script defined for predeploy, skipping pid=2372 mutator=scripts.predeploy
00:39:28 Debug: Apply pid=2372 mutator=lock:acquire
00:39:28 Info: Skipping; locking is disabled pid=2372 mutator=lock:acquire
00:39:28 Debug: Apply pid=2372 mutator=terraform:state-pull
00:39:28 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/terraform.tfstate&return_export_info=true
< HTTP/2.0 404 Not Found
< {
<   "error_code": "RESOURCE_DOES_NOT_EXIST",
<   "message": "Path (/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/terraform.tfs... (20 more bytes)"
< } pid=2372 mutator=terraform:state-pull sdk=true
00:39:28 Debug: non-retriable error: Path (/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/terraform.tfstate) doesn't exist. pid=2372 mutator=terraform:state-pull sdk=true
00:39:28 Info: Remote state file does not exist. Using local Terraform state. pid=2372 mutator=terraform:state-pull
00:39:28 Debug: Apply pid=2372 mutator=CheckDashboardsModifiedRemotely
00:39:28 Debug: Apply pid=2372 mutator=deploy:state-pull
00:39:28 Info: Opening remote deployment state file pid=2372 mutator=deploy:state-pull
00:39:28 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/deployment.json&return_export_info=true
< HTTP/2.0 404 Not Found
< {
<   "error_code": "RESOURCE_DOES_NOT_EXIST",
<   "message": "Path (/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/deployment.js... (18 more bytes)"
< } pid=2372 mutator=deploy:state-pull sdk=true
00:39:28 Debug: non-retriable error: Path (/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/deployment.json) doesn't exist. pid=2372 mutator=deploy:state-pull sdk=true
00:39:28 Info: Remote deployment state file does not exist pid=2372 mutator=deploy:state-pull
00:39:28 Debug: Apply pid=2372 mutator=ValidateGitDetails
00:39:28 Debug: Apply pid=2372 mutator=artifacts.CleanUp
00:39:28 Debug: POST /api/2.0/workspace/delete
> {
>   "path": "/Workspace/Users/[REDACTED]/.bundle/my_dab_projectdev/artifacts/.internal",
>   "recursive": true
> }
< HTTP/2.0 404 Not Found
< {
<   "error_code": "RESOURCE_DOES_NOT_EXIST",
<   "message": "Path (/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/artifacts/.internal... (16 more bytes)"
< } pid=2372 mutator=artifacts.CleanUp sdk=true
00:39:28 Debug: non-retriable error: Path (/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/artifacts/.internal) doesn't exist. pid=2372 mutator=artifacts.CleanUp sdk=true
00:39:28 Debug: failed to delete /Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/artifacts: file does not exist: /Workspace/Users/[REDACTED]/.bundle/my_dab_projectdev/artifacts/.internal pid=2372 mutator=artifacts.CleanUp
00:39:28 Debug: POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/artifacts/.internal"
> }
< HTTP/2.0 200 OK
< {} pid=2372 mutator=artifacts.CleanUp sdk=true
00:39:28 Debug: Apply pid=2372 mutator=libraries.ExpandGlobReferences
00:39:28 Debug: Apply pid=2372 mutator=CheckForSameNameLibraries
00:39:28 Debug: Apply pid=2372 mutator=SwitchToPatchedWheels
00:39:28 Debug: Apply pid=2372 mutator=libraries.Upload
00:39:28 Debug: Apply pid=2372 mutator=TransformWheelTask
00:39:28 Debug: Apply pid=2372 mutator=files.Upload
Source-linked deployment is enabled. Deployed resources reference the source files in your working tree instead of separate copies.
00:39:28 Debug: Apply pid=2372 mutator=deploy:state-update
00:39:28 Info: Loading deployment state from /Workspace/Users/[REDACTED]/my-dab-project/.databricks/bundle/dev/deployment.json pid=2372 mutator=deploy:state-update
00:39:28 Info: No deployment state file found pid=2372 mutator=deploy:state-update
00:39:28 Debug: Apply pid=2372 mutator=deploy:state-push
00:39:28 Info: Writing local deployment state file to remote state directory pid=2372 mutator=deploy:state-push
00:39:28 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/deployment.json?overwrite=true
> {
>   "cli_version": "0.248.0",
>   "files": null,
>   "id": "33386bb1-1a04-4eb7-9af1-7b3ca622a15c",
>   "seq": 1,
>   "timestamp": "2025-04-23T00:39:28.687717956Z",
>   "version": 1
> }
< HTTP/2.0 404 Not Found
< {
<   "message": "Request failed for POST /Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/s... (20 more bytes)"
< } pid=2372 mutator=deploy:state-push sdk=true
00:39:28 Debug: non-retriable error: Request failed for POST /Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/deployment.json pid=2372 mutator=deploy:state-push sdk=true
00:39:29 Debug: POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state"
> }
< HTTP/2.0 200 OK
< {} pid=2372 mutator=deploy:state-push sdk=true
00:39:29 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Users/[REDACTED]/.bundle/my_dab_project/dev/state/deployment.json?overwrite=true
> {
>   "cli_version": "0.248.0",
>   "files": null,
>   "id": "33386bb1-1a04-4eb7-9af1-7b3ca622a15c",
>   "seq": 1,
>   "timestamp": "2025-04-23T00:39:28.687717956Z",
>   "version": 1
> }
< HTTP/2.0 200 OK pid=2372 mutator=deploy:state-push sdk=true
00:39:29 Debug: Apply pid=2372 mutator=ApplyWorkspaceRootPermissions
00:39:29 Debug: Apply pid=2372 mutator=terraform.Interpolate
00:39:29 Debug: Apply pid=2372 mutator=terraform.Write
00:39:29 Debug: dashboard normalization diagnostic: unknown field: permissions pid=2372 mutator=terraform.Write
00:39:29 Debug: pipeline normalization diagnostic: unknown field: permissions pid=2372 mutator=terraform.Write
00:39:29 Debug: pipeline normalization diagnostic: unknown field: permissions pid=2372 mutator=terraform.Write
00:39:29 Debug: pipeline normalization diagnostic: unknown field: permissions pid=2372 mutator=terraform.Write
00:39:29 Debug: job normalization diagnostic: unknown field: permissions pid=2372 mutator=terraform.Write
00:39:29 Debug: job normalization diagnostic: unknown field: permissions pid=2372 mutator=terraform.Write
00:39:29 Debug: Apply pid=2372 mutator=check-running-resources
00:39:29 Debug: Apply pid=2372 mutator=terraform.Plan
00:39:44 Debug: Error: exit status 1

Error: Invalid function argument

  on bundle.tf.json line 19, in resource.databricks_dashboard.my_dashboard:
  19:         "serialized_dashboard": "${file(\"/Workspace/Users/[REDACTED]/my-dab-project/src/my_dashboard_source.lvdash.json\")}",
    ├────────────────
    │ while calling file(path)

Invalid value for "path" parameter: no file exists at
"/Workspace/Users/[REDACTED]/my-dab-project/src/my_dashboard_source.lvdash.json";
this function works only with files that are distributed as part of the
configuration source code, so if this file will be created by a resource in
this configuration you must instead obtain this result from an attribute of
that resource.
 pid=2372 mutator=terraform.Plan
00:39:44 Debug: Apply pid=2372 mutator=lock:release
00:39:44 Info: Skipping; locking is disabled pid=2372 mutator=lock:release
Error: exit status 1

Error: Invalid function argument

  on bundle.tf.json line 19, in resource.databricks_dashboard.my_dashboard:
  19:         "serialized_dashboard": "${file(\"/Workspace/Users/[REDACTED]/my-dab-project/src/my_dashboard_source.lvdash.json\")}",
    ├────────────────
    │ while calling file(path)

Invalid value for "path" parameter: no file exists at
"/Workspace/Users/[REDACTED]/my-dab-project/src/my_dashboard_source.lvdash.json";
this function works only with files that are distributed as part of the
configuration source code, so if this file will be created by a resource in
this configuration you must instead obtain this result from an attribute of
that resource.


00:39:44 Debug: failed execution pid=2372 exit_code=1
00:39:44 Debug: no telemetry logs to upload pid=2372

liamperritt avatar Apr 23 '25 00:04 liamperritt