Python model serverless cluster not working with packages

[ikhudur]

Describe the bug

Can not get run dbt python model on serverless cluster with packages that I specify to install. According to the documentation, submission_method: serverless_cluster accepts a list of packages that should be installed on the running cluster. Furthermore, the environment version on created inner run is always 1.

This is similar to issue https://github.com/databricks/dbt-databricks/issues/1009, except I can not get the dbt python model to use serverless cluster, regardless if submission_method is workflow_job or serverless_cluster

Steps To Reproduce

Use the following config in your dbt python model

version: 2
models:
  - name: my_model
    config:
      materialized: table
      submission_method: serverless_cluster
      packages:
        - xgboost==3.0.5
        - mlflow>=2.14.0

Or as suggested in https://github.com/databricks/dbt-databricks/issues/1009:

models:
  - name: my_model
    config:
      submission_method: workflow_job
      python_job_config:
        additional_task_settings: { "environment_key": "my_env" }
        environments: [
            {
                "environment_key": "my_env",
                "spec": {
                    "environment_version": "4",
                    "dependencies": [
                        "xgboost==3.0.5",
                        "mlflow>=2.14.0",
                    ]
                }
            }
        ]

Expected behavior

Serverless job run starts and runs your code with installed packages

Screenshots and log output

Error when using submission_method: serverless_cluster:

Error creating python run.
   b'{"error_code":"INVALID_PARAMETER_VALUE","message":"Libraries field is not supported for serverless task, please specify libraries in environment.","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"<request-id>","serving_data":""}]}'

Error when using submission_method: workflow_job:

Error creating Workflow.
   b'{"error_code":"INVALID_PARAMETER_VALUE","message":"A task environment can not be provided for notebook task inner_notebook. Please use the %pip magic command to install notebook-scoped Python libraries and Python wheel packages","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"<request-id>","serving_data":""}]}'

System information

The output of dbt --version:

dbt Cloud CLI - 0.40.6

The output of python --version: Python 3.12.3

Additional context

Add any other context about the problem here.

[ikhudur]

Found a workaround, explained it in https://github.com/databricks/dbt-databricks/issues/1009#issuecomment-3360605433

However, documentation has to be updated...