databrickslabs
[BUG]: ucx install Fails on Azure with AWS-linked validation error
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
Attempting to run install ucx on an Azure environment with defaults results in a failure after the Creating dashboards... step, indicating an input validation error on a Cluster. This validation error indicates the aws_attributes.availability attribute is required, which seems erroneous given this is Azure.
Expected Behavior
Expectation is for ucx install to complete successfully
Steps To Reproduce
❯ databricks --version Databricks CLI v0.220.0 ❯ databricks labs installed Name Description Version ucx Unity Catalog Migration Toolkit (UCX) v0.23.1
❯ databricks labs install ucx
Using workspace profile:
Cloud
Azure
Operating System
macOS
Version
latest via Databricks CLI
Relevant log output
❯ databricks labs install ucx
Using workspace profile: <redacted>>
17:09:51 INFO [d.l.ucx.install] Installing UCX v0.23.1
17:09:53 INFO [d.l.ucx.install] UCX v0.23.1 is already installed on this workspace
Do you want to update the existing installation? (default: no): yes
17:09:58 INFO [d.l.ucx.install] Installing UCX v0.23.1
17:09:58 INFO [d.l.ucx.install][installing_components_0] Creating ucx schemas...
17:09:58 INFO [d.l.ucx.install][installing_components_1] Creating dashboards...
Traceback (most recent call last):
File "/redacted/path/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 667, in <module>
workspace_installer.run()
File "/redacted/path/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 191, in run
workspace_installation.run()
File "/redacted/path/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 475, in run
readme_url = self._workflows_installer.create_jobs()
File "/redacted/path/.databricks/labs/ucx/lib/src/databricks/labs/ucx/installer/workflows.py", line 420, in create_jobs
self._deploy_workflow(workflow_name, settings)
File "/redacted/path/.databricks/labs/ucx/lib/src/databricks/labs/ucx/installer/workflows.py", line 522, in _deploy_workflow
new_job = self._ws.jobs.create(**settings)
File "/redacted/path/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/sdk/service/jobs.py", line 5257, in create
res = self._api.do('POST', '/api/2.1/jobs/create', body=body, headers=headers)
File "/redacted/path/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 130, in do
response = retryable(self._perform)(method,
File "/redacted/path/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper
raise err
File "/redacted/path/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper
return func(*args, **kwargs)
File "/redacted/path/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 241, in _perform
raise self._make_nicer_error(response=response, **payload) from None
databricks.sdk.errors.platform.InvalidParameterValue: Cluster validation error: Validation failed for aws_attributes.availability, the value must be present
Error: installer: exit status 1
Capturing some further debugging context: This appears to be down to improper handling of the suffix-matching approach for determining which environment the client is connecting through within the SDK, rather than UCX itself. When no match is found the SDK defaults to AWS. In the two scenarios I ran into:
- Using native token auth in DBR15 results in the
DATABRICKS_HOSTenvironment variable being suffixed with:443causing a failure - Using my local repro a trailing slash on the
hostcli config appeared to cause a similar failure.
In both cases the issue may be worked around by correcting the value of the host config or removing the erroneous aws_attributes.availability property from the generated UCX cluster policy.
