Support newer versions of databricks-sdk

[hauntsaninja]

Currently this pins databricks-sdk==0.17.0 for unclear reasons: https://github.com/databricks/dbt-databricks/blob/ac4dee7610a2a7d5ff9514a58f315699643242da/setup.py#L61

The pin was introduced in https://github.com/databricks/dbt-databricks/pull/572

[benc-db]

Working on this, though it has temporarily taken a back seat due to needing to respond to some platform breaking changes last week; the reason it was pinned is because 0.18.0 changes Azure OAuth behavior. The goal (which will release in 1.9) is to move to 0.29.0. It is not safe to unpin entirely because SDK is still in Beta and is still releasing breaking changes.

[nysthee]

If I understand correctly, this is related to this:https://github.com/databricks/dbt-databricks/issues/640. We are facing an issue where we hit the one-hour token lifetime limit (we generate the token in our Azure DevOps pipeline). We believe the M2M authentication flow supported below SDK 0.17.0 is fixing this. When would this be expected, or are there alternatives to look into?

[benc-db]

@nysthee we've been waiting for other changes in our dependency change, but we have a couple of PRs to address this, so I'm hopeful we'll resolve it soon.

[jonmellman]

Looks like this is resolved because v1.10.0a2 consumes databricks-sdk==0.46.0, is that right (source)? Do you know when v1.10 will be released?

[benc-db]

probably next week.