Default authentication chain doesn't work on Databricks Apps

[nchammas]

I am developing a Streamlit app for deployment to Databricks Apps. I am having trouble getting the same authentication flow to work both locally as well as on Databricks Apps.

Consider the following snippet:

from databricks import sql

with sql.connect(
    server_hostname=host,
    http_path=http_path,
    catalog=catalog,
) as connection:
    with connection.cursor() as cursor:
        cursor.execute("SELECT 1")
        result = cursor.fetchall()

This works fine locally.

However, when I deploy this to Databricks Apps, it just hangs forever.

I figured out that to get this snippet to work on Databricks Apps, it needs this tweak:

from databricks.sdk.core import Config

with sql.connect(
    server_hostname=host,
    http_path=http_path,
    catalog=catalog,
    credentials_provider=lambda: Config().authenticate,
) as connection:
    ...

This works on Databricks Apps, but now when I try to run it locally I get a new error:

ValueError: default auth: cannot configure default credentials, please check 
https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication
to configure credentials for your preferred authentication method.

OK great. I have one thing that works locally but not on Databricks Apps, and another that works the other way around.

Using the tip shared here, I combined these into a hybrid approach that seems to work both locally and on Databricks Apps:

from databricks.sdk.core import Config, databricks_cli

def universal_databricks_credentials_provider():
    try:
        return databricks_cli(Config(host=host))
    except:
        return Config().authenticate

with sql.connect(
    server_hostname=host,
    http_path=http_path,
    catalog=catalog,
    credentials_provider=universal_databricks_credentials_provider,
) as connection:
    ...

But this is all very awkward and unpleasant.

I think the default authentication chain that this library uses should work on Databricks Apps so that I don't have to specify credentials_provider.