sodadata
Support default GCP auth mechanism for BigQuery
Is your feature request related to a problem? Please describe. When using GCP runtimes to run soda scan, there's quite some workaround to be done to authenticate as there's no support of default auth gcp mechanism on Soda side.
Soda is expecting a json service account file path or value here and create a Credentials object used to auth against BigQuery.
In GCP runtime environment, the Credentials object can be automatically retrieved as the following :
import google.auth
credentials, project = google.auth.default()
AFAIK, there's no way to serialize this one as a .json , so the workaround is to store a .json key of the corresponding default service account used in these services to secret manager and retrieve it at runtime to be able to pass it for Soda.
Besides, this mechanism lookup automatically if GOOGLE_APPLICATION_CREDENTIALS is set and retrieve the json path
so it could simplify code on Soda and avoid having a custom env like BIG_QUERY_ACCESS like it is today and would be more standard for local development too.
Documentation about the default auth mechanism here
Describe the solution you'd like
Soda should be able to do a default auth mechanism with the above snippet and use standard env GOOGLE_APPLICATION_CREDENTIALS.
I'm happy to help for a PR - but I think some part of the BigQuery integration code is not public 🤔 ?
I was confused as I couldn't find the current env used for Soda on BigQuery Connection but I think the docs is not up to date, it refers BIG_QUERY_ACCESS here but in the code it's BIGQUERY_ACCOUNT_INFO. So should be pretty straight forward to implement this.
You can now (soda sql >= 2.1.2) provide use_context_auth and project_id to your warehouse.yaml. This will use context authentication https://docs.soda.io/soda/warehouse_types.html#gcp-big-query
