Submit python (pyspark) models to Google Dataproc

[candalfigomoro]

With dbt it is possible to run pyspark jobs using Google Dataproc, both in "cluster" and "serverless" mode (see the "BigQuery" section in https://docs.getdbt.com/docs/build/python-models#specific-data-platforms).

Is this something that is supported in sqlmesh?

[eakmanrq]

@candalfigomoro Thanks for making this request! This is not something we support today but we are planning on adding it soon. Will likely start looking at it next week.

[eakmanrq]

@candalfigomoro Although I was originally hoping to have time to do this soon it now looks like it will be a bit until we can look into this. It is something we want to support but just not sure on the timeline.