google-cloud-python `--properties-file` flag not available job classes

The flag --properties-file=PROPERTIES_FILE from gcloud dataproc jobs submit pyspark [doc] does not seem available through the PySparkJob python class.

This flag is very handy to add properties that depend on the cluster initialisation actions, such as the path of library installed during initialisation.

Apr 01 '23 07:04 d0choa

I'm going to transfer this issue to the google-cloud-python repository as we are preparing to move the code for google-cloud-dataproc to that repository in the next 1-2 weeks.

Apr 17 '23 20:04 parthea

Thanks for submitting this isse, @d0choa ! To be clear, what you're suggesting is that it would be useful to have an option to specify a properties file rather than a list of key/value pairs?

Jun 24 '24 19:06 vchudnov-g

The properties field is the recommended way to provide properties when using the Python Client Library. When using the gcloud CLI, there is a strong argument to have a convenience method for reading properties from a file but for client libraries we would need a strong reason to add these helpers to ensure that we're not introducing clutter. This seems like it's better handled at the application layer. I'm going to close this issue but please feel free to open a new feature request with more information.

Sep 03 '24 18:09 parthea

thanks!

Sep 04 '24 08:09 d0choa