[Feature][EngineConn] Enable customized and isolated python environment for Pyspark

[saLeox]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Problem Description

For Python developers, using a customized and isolated Python runtime environment is an indispensable requirement.

You and your colleagues may want to use different versions of python and python packages and don’t want to affect each others’ environment.

Thus it's necessary to support using customized and isolated Python environment in hadoop yarn cluster and achieve this for PySpark.

Description

No response

Use case

No response

solutions

From the official Spark Configuration, we can know that

Property spark.pyspark.driver.python take precedence than PYSPARK_DRIVER_PYTHON if it is set.

Property spark.pyspark.python take precedence than PYSPARK_PYTHON if it is set.

And Property archives is the same as spark.yarn.dist.archives.

We can leverage the global settings module, to let user define the variables.

By default, these variables can be empty, and in that case EnvVar in spark-defaults.conf or spark-env.sh will take precedence.

If user defines, the defined variables will be isolated from other users, and take precedence.

They will be appended into the Spark engine start script, when them can be read to overwrite the default setting in spark-defaults.conf or spark-env.sh.

Anything else

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

[github-actions[bot]]

:blush: Welcome to the Apache Linkis (incubating) community!! We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

WeChat Group:

Mailing Lists:

name	description	Subscribe	Unsubscribe	archive
[email protected]	community activity information	subscribe	unsubscribe	archive