kyuubi icon indicating copy to clipboard operation
kyuubi copied to clipboard

[FEATURE] kyuubi server support to start spark client remotely

Open 942011334 opened this issue 3 years ago • 1 comments

Code of Conduct

Search before asking

  • [X] I have searched in the issues and found no similar issues.

Describe the feature

At present, kyuubi server can only start the spark client under the same machine, and does not support starting the spark client under remote machines, which means that kyuubi server and spark client must be bound to the same machine, but the actual production environment needs to be deployed separately, whether to consider support to start spark client remotely.

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

942011334 avatar Sep 01 '22 04:09 942011334

I am not sure whether it is a generic requirement.

Seems that you can wrap the spark-submit script under $SPARK_HOME/bin to implement it.

wrap the spark-submit with

ssh $REMOTE_HOST "$REMOTE_SPARK_HOME/bin/spark-submit $@" 

turboFei avatar Sep 01 '22 04:09 turboFei