[ZEPPELIN-3994] Notebook serving

[Leemoonsoo]

What is this PR for?

This PR implements notebook serving explained here.

While this PR is already quite large, front-end implementation will have separate PR. This PR is work in progress.

How to run

1. Prepare kubernetes cluster
2. Prepare persistentVolumeClaim with name 'task-context-volume-claim' in your kubernetes cluster
3. Build Zeppelin docker image (follow instructions here
4. Deploy api router by running kubectl apply -f k8s/serving-api-router/serving-api-router.yaml
5. Start Zeppelin on Kubernetes by running kubectl apply -f k8s/zeppelin-server.yaml
6. Port forward to the api router (e.g. kubectl port-forward <zeppelin-serving-api-router pod name> 8080:80) and browse localhost:8080

*Note: This branch does not include front-end implementation. To try with front-end, checkout this branch.

Summary of changes

To be updated

What type of PR is it?

Feature

Todos

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-3994

How should this be tested?

• First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
• Strongly recommended: add automated unit tests for any new or changed behavior
• Outline any manual steps to test the PR here.

Questions:

• Does the licenses files need update? yes
• Is there breaking changes for older versions? no
• Does this needs documentation? yes

[amit-shahi]

Which version of Zeppelin do I run on Kubernetes to work with Flink 1.11.3? Is there a working Zeppelin Kubernetes YAML that works with Flink?

[zjffdu]

@amit-shahi You can download latest zeppelin 0.9 which support flink 1.11, but we haven't verified it on k8s

[amit-shahi]

@zjffdu Can I use the Docker Image apache/zeppelin:0.9.0 (https://hub.docker.com/r/apache/zeppelin/tags?page=1&ordering=last_updated)? Also, I have flink running on another POD on the same K8 cluster. Where do I configure Zeppelin to connect to my existing Flink Job Manager POD so Zeppelin can submit JOBS through it's UI?

[zjffdu]

I am afraid the docker image of 0.9 is still using 0.9.0-preview2, we are working on it to upgrade to 0.9.0 But it still support flink 1.11 IIRC.

[amit-shahi]

@zjffdu Okay. Which Zeppelin version I should from Docker? I have flink running on another POD on the K8s cluster. Where do I configure Zeppelin to connect to my existing Flink Job Manager POD so Zeppelin can submit JOBS through it's UI?

[zjffdu]

@amit-shahi You can refer the document here, remote mode is the way you should use I believe, https://app.gitbook.com/@jeffzhang/s/flink-on-zeppelin/execution-mode/remote-mode

[amit-shahi]

@zjffdu Thanks. I'm accessing Kubernetes on the Windows system. As per the above documentation, Do I need to set FLINK_HOME to my local Flink directory (not on Kubernetes)? Can I bypass this so it can use Kubernetes Flink's FLINK_HOME reference instead? Where do I configure zeppelin so it can use K8's Flink settings for FLINK_HOME? What's the right way to do that?

[zjffdu]

flink should be put in the same host of zeppelin. FLINK_HOME point to that location.

[amit-shahi]

@zjffdu I'm not sure if I got that. Let's say on Kubernetes, Flink's Job Manager is installed on POD1 and Task Manager is installed on POD2. When we install Zeppelin through YAML, it gets installed on POD 3, How do I use POD1's (Job Manager) FLINK_HOME location/reference in POD 3 so Zeppelin server can connect to existing Flink installation?

[zjffdu]

You need to put flink in pod3 and specify FLINK_HOME to this path. Then specify flink.execution.remote.host and flink.execution.remote.port as the JobManager rest api address

[amit-shahi]

@zjffdu I made the changes as suggested.

When I run default Flink Job in Zeppelin, this is what I see-

or

This also causes the Flink Pod Restart which gets created by zeppelin-server.

I'm using zeppelin 0.9.0 from Docker Hub, and Flink 1.11.3. I'm not sure what else I need to change/fix, so I can submit Flink SQL job from zeppelin to main POD1 Flink cluster.

[zjffdu]

You need to set zeppelin.run.mode as local in zeppelin-site.xml, otherwise zeppelin would launch flink interpreter in another pod which may cause this issue.

[amit-shahi]

@zjffdu Thanks. I'm using this k8 YAML https://github.com/apache/zeppelin/blob/master/k8s/zeppelin-server.yaml

Is there a different image for interpreter apache/zeppelin-interpreter:0.9.0-SNAPSHOT (as mentioned in above YAML) or can I use apache/zeppelin:0.9.0 from https://hub.docker.com/r/apache/zeppelin/tags as an image for ZEPPELIN_K8S_CONTAINER_IMAGE in zeppelin yaml?

[zjffdu]

@amit-shahi Let's discuss that in user mail list.

[amit-shahi]

@zjffdu Ok, I will create a thread on the mailing list.

[amit-shahi]

@zjffdu When I try to execute this YAML https://github.com/apache/zeppelin/blob/master/k8s/zeppelin-server.yaml thru kubectl apply -f zeppelin-server.yaml, I see this error-

Is the yaml file correct?

[amit-shahi]

@zjffdu How do I set zeppelin.run.mode as local in zeppelin-site.xml?