pip install can not stop and ImportError: cannot import name 'ServeRequest' from 'ray.serve.utils' in mnist e2e

[631068264]

/kind bug

What steps did you take and what happened: just use pip install kubeflow-fairing

keep installing for a long time and it try to install same package with different version.

finally I try this pip install kubeflow-fairing --use-deprecated=legacy-resolver

Then I run mnist e2e example py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import uuid

import yaml
from kubeflow import fairing
from kubeflow.fairing.kubernetes.utils import mounting_pvc
from kubernetes import client as k8s_client
from kubernetes import config as k8s_config

DOCKER_REGISTRY = '10.19.64.203:8080'
my_namespace = 'kserve-test'

num_chief = 1  # number of Chief in TFJob
num_ps = 1  # number of PS in TFJob
num_workers = 2  # number of Worker in TFJob
model_dir = "/mnt"
export_path = "/mnt/export"
train_steps = "1000"
batch_size = "100"
learning_rate = "0.01"

pvc_name = 'mnist-pvc'
pvc_yaml = f'''
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {pvc_name}
  namespace: {my_namespace}
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 10Gi
'''

k8s_config.load_kube_config()

k8s_core_api = k8s_client.CoreV1Api()
# k8s_core_api.create_persistent_volume(yaml.safe_load(pv_yaml))
k8s_core_api.create_namespaced_persistent_volume_claim(my_namespace, yaml.safe_load(pvc_yaml))

tfjob_name = f'mnist-training-{uuid.uuid4().hex[:4]}'

output_map = {
    "Dockerfile": "Dockerfile",
    "mnist.py": "mnist.py"
}

command = ["python",
           "/opt/mnist.py",
           "--tf-model-dir=" + model_dir,
           "--tf-export-dir=" + export_path,
           "--tf-train-steps=" + train_steps,
           "--tf-batch-size=" + batch_size,
           "--tf-learning-rate=" + learning_rate]

fairing.config.set_preprocessor('python', command=command, path_prefix="/app", output_map=output_map)
fairing.config.set_builder(name='docker', registry=DOCKER_REGISTRY,
                           image_name="mnist", dockerfile_path="Dockerfile")

fairing.config.set_deployer(name='tfjob', namespace=my_namespace, stream_log=False, job_name=tfjob_name,
                            chief_count=num_chief, worker_count=num_workers, ps_count=num_ps,
                            pod_spec_mutators=[mounting_pvc(pvc_name=pvc_name, pvc_mount_path=model_dir)])
fairing.config.run()

What did you expect to happen:

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Fairing version: (use python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"):

(.env) ➜  kubeflow git:(master) ✗ python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/__init__.py", line 2, in <module>
    from kubeflow.fairing.ml_tasks.tasks import TrainJob, PredictionEndpoint
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/ml_tasks/tasks.py", line 4, in <module>
    from kubeflow.fairing.backends import KubernetesBackend
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/backends/__init__.py", line 1, in <module>
    from kubeflow.fairing.backends.backends import *
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/backends/backends.py", line 8, in <module>
    from kubeflow.fairing.builders.cluster import gcs_context
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/builders/cluster/gcs_context.py", line 6, in <module>
    from kubeflow.fairing.kubernetes.manager import client, KubeManager
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kubeflow/fairing/kubernetes/manager.py", line 6, in <module>
    from kfserving import KFServingClient
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kfserving/__init__.py", line 16, in <module>
    from kfserving.kfmodel import KFModel
  File "/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/kfserving/kfmodel.py", line 24, in <module>
    from ray.serve.utils import ServeRequest
ImportError: cannot import name 'ServeRequest' from 'ray.serve.utils' (/Users/wyx/union_workspce/kubeflow/.env/lib/python3.9/site-packages/ray/serve/utils.py)

(.env) ➜  kubeflow git:(master) ✗ pip list |grep kube                                                     
kubeflow-fairing               1.0.2
kubeflow-pytorchjob            0.1.3
kubeflow-tfjob                 0.1.3
kubernetes                     10.0.1

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):

dev_local

• Kubernetes version: (use kubectl version): k3s Kubernetes 1.19
• OS (e.g. from /etc/os-release): core run on osx
  k3s on centos7

NOTE: If you are using fair from master, please provide us the git commit hash.