refactor yurtadm init/join and support to create high-availability OpenYurt cluster

[windydayc]

What type of PR is this?

/kind feature /kind enhancement

What this PR does / why we need it:

At present, the installation of OpenYurt cluster is still a little complicated, there is a lack of a unified installation way that can simply and automatically install OpenYurt cluster.

In addition, currently the yurtadm command cannot cope with scenarios that requiring high availability. Therefore, it is necessary to provide a way to create high availability OpenYurt cluster.

Related issue: #856 #902

Special notes for your reviewer:

/assign @rambohe-ch @Peeknut

[openyurt-bot]

@windydayc: GitHub didn't allow me to assign the following users: Peeknut.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to this:

What type of PR is this?

/kind feature /kind enhancement

What this PR does / why we need it:

At present, the installation of OpenYurt cluster is still a little complicated, there is a lack of a unified installation way that can simply and automatically install OpenYurt cluster.

In addition, currently the yurtadm command cannot cope with scenarios that requiring high availability. Therefore, it is necessary to provide a way to create high availability OpenYurt cluster.

Related issue: #856 #902

Special notes for your reviewer:

/assign @rambohe-ch @Peeknut

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openyurt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: windydayc To complete the pull request process, please assign rambohe-ch You can assign the PR to them by writing /assign @rambohe-ch in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[codecov[bot]]

Codecov Report

Merging #926 (e93fbf7) into master (482fc1d) will increase coverage by 8.36%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #926      +/-   ##
==========================================
+ Coverage   35.30%   43.66%   +8.36%     
==========================================
  Files          76       83       +7     
  Lines        9913    11260    +1347     
==========================================
+ Hits         3500     4917    +1417     
+ Misses       6146     5908     -238     
- Partials      267      435     +168     

Flag	Coverage Δ
unittests	43.66% <ø> (+8.36%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/yurthub/server/nonresource.go	31.08% <0.00%> (ø)
pkg/controller/kubernetes/util/node/node.go	82.35% <0.00%> (ø)
pkg/yurthub/server/certificate.go	0.00% <0.00%> (ø)
pkg/controller/nodelifecycle/metrics.go	100.00% <0.00%> (ø)
pkg/controller/kubernetes/util/taints/taints.go	82.08% <0.00%> (ø)
pkg/yurthub/server/server.go	0.00% <0.00%> (ø)
...troller/nodelifecycle/node_lifecycle_controller.go	56.15% <0.00%> (ø)
pkg/yurthub/cachemanager/cache_manager.go	68.22% <0.00%> (+0.07%)	:arrow_up:
pkg/yurthub/util/util.go	21.37% <0.00%> (+0.86%)	:arrow_up:
pkg/yurtadm/util/edgenode/util.go	9.02% <0.00%> (+7.35%)	:arrow_up:
... and 10 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[rambohe-ch]

@windydayc please do not push --force when you fixed above comments.

[rambohe-ch]

@Peeknut PTAL

[windydayc]

For config/yurtadm/cluster-image/openyurt-latest/cni directory, I have replaced the content in openyurt-cni-0.8.7-0.x86_64.rpm in https://github.com/openyurtio/openyurt/releases

Reason see: https://openyurt.io/docs/user-manuals/network/edge-pod-network/#ipam-pod-ip-address-kept

[windydayc]

For yurtadm init, since the image of the latest version often changes, its configuration file may also change, so the cluster image which constructed by the latest version may not be available. Therefore, an image with a fixed version number is used here to prevent cluster imgae unavailability caused by image:latest changes. For example, here use yurt-controller-manager:v0.7.0 ，not yurt-controller-manager:latest

[windydayc]

@Peeknut I have solved the above problems. Please have a look.

[huiwq1990]

@windydayc I think it's not a good idea to maintian the cni binaries in openyurt repo, could you explain the reason? I recommend:

1. the binaries directly download from https://github.com/containernetworking/plugins/releases/ or
2. after the k8s starup, use a daemonset put the binaries to cni directory, the yaml like:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: test-cni-tool
  name: test-cni-tool
spec:
  selector:
    matchLabels:
      app: test-cni-tool
  template:
    metadata:
      labels:
        app: test-cni-tool
    spec:
      hostNetwork: true
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
      - operator: "Exists"
      containers:
      - name: myapp-container
        image: busybox:1.28
        command: ['sh', '-c', 'echo The app is running! && sleep 3600h']
      initContainers:
        - name: hostname
          image: huiwq1990/cni-plugins
          imagePullPolicy: Always
          command: ["/bin/sh"]
          args: ["-c", "cp /cni/* /opt/cni/bin/"]
          volumeMounts:
           - name: task-pv-storage
             mountPath: /opt/cni/bin
        - name: restart
          image: jpetazzo/nsenter
          command: ["/bin/sh"]
          args: ["-c", "/nsenter --mount=/proc/1/ns/mnt systemctl restart containerd"]
          securityContext:
            privileged: true
      volumes:
        - name: task-pv-storage
          hostPath:
            path: /opt/cni/bin
            type: Directory

[windydayc]

@windydayc I think it's not a good idea to maintian the cni binaries in openyurt repo, could you explain the reason? I recommend:

1. the binaries directly download from https://github.com/containernetworking/plugins/releases/ or
2. after the k8s starup, use a daemonset put the binaries to cni directory, the yaml like:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: test-cni-tool
  name: test-cni-tool
spec:
  selector:
    matchLabels:
      app: test-cni-tool
  template:
    metadata:
      labels:
        app: test-cni-tool
    spec:
      hostNetwork: true
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
      - operator: "Exists"
      containers:
      - name: myapp-container
        image: busybox:1.28
        command: ['sh', '-c', 'echo The app is running! && sleep 3600h']
      initContainers:
        - name: hostname
          image: huiwq1990/cni-plugins
          imagePullPolicy: Always
          command: ["/bin/sh"]
          args: ["-c", "cp /cni/* /opt/cni/bin/"]
          volumeMounts:
           - name: task-pv-storage
             mountPath: /opt/cni/bin
        - name: restart
          image: jpetazzo/nsenter
          command: ["/bin/sh"]
          args: ["-c", "/nsenter --mount=/proc/1/ns/mnt systemctl restart containerd"]
          securityContext:
            privileged: true
      volumes:
        - name: task-pv-storage
          hostPath:
            path: /opt/cni/bin
            type: Directory

@huiwq1990

1. cni here is from openyurt-cni-0.8.7-0.x86_64.rpm in https://github.com/openyurtio/openyurt/releases, not directly download from https://github.com/containernetworking/plugins/releases/. Reason see: https://openyurt.io/docs/user-manuals/network/edge-pod-network/#ipam-pod-ip-address-kept
2. The config/yurtadm/cluster-image/openyurt-v0.7.0 dir here is just an example for user reference, users can customize their own cluster image.