cloudflared
cloudflared copied to clipboard
cloudflare
🐛 Cloudflared image cannot run with crio at kubernetes 1.24.10
Describe the bug Cloudflared at kubernetes did not open port for liveness probe when using crio.
To Reproduce Steps to reproduce the behavior:
- Configure kubernetes v1.24.10 with crio 1.24.4 with canal
- Create namespace wordpress
kubectl create namespace wordpress - Create directory config, and copy cert.pem and tunnel.json to that directory
- Run
kubectl create secret -n wordpress generic tunnel-credentials --from-file=./config/tunnel.json --from-file=./config/cert.pemto create secret - Create cloudflared.yaml with this config:
apiVersion: apps/v1
kind: Deployment
metadata:
name: cloudflared
namespace: wordpress
spec:
selector:
matchLabels:
app: cloudflared
replicas: 2 # You could also consider elastic scaling for this deployment
template:
metadata:
labels:
app: cloudflared
spec:
containers:
- name: cloudflared
image: cloudflare/cloudflared:latest
args:
- tunnel
# Points cloudflared to the config file, which configures what
# cloudflared will actually do. This file is created by a ConfigMap
# below.
- --config
- /etc/cloudflared/config/config.yaml
- run
livenessProbe:
httpGet:
# Cloudflared has a /ready endpoint which returns 200 if and only if
# it has an active connection to the edge.
path: /ready
port: 2000
failureThreshold: 1
initialDelaySeconds: 10
periodSeconds: 10
volumeMounts:
- name: config
mountPath: /etc/cloudflared/config
readOnly: true
# Each tunnel has an associated "credentials file" which authorizes machines
# to run the tunnel. cloudflared will read this file from its local filesystem,
# and it'll be stored in a k8s secret.
- name: cloudflared-auth-volume
mountPath: /etc/cloudflared/tunnel.json
subPath: tunnel.json
readOnly: true
- name: cloudflared-auth-volume
mountPath: /etc/cloudflared/cert.pem
subPath: cert.pem
readOnly: true
volumes:
- name: cloudflared-auth-volume
secret:
# By default, the credentials file will be created under ~/.cloudflared/<tunnel ID>.json
# when you run `cloudflared tunnel create`. You can move it into a secret by using:
# ```sh
# kubectl create secret generic tunnel-credentials \
# --from-file=credentials.json=/Users/yourusername/.cloudflared/<tunnel ID>.json
# ```
secretName: tunnel-credentials
# Create a config.yaml file from the ConfigMap below.
- name: config
configMap:
name: cloudflared
items:
- key: config.yaml
path: config.yaml
---
# This ConfigMap is just a way to define the cloudflared config.yaml file in k8s.
# It's useful to define it in k8s, rather than as a stand-alone .yaml file, because
# this lets you use various k8s templating solutions (e.g. Helm charts) to
# parameterize your config, instead of just using string literals.
apiVersion: v1
kind: ConfigMap
metadata:
name: cloudflared
namespace: wordpress
data:
config.yaml: |
# Name of the tunnel you want to run
tunnel: example-tunnel
credentials-file: /etc/cloudflared/tunnel.json
# Serves the metrics server under /metrics and the readiness server under /ready
metrics: 0.0.0.0:2000
# Autoupdates applied in a k8s pod will be lost when the pod is removed or restarted, so
# autoupdate doesn't make sense in Kubernetes. However, outside of Kubernetes, we strongly
# recommend using autoupdate.
no-autoupdate: true
# The `ingress` block tells cloudflared which local service to route incoming
# requests to. For more about ingress rules, see
# https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress
#
# Remember, these rules route traffic from cloudflared to a local service. To route traffic
# from the internet to cloudflared, run `cloudflared tunnel route dns <tunnel> <hostname>`.
# E.g. `cloudflared tunnel route dns example-tunnel tunnel.example.com`.
ingress:
# The first rule proxies traffic to the httpbin sample Service defined in app.yaml
- hostname: wp.nizarakbar.com
service: http://my-release-wordpress
# This rule sends traffic to the built-in hello-world HTTP server. This can help debug connectivity
# issues. If hello.example.com resolves and tunnel.example.com does not, then the problem is
# in the connection from cloudflared to your local service, not from the internet to cloudflared.
#- hostname: hello.example.com
# service: hello_world
# This rule matches any traffic which didn't match a previous rule, and responds with HTTP 404.
- service: http_status:404
```
6. Run `kubectl apply -f cloudflared.yaml`
7. See error
If it's an issue with Cloudflare Tunnel:
9. Tunnel ID : 19f950f6-969d-4f4f-8684-e116c03e2ddd
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment and versions**
- OS: [e.g. MacOS]
- Architecture: [e.g. AMD, ARM]
- Version: [e.g. 2022.02.0]
**Logs and errors**
Normally it will open port 2000 and success in liveness probe but when using crio it is failed to open port and doing liveness probe so the pod always restarting.
**Additional context**
error message at `kubectl describe pod cloudflared-5f987cc995-gw7ww -n wordpress`:
```Name: cloudflared-5f987cc995-gw7ww
Namespace: wordpress
Priority: 0
Node: workernonrke1/192.168.3.124
Start Time: Wed, 08 Feb 2023 06:02:18 +0000
Labels: app=cloudflared
pod-template-hash=5f987cc995
Annotations: cni.projectcalico.org/containerID: 2340cb0acafc5bdb1f98fd8ade32593ff9bf9da5bea35d2efcb68d42c07f7aab
cni.projectcalico.org/podIP: 10.42.3.9/32
cni.projectcalico.org/podIPs: 10.42.3.9/32
Status: Running
IP: 10.42.3.9
IPs:
IP: 10.42.3.9
Controlled By: ReplicaSet/cloudflared-5f987cc995
Containers:
cloudflared:
Container ID: cri-o://32a44e312e16a5f94d030af4f3536764e7dee310a14a14ac22fc1f3bc0404305
Image: cloudflare/cloudflared:latest
Image ID: docker.io/cloudflare/cloudflared@sha256:5235e62dc2f4afe707355a3ab30babe0f15523bba12bce496e7259ad4eb580d1
Port: 2000/TCP
Host Port: 0/TCP
Args:
tunnel
--config
/etc/cloudflared/config/config.yaml
run
State: Running
Started: Wed, 08 Feb 2023 06:02:22 +0000
Ready: True
Restart Count: 0
Liveness: http-get http://:2000/ready delay=10s timeout=1s period=10s #success=1 #failure=1
Environment: <none>
Mounts:
/etc/cloudflared/cert.pem from cloudflared-auth-volume (ro,path="cert.pem")
/etc/cloudflared/config from config (ro)
/etc/cloudflared/tunnel.json from cloudflared-auth-volume (ro,path="tunnel.json")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z9trz (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
cloudflared-auth-volume:
Type: Secret (a volume populated by a Secret)
SecretName: tunnel-credentials
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: cloudflared
Optional: false
kube-api-access-z9trz:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 41s default-scheduler Successfully assigned wordpress/cloudflared-5f987cc995-gw7ww to workernonrke1
Normal Pulling 40s kubelet Pulling image "cloudflare/cloudflared:latest"
Normal Pulled 37s kubelet Successfully pulled image "cloudflare/cloudflared:latest" in 3.411624889s
Normal Created 37s kubelet Created container cloudflared
Normal Started 37s kubelet Started container cloudflared
Warning Unhealthy 20s kubelet Liveness probe failed: Get "http://10.42.3.9:2000/ready": dial tcp 10.42.3.9:2000: connect: connection refused
Normal Killing 20s kubelet Container cloudflared failed liveness probe, will be restarted
```
And when i try to use containerd as container runtimes, the pod created normally without a problem at liveness probe. So as i think cloudflared image cannot work at CRI-O.
And for temporary solution, i change CRI-O with containerd.
To enable the /ready endpoint you'll want to enable the metrics endpoint in cloudflared --metrics 0.0.0.0:2000.
