🐛 502 HTTP errors on large number of JS files

[vDMG]

Describe the bug We are using cloudflared tunnels to reach some internal services secured using Zero trust and Access applications. One of them is Kibana running with ECK. Randomly we fail to load kibana UI which is secured using a kuberntes deployed cloudflare tunnel on the same cluster then Kibana. Some JS files are impossible to load leading to 502.

If I use a port-forward eveyrthing is loading fine ! on the other side if I use a cloudflare tunnel directly deployed on EC2 everything is fine. So my theory is that a cloudflared deployed on Kubernetes can have some internal issues. But even using the tracing loglevel impossible to have a clear reason :

2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Request content length 0
2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Status: 200 OK served by ingress 8
2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Response Headers map[Cache-Control:[max-age=31536000] Connection:[keep-alive] Content-Encoding:[gzip] Content-Type:[application/javascript; charset=utf-8] Date:[Fri, 07 Oct 2022 12:05:06 GMT] Kbn-License-Sig:[0bbc5af556468fe9fe09e1957c8f92785ed89b0239b47bc58124941e84cf5a83] Kbn-Name:[kube-logging] Keep-Alive:[timeout=120] Referrer-Policy:[no-referrer-when-downgrade] Vary:[accept-encoding] X-Content-Type-Options:[nosniff]]
2022-10-07T12:05:06Z DBG CF-RAY: 756680c7b851d2f5-CDG Response content length unknown

But we got some 502 on web browser side.

To Reproduce Steps to reproduce the behavior:

1. Configure a ECK stack and try to reach the kibana UI
2. Deploy a cloudflared tunnel to on Kubernetes configured to access Kibana UI

If it's an issue with Cloudflare Tunnel: 4. Tunnel ID : 9e216a1f-1828-4e91-8bdd-0c3a50ddb772 5. cloudflared config:

- hostname: logs.internal.mydomain.co
  service: https://kube-logging-kb-http.kube-logging:5601
  originRequest:
    noTLSVerify: true

Expected behavior We should be able to load the Kibana UI without any issue

Environment and versions

• OS: Debian GNU/Linux 11
• Architecture: AMD
• Version: 2022.10.0

Logs and errors 502 errors when using cloudflared kubernetes pod tunnel.

Additional context CF-RAY: 756680c7b851d2f5-CDG

I've tried to set those setting on the clouflared kubernetes pod but nothing change.

            sysctl -w net.core.rmem_max=4000000
            sysctl -w net.core.somaxconn=4096