projectcapsule
add support for Websockets
Describe the feature
Add support for reverse proxying websockets.
Expected behavior
When a client tries to open a websocket for something like exec or log, it should work.
The Capsule Proxy already supports websocket connections:
$: KUBECONFIG=capsule-proxy.kubeconfig kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
qwe12345-gpu 363b1077-5d99-45bb-abb1-eb3d17d09bf2 1/1 Running 0 42s
$: KUBECONFIG=capsule-proxy.kubeconfig kubectl -n qwe12345-gpu logs -f 363b1077-5d99-45bb-abb1-eb3d17d09bf2/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/08/07 06:38:07 [notice] 1#1: using the "epoll" event method
2024/08/07 06:38:07 [notice] 1#1: nginx/1.27.0
2024/08/07 06:38:07 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/08/07 06:38:07 [notice] 1#1: OS: Linux 5.15.0-113-generic
2024/08/07 06:38:07 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/08/07 06:38:07 [notice] 1#1: start worker processes
2024/08/07 06:38:07 [notice] 1#1: start worker process 29
2024/08/07 06:38:07 [notice] 1#1: start worker process 30
2024/08/07 06:38:07 [notice] 1#1: start worker process 31
2024/08/07 06:38:07 [notice] 1#1: start worker process 32
2024/08/07 06:38:07 [notice] 1#1: start worker process 33
2024/08/07 06:38:07 [notice] 1#1: start worker process 34
2024/08/07 06:38:07 [notice] 1#1: start worker process 35
2024/08/07 06:38:07 [notice] 1#1: start worker process 36
2024/08/07 06:38:07 [notice] 1#1: start worker process 37
2024/08/07 06:38:07 [notice] 1#1: start worker process 38
2024/08/07 06:38:07 [notice] 1#1: start worker process 39
2024/08/07 06:38:07 [notice] 1#1: start worker process 40
2024/08/07 06:38:07 [notice] 1#1: start worker process 41
2024/08/07 06:38:07 [notice] 1#1: start worker process 42
2024/08/07 06:38:07 [notice] 1#1: start worker process 43
2024/08/07 06:38:07 [notice] 1#1: start worker process 44
If you're facing issues I would ask you to share more details, as well as logs. Issues should be open when a bug report has been assessed correctly and it's reproducible. The best way is to interact with the community on the #capsule Kubernetes Slack workspace channel.
The issue is going to be closed, happy to get it opened back if we have actually a bug.
Kubectl does not use websockets. It uses spdy. There is ongoing work to migrate to websockets started in 1.30. I can share more details about my issue later today.
You're right, just reopened that.
We'll evaluate the support for WS once landed in GA, however, happy to receive contributions.
WebSocket support has been in the api for a long time. Only recently are they phasing out spdy.
I'm planning on trying to fix this myself, but all I'm going to try to do is reverse proxy the ws request. Once websocket becomes default in kubectl, what I do wont work when using watch. It looks like #57 would resolve this issue however.
#57 will be closed since the implementation efforts are unbearable.
Once WS are landing in stable mode, we'll find a viable solution, as well as planning a migration path.
After some more testing, it looks like this already works with websockets. I haven't had an issue with this and v1.30 and v1.31 which default to websockets.
My issue is due to how authorization is handled in the browser WebSocket API. My application uses WebSockets to get pod logs, exec, attach, etc... Due to API limitation you have to specify the auth bearer token as a protocol during WebSocket creation.
The token is placed under "sec-websocket-protocol:" in the headers. Like so:
Sec-Websocket-Protocol: base64url.bearer.authorization.k8s.io.<bearer as base64 unpadded>
I'm opening a PR to add support for tokens specified like this.