WebSocket connections return 403 when isolation mode is enabled

[hendrikgaffo]

Describe the bug We are running a WebSocket Service (using Socket.IO) on an AKS cluster with Bridge to Kubernetes installed. We are using the NGINX Ingress Controller for our services.

We are currently working on a different service, for which we use B2K to access data from other services in our cluster.

Now, whenever we are running the B2K debugger task with isolation mode enabled, any attempt to connect to the WebSocket service returns a 403: "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 403

If we turn isolation mode off, we get the proper 101 (Switching Protocols): "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 101

This has to do with the fact that in isolation mode, the request is being made to the Envoy pod instead of the original pod. I think the default Envoy configuration does now allow WebSocket connections.

In order to enable WebSocket connections, we thought about adding this part to the Envoy routing config of our WebSocket service, which gets automatically created by B2K:

upgrade_configs:
  upgrade_type: "websocket"

However, when updating the Envoy config for the service, it immediately gets set to the default value again (this is also mentioned in #159).

Mention the platform you are using VS Code: 1.83.0 Bridge to Kubernetes Extension: v2.0.120231004 AKS: 1.24.15

To Reproduce Steps to reproduce the behavior:

1. Deploy a Socket.IO server to AKS
2. Open up another service in VS Code and run the B2K debugger task with isolation mode enabled
3. Try to connect to the Socket.IO service

Expected behavior We expect the request to return a 101 - Switching Protocols, so that a WebSocket connection is established.

Screenshots This is a screenshot of Postman, showing an attempt to connect to the WebSocket Service with B2K isolation mode enabled.

Desktop (please complete the following information):

• OS: Ubuntu 22.04
• Browser: Edge for Business 117.0.2045.55 (Official build) (64-bit)

[hsubramanianaks]

@hendrikgaffo Thanks for raising this issue. we will check this and get back to you.

[hsubramanianaks]

@hendrikgaffo I looked into it, I have a question this upgrade_configs needs to be added only for websocket services right ? Not for all services, also you are looking for communication between ws --> ws not ws --> wss ? because ws --> wss need transport socket added with certificates in them. Please confirm on this. If you are willing to contribute here is the place where you can add this feature. Thanks for your patience.

Reference : https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/websocket#step-3-test-proxying-ws-ws

[hsubramanianaks]

@hendrikgaffo can you confirm on the above one please ? thank you.

[letmagnau]

some news about it??

[letmagnau]

Hi , can I ask if this problem will be fixed ?

It's very time locking debug only in non isolated way... If we are activating the isolated mode, wss can't work anymore as described

reagards

[hendrikgaffo]

TLDR; setup Soketi or get Azure Web PubSub for SocketIO

---

Hey @letmagnau and @hsubramanianaks. I am sorry for the late response, since I usually don't participate much in OSS conversations and therefore never check my notifications.

I have looked at the resources provided by @hsubramanianaks (thanks a ton for the support, I appreciate it!) and remember concluding back then that I was nowhere near proficient enough in C#, nor willing to learn the repos implementation details + a new language to fix this.

Since I didn't have another option and our entire SaaS relied on SocketIO, we decided to roll our own solution. Thankfully, right when we started refactoring, there was a banner on the SocketIO docs, that Azure is offering a native Integration (Link to Blogpost).

It's basically SocketIO on Azure Web PubSub (they also got serverless and all the kool-aid you might want).

I really dislike the pricing strategy/terminology (you have X messages for $y per Unit per Day on Standard, after that you pay $z per 1 Mio messages. What is a unit anyways? And why two tiers inside of one tier?)... BUT it's a phenomenal product. Never had any issues ever since, B2K worked like a charm, production is healthier than ever, every message gets delivered (we send double digit million SocketIO messages), and pay pretty much nothing for it (50€ maybe? Def. way less than setting the infra up ourselves).

Nowadays there's Soketi, too, which is crazy easy to use and scales nicely (talking regular SaaS scale, not Microsoft level of gazillion connections). Highly recommended if you're building something new with Websockets and are on a budget. Otherwise just go for the managed Azure version. Or fix the bug at the resources @hsubramanianaks pointed us to 😁

In case there are no questions left that I can help others with, this issue can be closed from my side, since it's resolved for us.