Strange issue when using WebSockets

[willemda]

Hello,

First of all, we've been using Halibut for a long time and we love it! (See our PR regarding routing)

We've recently switched to using WebSockets as it enable us sharing the port with other components running on the same machine easily.

From that moment, we've experienced some strange issues where our Polling Client would suddenly lose connectivity (sometimes having 100% CPU).

Here's what our setup looks like:

The Tentacle is a windows service running under net 4.8 (running latest standard Halibut, no routing) The Octopus is a windows service running under netcoreapp3.1 (running our Halibut routing)

Logs and dump/trace analysis on the Tentacle suggests that 3 of the PollingThread seems stuck forever (no mention of reconnection or activity in the logs).

We're wondering if it's something you've also experienced since you're using Halibut on such a large infrastructure?

We've also noticed the following issue that looks very similar to what we're experiencing but are unsure if this is applicable to our setup: https://github.com/dotnet/runtime/issues/16810

We do have dumps, logs & dotTrace exports to help if needed, we've tried a lot of things but couldn't figure out where this come from with certitude.

[janpieterz]

Regarding the above, we're running websockets for 14 days without issue now. We will switch our whole production fleet end of the month and, if that stays stable we can close. No clear insights into the cause of this issue, back then a couple of other small changes were made but none that have any reasonable connection to the websocket connections.

[jburger]

Thanks for letting us know @janpieterz keep us posted :crossed_fingers:

[willemda]

Looks you guys found the issue a while ago already so I'll close. I've been following the past month's commits and there are really cool updates being done, love it 👍