ValveSoftware
Crash seemingly related to Fake IP feature
We've received reports from several hosts of Unturned (AppID 304930) that their servers intermittently crash if Fake IP is enabled. Here's a crash dump from a Windows server which reports the crash happening in steamclient64.dll:
I asked a couple of hosts running into the crash to enable verbose output from ISteamNetworkingUtils::SetDebugOutputFunction. The log files are hundreds of thousands of lines, so I've cut out the last hundred or so before the crashes here:
SteamNetworkingSockets Log 1.txt SteamNetworkingSockets Log 2.txt
The frequency of the "we didn't know about a session on that relay for this connection" messages seems to increase gradually while the server is running.
Any help you can provide would be greatly appreciated! I'm not sure whether this is a bug on my end or not.
I have also realized the crashes happen more often when there are alot of players on the server.
We need a fix for this, would help the playerbase a lot to have easier to make servers
Commenting to update everyone following this thread (members of the Unturned plugin dev community AFAIK 😅):
On January 7th we received a comment on the help site ticket that the issue had been escalated internally.
However, it looks like the ticket has been closed out. I'm not sure whether this means it is an issue on my end, or whether it automatically closed because I didn't comment on the "escalated internally" comment. I've submitted a follow-up help site ticket hoping for clarification.
I would typically assume it's an issue on my end, but I can't think of what the cause would be: As far as I'm aware our Steam Networking Sockets implementation has been fairly solid except when Fake IP is enabled. (If that's not the case - i.e., if anyone is running into Unturned server crashes without Fake IP enabled - please let me know!) The Fake IP API on the server side is very straightforward, as simple as:
- We call
BeginAsyncRequestFakeIPimmediately beforeLogOnorLogOnAnonymous. Per the documentation, we don't wait for the result before logging on because the request is queued until login completes. - When creating server sockets (e.g.,
CreateListenSocketIP), we also callCreateListenSocketP2PFakeIP.
It's possible I'm missing something, but I didn't find any gotchas in the Fake IP documentation: https://partner.steamgames.com/doc/api/ISteamNetworkingSockets#1
If it is a SNS/GNS bug, I imagine it's a difficult prioritization issue for the devs at Valve. As far as I'm aware Fake IP isn't widely used (yet?), Unturned is a relatively small/niche game, and they're working on the networking for many big/critical projects like CS2 and Dota 2.
Another quick follow up: The help site ticket was indeed closed automatically and has now been re-opened.
Any alternative to safelly host at the moment?
If your server doesn't need to be publicly visible on the server list, the easiest option is to join by server code which also uses SDR.
For publicly visible servers, a few hosts are using proxies for the A2S queries to mitigate denial of service attacks. Anycast proxies are the most effective albeit rather contentious. We have some changes to anycast proxies pending for the next update, and will announce them soon - we just have an unusually high number of WIP announcements to finish sorting out. :)
Replying to https://github.com/ValveSoftware/GameNetworkingSockets/issues/350#issuecomment-2517244750
This might be related to container_pid_limit in your Pterodactyl config (/etc/pterodactyl/config.yml), it's set to 512 by default and perhaps this is related to why it would have trouble creating threads?
Alright thanks, I have changed the docker pid to 1024 and i will keep yall updated to see if it works!
The latest we've heard is there are some peer-to-peer and fake IP related fixes coming in the next update, so my fingers are crossed! 🤞 When any fixes are released it should show up in the Steamworks SDK Dedicated Server Redist history here: https://steamdb.info/depot/1006/history/ I'll comment when/if I hear it's released.
A server host reported to us that using the newest versions of the steamclient files (steamclient.dll or steamclient.so, etc) they haven't run into any crashes over the past week. If that's the case, as a temporary workaround it may be worth manually updating these files.
Elaborating:
As I understand it, dedicated servers get their copy of the Steam dynamic libraries through the dedicated server redist depots—for example, depot ID 1006. These are updated less frequently than the dynamic libraries installed with the Steam client. (Currently, the dedicated server files were last updated ~13 months ago.)
Although not endorsed nor officially supported by Valve, it may work to replace those files with the ones included in a regular Steam install.
On Linux, those are:
-
steamclient.so -
linux64/steamclient.so
On Windows, those are:
-
steamclient.dll -
steamclient64.dll -
tier0_s.dll -
tier0_s64.dll -
vstdlib_s.dll -
vstdlib_s64.dll
Replying to https://github.com/ValveSoftware/GameNetworkingSockets/issues/350#issuecomment-2937367869
So I'm guessing the way to get the Linux ones is to download the steam client on e.g Debian and copy it's .so file?
@RBMKBlaze I could be wrong, but I think the copies included in a steamcmd install are the same as in a full Steam client install. So, if you have steamcmd installed already, you could try with the copies from there.
@SDGNelson ~~Do you mean this? I might be in the wrong place 😥. Anyways, I tried that .so file on a server that is unused and hasnt been turned on in a while and it failed to update, am I just copying the wrong file?~~
Turns out I copied it to the wrong place, pointed out by Ghosticollis, I put it in \steamclient.so instead of \linux64\steamclient.so
@SDGNelson Would it be possible for you to just send/attach the file? I don't see why we need to do a hunt for it. (Not trying to be rude but this would save both of us a lot of time) Thanks!
Good news: the dedicated server redistributables have been updated! Is this resolved for everyone?
Seems to be fixed, I still get crashes (but rarely, really rare) and no issues with mass joining. Looks like it is and maybe will get improved (I am running redist from like april-may when you told us about it tho)
by now 2 months has passed since we started using the updated steamclient file. I haven't noticed any crash since then. thank you.
p.s. the issue that Blaze refers to as "mass joining" still exist. which happens at wipe moment when many players try join at once, only few players manage to join, the rest get blocked by Steam, and no one can join the server for 20-30 minutes. this second issue is not related to the topic of this crash issue. it seems to be kind of steam protection. hopefully they config it better because currently it ruins wipe events. if I am not mistaken this happens only when fake ip feature is enabled.
@KtoemYsw re issue after a wipe: we received a report from a server owner describing how they wiped their server by deleting all the savedata files, and that they realized the deletion was still running while players were trying to join which caused some issues. In case it's a similar issue, if you rename/move the savedata folder before deletion does it fix the problem?
@SDGNelson I doubt that's the issue, I had the same problem while I killed the server, deleted the savedata and then started it.
@SDGNelson sorry for late response, I haven't noticed the notification mail until now. at wipe we shutdown the server first, remove data files, then start server again. the issue seems to be a Steam issue, not game issue. it feels like Steam protection get activated when many join requests happen at once. new joins get fully blocked for around 20 minutes, then it start allowing more player to join gradually. and ofc this ruins the wipe moment :( some networks like Pandahut use tricks to allow everyone to join, by telling those who not able to join that server to join any other Pandahut server and do /tunnel [server id] , which apparently will relay player to the target server using a "tunnel" ip, not the steam fake ip.
Tunnel IP means that it doesn't use SDR to connect rather IP sockets (afaik). You probably have outdated libraries although they have been updated. Infected, Unbeaten and SOD haven't had these issues for a few months now.
Blaze it is me Ghosticollis, sorry for using random name account. yes we don't have the crash issue anymore after updating the Steam library. but I realized that we still have the mass join issue at wipe time.