[Bug] Unity App Crash on launch with Firebase 12.10.0

[mmiro1]

Description

After upgrading the Firebase SDK from 12.1.0 to 12.10.0, on some Windows PCs the app crashes on launch.

I have tried to use the FirebaseApp.CheckAndFixDependenciesAsync().ContinueWithOnMainThread instead of FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith, and also, I have tried to wait some frames before calling that method.

For now I have downgraded to 12.9.0 and the issue seems to be resolved.

I have seen this issue too and the last message seems to be the same problem, I wanted to open another one since that one is pretty old. https://github.com/firebase/firebase-unity-sdk/issues/728

Thank you in advance.

Reproducing the issue

I have two similar computer with the same Windows version, in one the app crashes with the SDK 12.10.0 and the other works fine. The build with the SDK 12.1.0 & 12.9.0 works on both computers.

I have attached the Player.log, the crash.dmp and the PC specs: Player.log crash.dmp PC-Specs.LOG

Firebase Unity SDK Version

12.10.0

Unity editor version

2022.3.49f1

Installation Method

Unity Package Manager

Problematic Firebase Component(s)

App Check

Other Firebase Component(s) in use

Analytics, Firestore, Crashlytics, Authentication, Messaging

Additional SDKs you are using

None.

Targeted Platform(s)

Desktop

Unity editor platform

Windows

Scripting Runtime

IL2CPP

Release Distribution Type

Pre-built SDK from https://firebase.google.com/download/unity

Relevant Log Output

From the Player.log:

========== OUTPUTTING STACK TRACE ==================

0x00007FF8D2843020 (MSVCP140) Thrd_yield
0x00007FF806F67CCA (FirebaseCppApp-12_10_0) uWS::HttpSocket<0>::upgrade
0x00007FF806F5BE0A (FirebaseCppApp-12_10_0) uWS::HttpSocket<0>::upgrade
0x00007FF806F5AE27 (FirebaseCppApp-12_10_0) uWS::HttpSocket<0>::upgrade
0x00007FF8EE2F37B0 (ucrtbase) wcsrchr
0x00007FF8EEA9E8D7 (KERNEL32) BaseThreadInitThunk
0x00007FF8F06DC34C (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========

If using CocoaPods for Apple platforms, the project's Podfile.lock

Issue in Windows.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[argzdev]

Hey @mmiro1, thanks for reaching out. While we investigate this further, could you help answer a few questions below:

1. Is the issue consistently reproducible?
2. Just to confirm, you mentioned that the problematic SDK is App Check. Is the issue only specific to App Check SDK, so the issue is not reproducible when App Check is excluded?
3. Are you able to able to reproduce the behavior on an MCVE?
4. Could you check if the latest Visual C++ Redistributable for Visual Studio 2015-2022 is installed on both windows machine? Or perhaps are using the same version?

In my investigation, it doesn't seem like we had any changes from version 12.9.0 to 12.10.0 that's related to this issue, aside from using latest windows version in workflows.

[mmiro1]

Hi @argzdev , thank you for the fast reply.

1. Yes, the crash happens 100% of the times on the affected PC, and I have reports of users with the same issue and stacktrace.
2. Sorry I may have missclicked. I wanted to mark the App Core SDK. I am not sure which of the installed SDKs is the responsible for the issue, the only information I have is the stacktrace from the Firebase App Core.
3. I haven't tried this yet, as mentioned I have downgraded the SDKs version to 12.9.0 (were I don't have issues) and I am currently trying to deploy a build for affected users.
4. On the affected PC the versions are: Microsoft Visual C++ 2015-2022 Redistributable (x64) - 14.29.30133 & Microsoft Visual C++ 2015-2022 Redistributable (x86) - 14.30.30704. And in the PC without issues: Microsoft Visual C++ 2015-2022 Redistributable (x64) - 14.42.34438 & Microsoft Visual C++ 2015-2022 Redistributable (x86) - 14.42.34438.

I did look the patch notes to but nothing related to Desktop did appear.

[argzdev]

Thanks for the additional details, @mmiro1. We'll keep investigating this, and update back here once we have feedback to share.

[CurtainNie]

I encountered the same problem, below is the stack trace when I crashed

========== OUTPUTTING STACK TRACE ==================

0x00007FFD0EB93028 (MSVCP140) Thrd_yield 0x00007FFCA54B8AF9 (FirebaseCppApp-12_10_1) uWS::WebSocket<0>::send 0x00007FFCA54B2BFE (FirebaseCppApp-12_10_1) uWS::WebSocket<0>::send 0x00007FFCA54B21BE (FirebaseCppApp-12_10_1) uWS::WebSocket<0>::send 0x00007FFD1ED79333 (ucrtbase) recalloc 0x00007FFD1FDD259D (KERNEL32) BaseThreadInitThunk 0x00007FFD2126AF78 (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========

Firebase Unity SDK Version 12.10.1

Unity editor version 2022.3.57f1

I now have to stop running Firebase related code in Unity Editor

[mmiro1]

Have you tried to downgrade the version to 12.9.0?

Firebase Unity SDK Version 12.10.1

[NightCoded]

I'm experiencing the same problem. I've tried several versions of the Firebase SDK, including the latest one. I've now downgraded to 11.6.0.

Initially, it ran smoothly without crashing, but then it would crash occasionally. All SDK versions I've tried always had the same problem.

========== OUTPUTTING STACK TRACE ==================

0x00007FFD47C9178D (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47C92C9B (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47C8EB21 (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47C8F6F0 (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47C90359 (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47C90649 (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47C8CE83 (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47C7F931 (FirebaseCppApp-11_6_0) uWS::HttpSocket<0>::upgrade 0x00007FFD47ACCA64 (FirebaseCppApp-11_6_0) uS::TLS::Context::operator bool 0x00007FFD47AB65B7 (FirebaseCppApp-11_6_0) uS::TLS::Context::operator bool 0x00007FFD47A99FF1 (FirebaseCppApp-11_6_0) uS::TLS::Context::operator bool 0x00007FFD47A9A7B6 (FirebaseCppApp-11_6_0) uS::TLS::Context::operator bool 0x00007FFD47A92CE9 (FirebaseCppApp-11_6_0) uS::TLS::Context::operator bool 0x00007FFD47A88758 (FirebaseCppApp-11_6_0) uS::TLS::Context::operator bool 0x00007FFD47A33A3C (FirebaseCppApp-11_6_0) uS::Socket::freeMessage 0x00007FFD47A3CF0F (FirebaseCppApp-11_6_0) uS::Socket::freeMessage 0x00007FFDC0781BB2 (ucrtbase) configthreadlocale 0x00007FFDC24F7374 (KERNEL32) BaseThreadInitThunk 0x00007FFDC2E5CC91 (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========

I've disabled all Windows Defender and I'm not using a VPN, I'm just using my home Wi-Fi.

Is this a bug or something else? My project has been stuck for days because of this issue. I only use FirebaseAuth and FirebaseDatabase.

Unity 2022.3.62f1

[timera]

Updating Microsoft Visual C++ 2015-2022 Redistributable (x64) solved it for me

https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170#latest-supported-redistributable-version

[adabru]

I'm observing the same crash-log with multiple lines of FirebaseCppApp-12_10_0) uWS::HttpSocket<0>::upgrade. I'm observing it with version 12.4.0 and 13.4.0.

It doesn't crash on the very first app start. It crashes on the second start.

It doesn't crash on the first start after I've deleted the two folders %LOCALAPPDATA%\firestore and %LOCALAPPDATA%\firebase-heartbeat. It crashes on the second start.

It doesn't crash on the first start after updating the Microsoft Visual C++ 2015-2022 Redistributable (x64). It crashes on the second start.

It crashes with the following code:

await FirebaseApp.CheckAndFixDependenciesAsync();
FirebaseFirestore.DefaultInstance.Settings.PersistenceEnabled = true;
var userCredential = await auth.SignInWithEmailAndPasswordAsync("<email>", "<password>");
var user = userCredential.User;
await FirebaseFirestore.DefaultInstance.Collection("User").Document(user.UserId).GetSnapshotAsync();

It also crashes with the following code:

await FirebaseFirestore.DefaultInstance.TerminateAsync();
await FirebaseFirestore.DefaultInstance.ClearPersistenceAsync();

The only used dependencies:

• First test: com.google.external-dependency-manager-1.2.183.tgz com.google.firebase.app-12.4.0.tgz com.google.firebase.auth-12.4.0.tgz com.google.firebase.firestore-12.4.0.tgz
• Second test: com.google.firebase.app-13.4.0.tgz com.google.firebase.auth-13.4.0.tgz com.google.firebase.firestore-13.4.0.tgz com.google.external-dependency-manager-1.2.186.tgz

[adabru]

This issue looks very similar to https://github.com/firebase/firebase-unity-sdk/issues/561, which was closed due to inactivity.

[adabru]

I've created a MRE for this crash: https://github.com/adabru/MRE-firebase-crash

You need to click init, then login, then get document. It will crash on the second start when you click get document.

I've created another MRE with the firebase cpp sdk that would call:

// source: https://github.com/adabru/mre-firebase-crash-cpp/blob/main/src/main.cpp
// The following is an edited extract from that file
firebase::App *app = firebase::App::Create(options);
firebase::auth::Auth *auth = firebase::auth::Auth::GetAuth(app);
firebase::auth::User user = await(auth->SignInWithEmailAndPassword(email.c_str(), password.c_str()));
firebase::firestore::Firestore *firestore = firebase::firestore::Firestore::GetInstance(app);
firestore.set_persistence_enabled(true);
firebase::firestore::DocumentReference doc_ref = firestore->Collection("User").Document(user.uid());
firebase::firestore::DocumentSnapshot doc_snapshot = await(doc_ref.Get());

That code runs fine with the Firebase cpp sdk version 13.0.0, it does not crash on the second start.

Now I'm a little lost of what to do next. @argzdev do you have a wiki that would explain to me how to debug the firebase-unity-sdk? I can try building it and putting it into the Unity Editor after every change. But that is rather time-expensive. Do you use some tools to run the DLLs from CLI or something like that, that would allow for faster iterations than building and putting it into the Unity Editor?

[ManuelWessely]

Some details on the firebase folders

I added some delete buttons to the MRE from @adabru

1. A button that deletes the %LOCALAPPDATA%\firebase-heartbeat
2. One that deletes all files in %LOCALAPPDATA%\firestore except the .ldb database files
3. One that deletes %LOCALAPPDATA%\firestore (including database files)

Using 1 and 2 was not enough to resolve the error. Apparently, The .ldb database files or a combination cause the issue.

There is another interesting behaviour:

Online use

• If I create data while connected to a remote firestore, deleting these folders has no visible effect. The data is still (or again) there as expected

Offline use

• If I create data while there is no internet (persistence enabled), the data appears as a new .ldb entry in the %LOCALAPPDATA%\firestore folder
• If I delete the %LOCALAPPDATA%\firestore folder, this new data is lost. However the data that was there before remains
• Going online after creating offline data, does not remove the .ldb entries (another one appeared)
• Deleting the %LOCALAPPDATA%\firestore in online mode has no visible effect. The data is still (or again) there as expected.

[adabru]

This bug could be reproduced on a Window 11 VM installed on macOS via Parallels. It appears on 2 out of 2 Windows VMs which were set up this way. It also appears on a native Windows laptop that I don't have access to.

[TeorikDeli]

Well, we can't release our game to Steam because of this bug. At least we did get rid of realtime database, firestore and firebase storage. Next is analytics & remote config. I am so tired of struggling with Firebase related crashes, bugs and problems. Nobody at Google seems to try to solve any serious issues. Unity with firebase crashes, Player with firebase crashes, macOS with firebase Auth doesn't work (but at least was not a crash).

[EllieTellie]

I mean in theory it could be related to https://github.com/firebase/firebase-cpp-sdk/commit/a06d206c2b6ac45e45cee7652d046c30df0b7c61 as it's one of the few commits going into 12.10.0.

MutexLock lock(*g_registration_token_mutex);

It introduces new locking in this commit. It's possible that this breaks on the older/broken MSVC++ runtimes which is fixed on more modern runtimes.

Would probably be worth investigating @argzdev I know it's not 'fun' to try to repro this but I would say messaging (considering it is doing sockets for subscribing) is the most likely thing to deadlock/crash here.

[EllieTellie]

Maybe related to this: https://github.com/microsoft/STL/issues/2285 i.e.: Firebase SDK is compiled with contextexp mutex but end user has incompatible vcruntime?

[EllieTellie]

Actually more likely the commit I mentioned above is harmless but we know it is a MSVC bug so it's feasible the compiler changed in 12.10.0 as it uses windows-latest in GitHub workflow. Some code like Firestore and RemoteConfig uses std::mutex and not the Firebase mutex which uses native Win32 apis. Again I don't know enough about this to confirm that is the issue, but it sounds plausible.

[EllieTellie]

I could try and compile Firebase manually with the older VS 17.9 and see if that resolves the issue.

[TeorikDeli]

Until we can replace Remote Config with Unity's or our's, I did delay the remoteconfig fetch about 5 seconds. Right now it is not causing crashes most of the time (at least we can pass the steam review). But it is kinda random. We'll remove it when we can. Even the old versions (12.5.0) didn't work for us. BTW, even changing Firebase log level with FirebaseStorage.DefaultInstance.LogLevel caused crashes for us. Maybe the reason is using FirebaseStorage.DefaultInstance, idk.

[adabru]

@EllieTellie

I could try and compile Firebase manually with the older VS 17.9 and see if that resolves the issue.

That would be nice. ManuelWessely could test your compiled firebase-unity-sdk on his problematic Windows VM. You could also share the steps you used to compile the dlls with us so we can use them when we try something out with the firebase-unity-sdk code ourselves.

When I compiled a cpp version of the problematic code with the prebuilt firebase-cpp-sdk, the crash did not happen (https://github.com/firebase/firebase-unity-sdk/issues/1291#issuecomment-3458985410). Maybe compiling the firebase-cpp-sdk with a "problematic" MSVC version might actually break that cpp version? If you're compiling the firebase-cpp-sdk at some point, that could be something worth trying out. It's probably not worth it, I guess.

@TeorikDeli how did you delay the remote config fetch?

[TeorikDeli]

@adabru with a basic IEnumerator yield return new WaitForSeconds(5f); before the FirebaseRemoteConfig.DefaultInstance?.FetchAsync().ContinueWithOnMainThread(OnFetchComplete)

[EllieTellie]

@adabru I've managed to finally build some packages with add_compile_definitions(_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR) in the CMakeLists.txt

This was built from commit: 8bc7cf9beb742c880b03e95f549c08fcbf8a0499 and d29967394fd32a16cb11aaa7dc933486ee9e4550 for firebase-unity-sdk and firebase-cpp-sdk respectively.

Essentially just: python scripts/build_scripts/build_zips.py --platform=windows and then copied over the cpp dlls.

You can find them here: https://drive.google.com/drive/folders/1gl6QO9w747UuO5HC3uGheBffdAqoUtAS?usp=drive_link

Use at your own risk I've deployed these for testing, but I've not checked if it fixes this issue as I don't have the old MSVC version on my PC atm. I also only need these 5 packages.

[adabru]

@EllieTellie thanks for the update and uploading the files! To test the MRE, i would need the com.google.firebase.auth and com.google.firebase.firestore as well. Would it be possible for you to compile and upload them as well?

[EllieTellie]

@EllieTellie thanks for the update and uploading the files! To test the MRE, i would need the com.google.firebase.auth and com.google.firebase.firestore as well. Would it be possible for you to compile and upload them as well?

I've uploaded the patched .tgz for auth and firestore and I've also uploaded the zip of the cpp files that were recompiled in case someone wants to try it. Same link: https://drive.google.com/drive/folders/1gl6QO9w747UuO5HC3uGheBffdAqoUtAS?usp=drive_link

My confidence on the fix is still low, but it was interesting and fun learning to rebuild the cpp libs.

[EllieTellie]

Ok I've finally gotten around to install "Visual C++ Redistributable for Visual Studio 2019 v16.11" which reproduces the issue. I switched to my fixed build and it no longer reproduces the crash. I am pretty sure recompiling it with _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR will fix it going forward.

Steps to reproduce:

• Download Visual C++ Redistributable for Visual Studio 2019 v16.11 and install it
• Extract DLLs from C:\ProgramData\Package Cache{...}v14.29.30156\packages\vcRuntimeMinimum_amd64\cab1.cab with 7-Zip
• Copy vcruntime140.dll, vcruntime140_1.dll, msvcp140.dll to game folder
• Launch game

I'll attach the dlls to this comment as well to save you having to extract it as it does not like downgrading my currently installed version. I.e.: even if you install older versions my System32 versions are still the newest version I had previously.

I've attached the patch for the Firebase Unity SDK although it would not hurt to also have it applied to the CPP Firebase project I imagine. @argzdev could I beg you to have a look over this please?

msvcp140_v16.11.zip

firebase_win32_patch_cmake.patch

[EllieTellie]

And another thread about the solution: https://developercommunity.visualstudio.com/t/Access-violation-in-_Thrd_yield-after-up/10664660

[adabru]

Hi @EllieTellie, thanks a lot for your work :)

Crash reproduction with redist v16.11

So I did the following:

• Install Visual C++ Redistributable for Visual Studio 2019 v16.11 from https://my.visualstudio.com/Downloads?q=Visual%20Studio%202019
• Reboot.
• Copy the three dll's from https://github.com/firebase/firebase-unity-sdk/issues/1291#issuecomment-3633097571 into the built MRE's folder.
• Start the MRE.

Now, when I click on Init, I'm getting a crash with the following in the Player.log:

========== OUTPUTTING STACK TRACE ==================

0x00007FFADBBD3080 (MSVCP140) Thrd_yield
0x00007FF9FB58B0BA (FirebaseCppApp-13_4_0) uWS::HttpSocket<0>::upgrade
0x00007FF9FB57E9DA (FirebaseCppApp-13_4_0) uWS::HttpSocket<0>::upgrade
0x00007FF9FB57D9F7 (FirebaseCppApp-13_4_0) uWS::HttpSocket<0>::upgrade
0x00007FFAF9A837B0 (ucrtbase) wcsrchr
0x00007FFAFA52E8D7 (KERNEL32) BaseThreadInitThunk
0x00007FFAFC0CC53C (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========

The Player.log is attached.

Crash fix with patched Firebase libraries

Then I did the following:

• Remove com.google.firebase.{app,auth,firestore}-13.4.0.tgz from Unity package manager.
• Download and add the packages com.google.firebase.{app,auth,firestore}-12.10.1.tgz from your shared Google Drive link.
• Clean build MRE.
• Ensure the three DLLs are still in the built MRE's folder.
• Start the MRE.

Now, there is no crash and I can retrieve the document successfully.

Comparison to ManuelWessely's system

I had a call with ManuelWessely (colleague). He tried it on the Parallels Windows VM on his Mac device.

The crash that he observes there is a little different from the crash that I could reproduce with your instructions. ManuelWessely's MRE only crashes when clicking GetDocument. Clicking Init and Login works fine, without crashing.

The patched Unity libraries sadly didn't fix the crash on his system.

More updates

The version on ManuelWessely's device stopped crashing after he did a Windows update today. He reproduced it once by setting up the Parallels Windows VM from scratch, then trying before and after the update.

I plan to check with our customers next if either the Windows update or the MRE with your patched Firebase Unity libraries can prevent the crash.

[EllieTellie]

Hi @adabru,

I would be interested to see what stacktrace that is producing on the Windows VM. Are we sure it's not just a Parallels VM oddity at that point?

Changing the _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR should definitely fix the mutex crash at least as this fixes the binary code generated to be backwards compatible. I'm also not a CMake wizard, so while I think it is correct I don't know if it applies to all linked libs.

For the record I only use remote config for standalone Windows and I don't use Auth or Firestore so I have no way of testing that myself.

[adabru]

Are we sure it's not just a Parallels VM oddity at that point?

We'll find out when our customer tries it on his native Windows device.

Here ist the Player.log.txt. It's using the com.google.firebase.{app,auth,firestore}-12.10.1.tgz packages from the shared Google Drive.

[adabru]

The customer's native Windows device is crashing as well with the libraries from the Google Drive: Player.log.

Intel(R) Celeron(R) N4120 CPU Edition Windows 11 Home Version 24H2, build 26100.7462 After updating, still crashing: Windows to 25H2, build 26200.7462

Microsoft Visual C++ v14 Redistributable (x64) - 14.50.35719