Crashpad_handler need to be reinstalled during runtime

[arijith96]

Description

When does the problem happen

• [ ] During build
• [ ] During run-time
• [x] When capturing a hard crash

Environment

• OS: [e.g. Windows 10, 64-bit] Windows 10, 64-bit
• Compiler: [e.g. MSVC 19] MSVC 19
• CMake version and config: [e.g. 3.17.2, SENTRY_BACKEND=inproc] 3.16.4, SENTRY_BACKEND=crashpad_handler

Steps To Reproduce

Log output I too am facing the same problem as that of this. But the difference here is I am using Qt instead.

I am able to get a crash report in Sentry if the hard crash is performed immediately after I initialize Sentry, but if I perform the crash a while later during the runtime of the application, it is not being reported.

int* a = nullptr;
*a = 16;

Funnily enough, this was not the case last week. It was functioning as expected and used to report hard crashes at any time during the runtime too. It is only starting this week that I am no longer able to do it. I can confirm that I have not made any code changes that would even slightly affect Sentry.

Update -> Performing a sentry_reinstall_backend(); right before manually performing the hard crash seems to make the crash get reported, but this is not applicable in real time.

Could you please help triage the issue?

Thanks

[Swatinem]

As you have mentioned yourself, this might be related to other libraries overwriting the global unhandled exception filter.

I’m not too familiar with Qt, how it handles things internally or if it even builds on top of WinUI or similar internally.

However, if they are setting an unhandled exception filter for some purpose, it is their responsibility to reset the previously installed one.

[arijith96]

Thanks for you comments.

But it still does not explain how it worked as expected before (i.e. last week) and all of a sudden stopped and need to be reinstalled again. Once again, I am sure that none of my other code changes affect Sentry at all.

Any other idea what could be going wrong?

[Swatinem]

We did release a new sentry version recently, which also brought with it a code bump of crashpad.

Since you say you have an earlier version around that works fine, can you verify that the older version still works (as in: it did not break due to a system library update); and also verify which dependencies were updated alongside sentry (and: did you bump the sentry version; from which version to which?) that might have an influence on this?

[arijith96]

Oddly, the previous version of my code with sentry also does not seem to work anymore (only crashes that occur right after sentry_init are sent) 😮‍💨

I will test your other suggestion and follow up.

[arijith96]

print("Sentry Crash point 3");
a = nullptr;
//*a = 16;
mMainWindow->show();
print("Sentry Crash point 4");
a = nullptr;
*a = 16;

In the above code, I am getting a crash report in sentry when the crash occurs on Sentry Crash point 3 but not on Sentry Crash point 4, where mMainWindow->show() is QWidget::show()

Is this now a problem with Qt?

Also I noticed that sentry is not logging to my MSVC Output window. Is there anything that I need to do other than sentry_options_set_debug(options, 1) to get it to print useful information? I think seeing the logs could really help solve this.

[Swatinem]

set_debug is the right thing, yes. By default it logs to stderr, not sure if that is the right thing for it to end up in the MSVC output window. You can also define and hook up your own logger, so you can redirect the log wherever you want.

Have you checked if your Qt version or any other system libraries were updated in the meanwhile? Would be interesting to test with older versions as well.

[arijith96]

Thanks for the tip of redirecting the errors! I was able to get the following logs. (see image)

The left is performing a crash at the aforementioned Sentry Crash point 3 and the right is at Sentry Crash point 4.

Any idea what's going on?

BTW, The check of any update of libs is also parallelly ongoing. Will follow up.

[arijith96]

More updates that could help. (please see images)

Using the SetUnhandledExceptionFilter() function, I was able to conclude that after mMainWindow->show(), that whatever is used as the global unhandled exception filter is being replaced and "probably" set to NULL.

Global filter before calling mMainWindow->show():

Global filter after calling mMainWindow->show():

Any thoughts on how to work around this? I had an idea of making a re-install of the backend using a before_send() the moment we get a crash, but as stated in your docs, before_send() is not called before sending crashes using crashpad.

[Swatinem]

@arijith96 Thank you for the detailed analysis!

As you have shown, something inside of Qt, or one of its dependencies messes with the UnhandledExceptionFilter. I would say this is an upstream issue then, outside of our control :-(

[arijith96]

Hi @Swatinem, I have some updates again.

It seems like there are dependencies problems from the Intel(R) UHD Graphics drivers.

As I mentioned before, I started facing these issues with Sentry all of a sudden. Seems like my Dell notebook was updated on one specific reboot and it had also updated the driver for Intel(R) UHD Graphics drivers.

After some testing, I found that any version above 30.0.101.1191 no longer functions with the Senty SDK. Could be something that you might find interesting and worth further investigation from your side.

I rolled back from my updated version from 30.0.101.1404 to 30.0.101.1191 and everything works again! After making this rollback, the global exception handler is no longer being changed from Sentry and works as expected too. ( in regards to my previous update)

Similarly, devices with no Intel cards can also send crash reports without any issues, which is self-explanatory.

[Swatinem]

Wow, thanks for the detailed analysis here!

I’m afraid in this case there is even less that we can do about this :-(

There is no "you have been unhooked" callback if the 3D driver overrides the installed handler.

I would call this a Bug in their driver package then. Drivers should absolutely not mess with application-global state in such a way.

[Mixaill]

There is no "you have been unhooked" callback if the 3D driver overrides the installed handler.

Well, you can try to hook SetUnhandledExceptionFilter() and rejects other filters :D Something like https://gitlab.com/xvm/xvm-extensions/xfw.extension.bugreport/-/blob/master/src_native/uef_manager.cpp

(It hooks SetUnhandledExceptionFilter() via MinHook and then checks that UEF within our module address space. Primary goal was to achieve coexistence of 3rdparty crashpad and sentry-native)

[Swatinem]

That is indeed interesting and might work, though I think it might be a bit fragile, and probably not something we would want to actively maintain probably. Also we would probably have to do it for all of the backends then, which makes it even more of a burden to maintain.

[Mixaill]

Yeah, that's a way too hacky to be a part of sentry-native, but might be a working solution for @arijith96