getsentry
Symbolisation without fno-unwind-tables
Problem Statement
A customer reported they were capturing Issues in Sentry without symbolised stacktraces.
"For some reason, the stack consists of just one line. This happens on Android Prod builds (Google Play). On iOS builds, everything is fine, though."
"We discovered that it was the -fno-unwind-tables option. If it is there, there is a problem with symbolisation.
If we don’t use this option for prod builds, the build size increases a lot (~3Mb). For us, this is critical.
When we used AppCenter, it could symbolise without this option.
Is there any possibility not to use this option and get normal stack symbolisation?"
Solution Brainstorm
A possibility not to use fno-unwind-tables and get normal stack symbolisation.
Sentry does not rely on frame pointers as they are not mandatory per the platform rules. Using them when the compiler is using them for other purposes is super unsafe, thats why Sentry does not allow the solution being requested.
I think that it could be an option for client's settings. If right now frame pointers work on Android platform, why not use it and save some space in the build. If one day it stops working, we could switch to current solution. I understand that it could be complicated, but this space is really important for us.
You are right that this could be configurable. I rather tend to a compile time flag rather than a runtime setting though.
The situation with frame pointer unwinding and ABI rules is complicated. As a counter example, on Apple ARM64, the rules about LR / FP usage are very strict, and you are guaranteed to be able to use frame pointer unwinding. However, on Linux / Android, the rules are a lot looser. I have seen examples in the wild where the LR register was used as a general purpose register, as the ABI rules allow that. Not to mention the whole debate in the Linux Desktop land about enforcing frame pointer usage. Admittedly, thats about x64 and Desktop Linux, but I assume the rules for Linux on Android are also not as strictly enforced as on Apple. Frame pointer unwinding is unfortunately an all or nothing approach that all the system libraries, and third party libraries need to agree on. Which is hard to do considering the multitude of third party Android flavors.
Doing frame pointer unwinding safely also requires process_vm_readv which is only available on Android API > 17. That is crazy old, but still something our SDK is concerned about. (any update on that @kahest ? :-D)
All that being said, I don’t think we can bake in frame pointer unwinding safely into our SDK as a default just yet, but it should be possible to offer an opt-in option, either via a build time flag, or a runtime option.
Another thing to note is that also unwind info is not always available as your example shows, and the same restriction on process_vm_readv apply to do it safely.
Speaking about Android API level. I think that API level 19 (Android 4.4) is now de facto standard. Older Android versions are extremely rare. And more than that, they generally don't work properly, so they can't be normally supported along with modern Android versions. At least situation when crashes on Android 4.2 - 4,3 won't symbolicate - this doesn't bother us at all.
Build time flag, which can be enabled or disabled for a certain OS, sounds great.
Hi. As I get it you've decided to implement an option enabling frame pointers. But it's not exactly what we need. The point is, before using Sentry(with AppCenter) we could disable both FP(-fomit-frame-pointer) and unwinding(-fno-unwind-tables) for our Android Prod builds and somehow AppCenter managed to "do the trick" and show crash stack. I've not seen AppCenter sources, so don't know the details how AppCenter does it. As a chance, they might send whole thread stack dump to the server. Then using debug info and this dump to unwind the crash stack with custom unwinding tool. May be this issue is a good opportunity for your team to test AppCenter and unravel the technique.
We have our own pipeline symbolicating "minidump" files on the server side, using unwind info and debug info that you upload. In theory you can then strip that unwind info from the shipped binaries.
However shipping breakpad or any other solution to capture minidump files on the client also adds to the app size, so there is no silver bullet here.
Correct me please if I get it wrong. It sounds like you don't need unwind tables in binaries on the client(inside our APK), only in binaries(with DebugInfo) we upload to Sentry. Am I right?
I am little confused because our experiments with Android Sentry showed that there was no full stacktrace in report(only last line) if we shipped app without unwind tables to the client itself. I thought you unwind crash stack on the client itself. Is it so?
If not(you do unwinding it on server side) it means we might try to find a way of how to strip unwind tables from APK binaries in Android Gradle Plugin :app:stripReleaseDebugSymbols build step. By default it strips only DebugInfo and there is no way to pass extra options to strip command as I know. Have to think. May be you have an idea? As a variant we might make a custom shell script to reassemble APK with manually stripped binaries, just after gradle finish.
Sorry about the confusion here.
The Sentry Android SDK does stack traces on the client. On desktop, we have a way to capture a minidump and do the stack traces on the server side. This is not enabled on Android.
We did some experiments a long time ago, and it is in theory possible to do, but has other tradeoffs. Namely, we don’t have any unwind or debug info for system libraries on Android. Thats why we prefer client side stack traces on that platform, as all the system libraries are available on the client side. Obviously, you need either unwind info, or frame pointers in that case, which is also not a given.
Long story short, the situation is complex, and there is no good solution that ticks all the boxes.
Hey @viacheslav-e, I'm still wondering if using frame-pointer enabled builds would be an option for you.
As mentioned elsewhere unwind tables add ~3MB to your builds, any chance you could measure the impact of frame pointers? (-fomit-frame-pointer vs -fno-omit-frame-pointer)
Exactly right.
Frame pointers is a point of consideration for us at the moment. We have to make experiments with -fno-omit-frame-pointer to estimate impact on size and performance
Hi. We've done several tests with -fno-omit-frame-pointer and decided that it's more preferable variant than -funwind-tables.
Frame-pointer has lass impact on size: ~150Kb(FP) vs 3Mb(UT) in one of our app.
Performance impact might be neglected. Especially on armv8(our main arch).
Conclusion: Yes, frame-pointer is an option for us.
Quick update - we will likely go with the framepointer approach. We're currently ironing out the how's and when's, so no concrete timeline yet.