Partial source context in Release with multi-project MAUI apps

[NathanielJS1541]

Package

Sentry.Maui

.NET Flavor

MAUI

.NET Version

9.0.301

OS

Android

OS Version

Android API 34/35.

Development Environment

Visual Studio v17.x

SDK Version

6.10.0

Self-Hosted Sentry Version

No response

Workload Versions

Installed Workload Id      Manifest Version       Installation Source
---------------------------------------------------------------------
android                    35.0.61/9.0.100        VS 17.14.36203.30
ios                        18.4.9288/9.0.100      VS 17.14.36203.30
maccatalyst                18.4.9288/9.0.100      VS 17.14.36203.30
maui-windows               9.0.51/9.0.100         VS 17.14.36203.30
wasm-tools                 9.0.6/9.0.100          VS 17.14.36203.30

Use `dotnet workload search` to find additional workloads to install.

UseSentry or SentrySdk.Init call

            var builder = MauiApp.CreateBuilder();
            builder
                .UseMauiApp<App>()
                .UseSentry(options => {
                    options.Dsn = "";

                    // Set SDK to debug mode, as this is a repro app.
                    options.Debug = true;

                    // Application release and distribution.
                    options.Release = $"maui-sentry-test@{AppInfo.VersionString}";
                    options.Distribution = AppInfo.BuildString;

                    // Set the environment based on build configuration.
#if DEBUG
                    options.Environment = "Development";
#else
                    options.Environment = "Production";
#endif
                })

Steps to Reproduce

1. Clone my repro project.
2. Follow the README.md to configure the Sentry application and organisation slugs, and DSN.
3. Build the app in Debug mode, and deploy to a device (i.e. Android).
4. Launch the app and click the CommonService Exception button.
5. Check your Sentry issues for the report, observe the behaviour from the "Expected" section.
6. Build the app in Release mode, and deploy to a device.
7. Launch the app and click the CommonService Exception button.
8. Check your Sentry issues for the report, observe the behaviour from the "Actual Result" section.

This issue seems to be exhibited specigically when an exception occurs in a project separate to the main MAUI app, and the build is trimmed / optimised.

Expected Result

The stack trace will show line numbers and source context for all managed source:

Actual Result

In release mode, the stack trace only shows line numbers and source context for code within the "App" project:

I will note that I'm also seeing similar issues on iOS, but currently I can only deploy the test app to the simulator and it doesn't seem to be an issue there. Windows doesn't seem to show the same issue.

[NathanielJS1541]

Another thing that might be worth noting is that I'm seeing similar issues with areas that appear to be optimised, such as methods that are inlined. For example, clicking the InternalService Exception on a Release Android build results in the following stack trace:

This leads me to believe that these issues may be to do with trimming / optimisations, and may need to be documented somewhere.

If I prevent the method being inlined, I get the stack trace I would expect:

Although these cases appear to be different as the stack trace doesn't show the method call in the stack trace at all.

[NathanielJS1541]

An example of this that we're seeing in production (that lead me to believe this issue was related to our multi-project structure) is this:

In green, you can see code in our "App" project with line numbers and source context, and in red two method calls to our "Core" project with no line numbers or source context.

[NathanielJS1541]

I've been sent #4176 and #2180 by Sentry support, which may be related.

[jamescrosswell]

Thanks @NathanielJS1541 and thanks for the repro.

It might take me a little while to get to the bottom of this one (anything that only happens in release mode can be a bit tricky to debug).

Let's focus on the Android side of things first and then see if any of what we uncover there can be applied to iOS. They're quite different beasts (only very small portions of Android applications get AOT compiled and Mono is only ever used on iOS when running on a simulator with the interpreter enabled).

[NathanielJS1541]

@jamescrosswell yeah I figured as much. Let me know if there's anything I can help with. I'll keep having a look to see if I can see anything else too.

One thing I did notice is that in the Sentry web interface, it doesn't seem to try and load the "images" for the "Common" project in Release:

Whereas in Debug, it loads both "images" the different projects:

You can see it's also showing missing symbolication for various libraries too in Debug, which it isn't in Release.

[jamescrosswell]

You can see it's also showing missing symbolication for various libraries too in Debug, which it isn't in Release.

Hm, those I would expect to be pulled from Microsoft's symbol servers (there's no point in pushing up one copy per Sentry customer for them).

What do you see in your the Built in Repository settings for your project? Ideally it would look something like this:

[NathanielJS1541]

Hm, those I would expect to be pulled from Microsoft's symbol servers

I would've expected that too... Here's my project's built-in repository settings look like:

[bruno-garcia]

It's possibly that during optimization some stuff is being lost. Specifically framework libraries. In that case we'd need to raise with the MAUI team and see if there's an option to tell the compiler to keep enough metadata at runtime for us to collect and later find the debug files and look up details. I don't imagine the issue is lack of data in debug files themselves since symbolication is server side.

[jamescrosswell]

@NathanielJS1541 I can reproduce this (when building in Release mode), which is a good start - thanks for the repro.

Now I just need to go deep down the logging rabbit hole to work out why this is happening...

[jamescrosswell]

So it looks like when there are multiple assemblies in an application, the naming for those assemblies in the AssemblyStore is a bit different. Specifically, the name includes a file extension (.dll).

The module name that we are using to look for the best assembly does not include any file extension, so we're not resolving the assembly from the assembly store.

So there are 4 different permutations:

Module Name	Assembly Store Name
Foo	Foo
Foo.dll	Foo
Foo	Foo.dll
Foo.dll	Foo.dll

We currently match 1, 2 and 4 but not 3.

We'll have to be a bit looser about how we do the matching then...

[jamescrosswell]

I've resolved one problem (the assembly reader wasn't finding the assembly for the module) in:

• https://github.com/getsentry/sentry-dotnet/pull/4294

However this doesn't appear to fully resolve the issue. See here for an example (JSON attached here also):

• event.json

In the JSON you can see the debug id is now correctly resolved:

       "code_id": "e0fbe16e8000",
        "code_file": "MauiSentryTest.Common",
        "debug_id": "2759ae4e-b3db-4dac-a8bd-980cc2b473a4-fb486b0b",
        "debug_file": "/Users/jamescrosswell/code/MauiSentryTest/src/MauiSentryTest.Common/obj/Release/net9.0-android/MauiSentryTest.Common.pdb",
        "debug_checksum": "SHA256:4eae5927dbb3ac3d28bd980cc2b473a40b6b487b1741ccc9ff4688aa9027ea8e",

And that debug file has been uploaded correctly to the project:

However symbolication still isn't happening:

All the necessary components appear to be in the stack trace, but it's still indicating "symbolicator_status": "missing":

            {
              "function": "void CommonService.PollService(DateTime pollTime)",
              "package": "MauiSentryTest.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
              "in_app": true,
              "data": {
                "client_in_app": true,
                "symbolicator_status": "missing"
              },
              "instruction_addr": "0x26",
              "addr_mode": "rel:0",
              "function_id": "0x5"
            }

@bruno-garcia any ideas what I could check next? It seems like I'd need to dive into the Symbolicator to try to debug this further...

[bruno-garcia]

I'm not sure what's going on. But the specialist on Symbolicator is @loewenheim. Any suggestions Sebastian?

[loewenheim]

@jamescrosswell The JSON for the linked example event differs from the one you attached—in the linked event, the frame in question appears to have been symbolicated correctly.

[jamescrosswell]

@loewenheim that's not what I'm seeing. By the "linked example" do you mean this?

However this doesn't appear to fully resolve the issue. See here for an example (JSON attached here also):

When I look at that event, this is what I see in the UI:

And in the JSON it's a wall of "symbolicator_status": "missing":

[loewenheim]

It appears that you don't necessarily get the same event when clicking on that link, and in some of the events the file was successfully fetched.

ETA: In this event the debug file wasn't applied because the event came in before the file was uploaded. In this slightly later event it was applied.

Is the second event how you would expect this to look or is there still something missing?

[bruno-garcia]

by default Sentry selects the 'recommended' event, which will consider things like having a trace, screenshot, etc.

But in that case, the URL has /recommeded on it. e.g; this. vs this event which has the event-id in the URL instead of /latest or /recommended etc

[jamescrosswell]

I think that's the source of the confusion, thanks @bruno-garcia... I configured Sentry to show the "Lastes" event ages ago and forgot I'd done this.

@loewenheim the specific event that I actually meant to give you a link to then is:

• https://sentry-sdks.sentry.io/issues/6689281761/events/77b949228a0c4456b6a6ca05a7a9fefa/

If you click on that link, you can see all of the behaviour I described in the comment above.

[loewenheim]

In the linked event, the UI claims that the debug file was uploaded 8 minutes after the event came in.

[jamescrosswell]

Curious... the debug files are being uploaded as a build action. So the upload itself must (of necessity) happen before the event gets sent... but maybe the debug file was processed by Sentry after the actual event was processed (different ingest queues or whatever).

In any event, after reprocessing the event, it's now correctly symbolicated.

Thanks @loewenheim ! btw: after seeing your screenshot above I hunted through the docs and just found this (which is magic). That section is collapsed by default and I'd never noticed that it could be expanded, so hadn't noticed it before (in over 2 years working on this SDK)... We could maybe consider expanding that section by default if/when there are issues symbolicating an event.

[loewenheim]

Yeah the Images Loaded section is frustratingly unobtrusive. As an additional tip, you can also click on a frame's instruction address and it will take you to the corresponding image.