sentry-java Sentry Android does not capture LogCat logs

Platform:

[x] Android -> If yes, which Device API (and compileSdkVersion/targetSdkVersion/Build tools) version?
[ ] Java -> If yes, which Java (and sourceCompatibility/targetCompatibility) version?
[ ] Kotlin -> If yes, which Kotlin (and jvmTarget) version?
[ ] NDK -> If yes, which NDK/CMake version?
[ ] React-Native -> If yes, which version?
[ ] Timber -> If yes, which version?
[ ] Log4j2 -> If yes, which version?
[ ] Logback -> If yes, which version?
[ ] Spring -> If yes, which version?

IDE:

[x] Android Studio -> If yes, which version?
[ ] IntelliJ -> If yes, which version?
[ ] Other -> If yes, which one?

Build system:

[x] Gradle -> If yes, which version?
[ ] Buck -> If yes, which version?
[ ] Bazel -> If yes, which version?
[ ] Maven -> If yes, which version?
[ ] Other -> If yes, which one?

Android Gradle Plugin:

[ ] Yes -> If yes, which version?
[x] No

Sentry Android Gradle Plugin:

[ ] Yes -> If yes, which version?
[x] No

Proguard/R8:

[x] Enabled
[ ] Disabled

Platform installed with:

[ ] JCenter
[ ] Bintray
[ ] Maven Central
[ ] Manually

The version of the SDK: 3.x.x

I have the following issue:

The description goes here ...

Steps to reproduce:

run sample

Actual result: Sentry Android does not capture LogCat logs

Expected result: when having things like Log.i(...), I hope sentry can automatically capture it. thank you!

Jul 24 '21 09:07 fzyzcjy

Thanks for raising @fzyzcjy. Last we looked at this it wasn't as straight forward as something like Timber. We'll do another investigation. Do you have suggestions on how to implement this and/or is willing to make a PR to add support?

Jul 25 '21 15:07 bruno-garcia

@bruno-garcia I also do not have quite good solutions. If there is an idea I can make a PR. In addition, it is not possible to let every library use Timber. For example, one third party library may print out some Log.w or Log.e which I want to capture and recognize (because it can be the cause of some bugs), but it is not possible now :(

However, IMHO this is a very useful feature!

Jul 26 '21 00:07 fzyzcjy

Now that we have attachment support it makes more sense to look into logcat again. It actually came up here: https://github.com/getsentry/sentry-java/issues/1615

If you're willing to discuss it would be nice to discuss ideas. @marandaneto I'm sure will have opinions here btw.

Very high level it would be nice to have an options: attachLogcat and the SDK automagically sends logcat files cap'ed at some size on each event captured. We could build in parts but the final version of such features would propagate this flag to sentry-native which would also support it, and events from any layer would include the attached logcat file. We could eventually opt-in by default to this feature too.

Jul 26 '21 22:07 bruno-garcia

@bruno-garcia Interested in the suggestion!

Jul 27 '21 01:07 fzyzcjy

it'd be possible ootb only via bytecode manipulation or executing logcat params... and writing the output to a file or parsing the text, of course. I'm not sure if it's a good idea though since it's rather a good practice to remove every logcat call from production apps via Proguard/R8 to reduce the app's size, there's some cost in it, but obviously, a nice feature to have. I'm just questioning the usage of logcat in production Apps here to identify issues.

Jul 27 '21 09:07 marandaneto

@marandaneto @bruno-garcia I realize this is REALLY VERY VERY useful, when I am debugging today. I am debugging a third party sdk, and finally one useful clue is found by examing system logcat. without using logcat to see what the 3rd party library logs out, I cannot find any useful clue.

Jul 31 '21 03:07 fzyzcjy

We can execute Runtime.getRuntime().exec("logcat -v time"); and listen the buffer in a while loop. https://github.com/pedrovgs/Lynx/blob/master/lynx/src/main/java/com/github/pedrovgs/lynx/model/Logcat.java

Mar 30 '22 20:03 marandaneto

Sounds great! Will it require extra permission or slow down the app - inside release app builds?

Mar 31 '22 00:03 fzyzcjy

Sounds great! Will it require extra permission or slow down the app - inside release app builds?

It does, but not for all versions apparently. https://github.com/pedrovgs/Lynx/blob/7b1f25172f2bd3f4563076dcd58ec532aebfb9f1/sample/src/main/AndroidManifest.xml#L20-L21 Well, ideally this is done on a background thread, but I don't have benchmarking data, it'd be an opt-in feature till we get proper feedback anyway.

Mar 31 '22 06:03 marandaneto

Aha, that is for android 2.x if the comments are right. Then almost nobody is using 2.x :)

Mar 31 '22 06:03 fzyzcjy

Use code below to upload logcat including native crash happened

fun initSentry(context: Context, dsn: String) {
    SentryAndroid.init(context) {
        it.dsn = dsn
        it.tracesSampleRate = 0.03
        it.addEventProcessor(object: EventProcessor {
            override fun process(event: SentryEvent, hint: Any?): SentryEvent? {
                Runtime.getRuntime().exec("logcat -t 1000 *:I").inputStream.use { input ->
                    context.openFileOutput("logcat.txt", MODE_PRIVATE).use { out ->
                        input.copyTo(out)
                    }
                }
                return super.process(event, hint)
            }
        })
        it.isEnableScopeSync = true
    }
    Sentry.configureScope { scope ->
        scope.addAttachment(
                Attachment(
                        context.filesDir.path + "/logcat.txt",
                        "logcat.txt", "text/plain"
                )
        )
    }
}

Apr 13 '22 15:04 kyze8439690

https://developer.android.com/studio/preview/features#logcat

Not sure if the logcat output has changed or the IDE is parsing it, but worth checking it when working on this issue, worst case we have to parse 2 formats.

May 12 '22 09:05 marandaneto

I suspect we can reuse this logcat parser from the Android framework itself https://cs.android.com/android/platform/superproject/+/master:development/tools/bugreport/src/com/android/bugreport/logcat/LogcatParser.java

Nov 16 '22 07:11 marandaneto

What approach would be most helpful - having breadcrumbs per log entry (only selected/filtered logs), having logs attached to crashes, or perhaps allowing both options?

Mar 17 '23 12:03 buenaflor

More options looks more flexible IMHO. If possible, it would be the best to make this a library or a separate module, such that I can even gather those logcat logs and send to my own logging infra (e.g. when there is no crash but has warnings so I would like to have a look)

Mar 17 '23 12:03 fzyzcjy

Having done our first feasibility checks (looking good! 🚀 ) we have a few options on how logs are collected and displayed on sentry.io A) Collecting the logs and adding them as an attachment This could be especially useful for crashes, to have a single file with the recent N log entries. Provides nice extra context, is only limited by attachment file size, but probably a bit hard to read and separated from the main sentry.io experience.

B) Collecting the logs as breadcrumbs Probably easier to consume, but this could get very noisy and is limited (default breadcrumb limit: 100). We could filter the logs (e.g. only show breadcrumbs for Warnings and Errors).

I'm wondering if there are more options or even providing both could make sense.

@kyze8439690 @fzyzcjy it would be great to get your thoughts on this to help us shape this feature!

Mar 20 '23 14:03 markushi

@markushi Congratulations on the checks! I personally prefer the former, since the latter is very limited in length.

In addition, I wonder whether it will be provided as a separate easy-to-use module? Then, for example, I can add a few lines (sentry_logcat_api_register_callback { log_message -> my_logging_service.add(log_message) }) besides the traditional Sentry initialization code, and then the logcat logs will be reported to my logging infra as well. IMHO, this will not cause heavy extra work, as long as the code has good modularization (which I think so)

Mar 20 '23 14:03 fzyzcjy

I don't think we should raise error events because we'd not have stack traces anyway (making it less useful?). I suggest adding all logs within the app as breadcrumbs, but having a sensitive min level, such as error, but configurable similar to how other logging integrations do: minBreadcrumbLevel = Error. If it's still too noisy before can still filter out crumbs using the beforeBreadcrumb callback or configure a higher minBreadcrumbLevel level if any.

Using a file requires constant io, has to be done in a background thread, and is aware of concurrency issues.

The question is to try it out and check how verbose it is, compare the trade-offs and decide based on user experience. Do we know any open-source app or library that is very verbose with logcat?

Mar 20 '23 14:03 marandaneto

We have transferred the logcat file to the sentry as an attachment. After describing the file type as 'text/plain', sentry can directly view the file content in text form on the web page without additional file download.

Mar 21 '23 02:03 kyze8439690

I also tend to go for the first solution since we use breadcrumbs to log user behavior instead of app logs.

Mar 21 '23 02:03 kyze8439690

@kyze8439690 repos for context https://github.com/kyze8439690/logcatviewer/blob/master/src/main/java/com/github/logviewer/FloatingLogcatService.java

Mar 21 '23 07:03 marandaneto

@kyze8439690 did you check on which Android versions your code running logcat during runtime works? After some checks this only works on API >= 23 and will reject if below that due to permissions.

Mar 21 '23 08:03 buenaflor

@kyze8439690 repos for context https://github.com/kyze8439690/logcatviewer/blob/master/src/main/java/com/github/logviewer/FloatingLogcatService.java

https://github.com/kyze8439690/logcatviewer/blob/master/src/main/java/com/github/logviewer/LogItem.java#L26

Mar 21 '23 08:03 kyze8439690

@kyze8439690 did you check on which Android versions your code running logcat during runtime works? After some checks this only works on API >= 23 and will reject if below that due to permissions.

Sorry, we didn't do a complete compatibility test because the logcat file upload is not for end users. But I think it is acceptable to support only API level 23 and above.

Mar 21 '23 08:03 kyze8439690

We've shipped this. See https://docs.sentry.io/platforms/android/configuration/integrations/logcat/

Apr 06 '23 07:04 romtsn

How many lines of logcat can be recorded right now?

Apr 06 '23 07:04 kyze8439690

Our logcat integration only includes creating breadcrumbs out of logs with a specified minLevel which you can configure in your build.gradle (which also includes logs by third-party libraries since our implementation is based on bytecode manipulation).

If you still want to log the last n number of logs as an attachment you can still do that as you currently do by adding an EventProcessor and injecting a logcat shell command.

Apr 06 '23 08:04 buenaflor

Is there a way to manual install Logcat Integration? I don't want to install with the Sentry Android Gradle Plugin. @buenaflor

Apr 21 '23 03:04 kasogg

@kasogg You would have to manually replace all of your Log.e/d/w... calls with SentryLogcatAdapter.e/d/w..., though the class is marked as Internal, so we do not guarantee api stability.

We can reconsider though, if there's a good usecase. Are you fine with manually replacing the log calls? We could make it officially public.

Apr 21 '23 19:04 romtsn

@kasogg You would have to manually replace all of your Log.e/d/w... calls with SentryLogcatAdapter.e/d/w..., though the class is marked as Internal, so we do not guarantee api stability.

We can reconsider though, if there's a good usecase. Are you fine with manually replacing the log calls? We could make it officially public.

My requirement is:

Only when the user agrees privacy policy can app upload the logcat message.
I want to upload not only app logs but also system logs(Not necessarily)
logs upload only when Crash/ANR

I find a workaround:

options.addEventProcessor(object : EventProcessor {
     override fun process(event: SentryEvent, hint: Hint): SentryEvent? {
          if (event.isCrashed || event.exceptions?.any { it.mechanism?.type == "ANR" } == true) {
                                    Runtime.getRuntime().exec("logcat -t 1000 *:I").inputStream.use { input ->
                                        application.openFileOutput("sentry_logcat.txt", MODE_PRIVATE).use { out ->
                                            input.copyTo(out)
                                        }
                                    }
                                    hint.addAttachment(Attachment(application.filesDir.path + "/sentry_logcat.txt", "sentry_logcat.txt", "text/plain"))
                            }
                            return super.process(event, hint)
                        }
                    })

Apr 27 '23 08:04 kasogg