sentry-java icon indicating copy to clipboard operation
sentry-java copied to clipboard

Add New UI Tests App for critical tests

Open krystofwoldrich opened this issue 1 year ago • 2 comments

:scroll: Description

This PR adds crash recovery and corrupted envelope recovery UI Tests for Sentry Android.

:bulb: Motivation and Context

  • Part of https://github.com/getsentry/projects/issues/165

:green_heart: How did you test it?

locally, ci

:pencil: Checklist

  • [ ] I reviewed the submitted code.
  • [ ] I added tests to verify the changes.
  • [ ] No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
  • [ ] I updated the docs if needed.
  • [ ] Review from the native team if needed.
  • [ ] No breaking change or entry added to the changelog.
  • [ ] No breaking change for hybrid SDKs or communicated to hybrid SDKs.

:crystal_ball: Todo

  • [ ] Add workflow
  • [ ] Add make command for local execution

krystofwoldrich avatar Oct 17 '24 14:10 krystofwoldrich

Performance metrics :rocket:

  Plain With Sentry Diff
Startup time 491.93 ms 568.02 ms 76.09 ms
Size 1.70 MiB 2.35 MiB 667.95 KiB

Previous results on branch: kw/test/add-critical-ui-tests

Startup times

Revision Plain With Sentry Diff
c2743c4cbd0c60dfaa11087b48de96976e8723f2 427.73 ms 439.66 ms 11.93 ms
d726180eca7e12811624974f887f16ce2eb5d9d3 388.76 ms 414.46 ms 25.70 ms
ffd2c3c5527a6efb94e5c7007fa58c2039795442 405.94 ms 419.46 ms 13.52 ms
8e916549ccebacbd74f94907f7fc59d270e7f9d8 399.11 ms 458.86 ms 59.75 ms
4e066a140530958ade3315a0cd00d5a196709d30 401.74 ms 460.76 ms 59.02 ms
34446d9234eea96c4dbf45ef90cdf7cae95c5762 408.82 ms 415.42 ms 6.60 ms
b4967a1003603607a8571d8343d1c642514b26ca 437.10 ms 492.22 ms 55.12 ms

App size

Revision Plain With Sentry Diff
c2743c4cbd0c60dfaa11087b48de96976e8723f2 1.70 MiB 2.35 MiB 667.95 KiB
d726180eca7e12811624974f887f16ce2eb5d9d3 1.70 MiB 2.35 MiB 667.95 KiB
ffd2c3c5527a6efb94e5c7007fa58c2039795442 1.70 MiB 2.35 MiB 668.15 KiB
8e916549ccebacbd74f94907f7fc59d270e7f9d8 1.70 MiB 2.35 MiB 667.95 KiB
4e066a140530958ade3315a0cd00d5a196709d30 1.70 MiB 2.35 MiB 667.95 KiB
34446d9234eea96c4dbf45ef90cdf7cae95c5762 1.70 MiB 2.35 MiB 667.95 KiB
b4967a1003603607a8571d8343d1c642514b26ca 1.70 MiB 2.35 MiB 667.95 KiB

github-actions[bot] avatar Oct 17 '24 15:10 github-actions[bot]

I'm currently waiting for the job to run multiple times (50-100x) to ensure it's stable, before merging.

krystofwoldrich avatar Oct 21 '24 12:10 krystofwoldrich

Run from stability testing https://github.com/getsentry/sentry-java/actions/runs/11439418099/job/31828821106?pr=3802 (27 runs, 1 emulator boot timeout)

krystofwoldrich avatar Oct 21 '24 14:10 krystofwoldrich