sentry-react-native
sentry-react-native copied to clipboard
getsentry
Flaky E2E Metrics App Start Time
Description
Can we improve the app start time measurement tests?
Often the test fails (looking at the last 30 commits on main cca 1/3 where only metrics failed) because the App With Sentry started faster.
https://github.com/getsentry/sentry-react-native/actions/runs/6710776019/job/18236705951
@vaind I've seen there are some conditions in place to avoid this, is there something else that we could improve?
Recently I also noticed this
https://github.com/getsentry/sentry-react-native/actions/runs/6954144218/job/18920431622
StartupTimeTest > startup time() FAILED
org.openqa.selenium.WebDriverException: An unknown error has occurred
Build info: version: '4.2.1', revision: 'ac4d0fdd4a'
System info: host: 'Mac-1700637830941.local', ip: '10.213.5.236', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '12.7.1', java.version: '11.0.21'
Driver info: io.appium.java_client.ios.IOSDriver
Command: [744b0c72-5b9d-4a04-b3a2-c9f0a4490f4c, terminateApp {bundleId=io.sentry.rn.perf-test-app-sentry}]
Capabilities {appium:appiumData: {appiumVersion: 2.0.0, automationName: XCUITest}, appium:automationName: XCUITest, appium:chromeOptions: {w3c: false}, appium:commandTimeouts: {default: 60000}, appium:databaseEnabled: false, appium:deviceContextId: b77f8d23-2f05-4e32-81a8-d89..., appium:deviceName: 00008101-000559D20203001E, appium:disableWindowAnimation: true, appium:javascriptEnabled: true, appium:jobUuid: b68759cda36f451eb12b6484163..., appium:locationContextEnabled: false, appium:networkConnectionEnabled: false, appium:orientation: PORTRAIT, appium:platformVersion: 14.8, appium:takesScreenshot: true, appium:testobject_device: iPhone_12_Pro_Max_real_us, appium:testobject_device_name: iPhone 12 Pro Max, appium:testobject_device_session_id: 926ee55c-d870-4465-b171-dc5..., appium:testobject_test_report_api_url: https://api.us-west-1.sauce.../, appium:testobject_test_report_url: https://app.saucelabs.com/t..., appium:testobject_user_id: sentryio, appium:udid: 00008101-000559D20203001E, appium:usedCachedDevice: false, appium:webDriverAgentUrl: http://127.0.0.1:5720/, appium:webStorageEnabled: false, browserName: safari, platformName: IOS, sauce:options: {appiumVersion: 2.0.0, build: CI getsentry/sentry-react-n..., name: Performance tests, realDevice: true, tags: [ios, ci]}}
Session ID: 744b0c72-5b9d-4a04-b3a2-c9f0a4490f4c
at [email protected]/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at [email protected]/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at [email protected]/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at [email protected]/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:53)
at app//org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
at app//io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:180)
at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:551)
at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:606)
at app//io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:[262](https://github.com/getsentry/sentry-react-native/actions/runs/6954144218/job/18920431622#step:14:267))
at app//io.appium.java_client.CommandExecutionHelper.execute(CommandExecutionHelper.java:27)
at app//io.appium.java_client.InteractsWithApps.terminateApp(InteractsWithApps.java:179)
at app//io.appium.java_client.InteractsWithApps.terminateApp(InteractsWithApps.java:163)
at app//StartupTimeTest.collectAppStartupTimes(StartupTimeTest.kt:194)
at app//StartupTimeTest.access$collectAppStartupTimes(StartupTimeTest.kt:16)
at app//StartupTimeTest$collectStartupTimes$1.invoke(StartupTimeTest.kt:114)
at app//StartupTimeTest$collectStartupTimes$1.invoke(StartupTimeTest.kt:100)
at app//TestBase.withDriver(TestBase.kt:26)
at app//StartupTimeTest.collectStartupTimes(StartupTimeTest.kt:100)
at app//StartupTimeTest.startup time(StartupTimeTest.kt:44)
We could ignore the invalid results. And not mark the job as failed.