dotnet-client [Bug]: IOS Real Device Tests started failing

Description

Real Device tests started failing after upgrade to Appium dotnet client 5.1.0 and Selenium 4.23.0 on Saturday 10/08 on SauceLabs.

Environment

.NET client build version or git revision if you use some snapshot: 5.1.0 and Selenium 4.23.0
Mobile platform/version under test: IOS devices
Real device or emulator/simulator: Real Devices

Details

SauceLabs tests for iPhone 15 Pro Max Real Device started failing on Saturday 10th August.

Please provide more details, if necessary. Error returned from SauceLabs: "Status Code: 500 - Error while starting appium session: An unknown server-side error occurred while processing the command. Original error: No webview has been detected after 13926ms."

Aug 10 '24 01:08 agray

@agray, have you checked the https://github.com/appium/dotnet-client#compatibility-matrix?

Aug 10 '24 04:08 Dor-bl

I am using Selenium 4.23.0

Aug 10 '24 08:08 agray

And dotnet client version 5.0.0, works for you?

Aug 10 '24 09:08 Dor-bl

Could you attache the full Appium server log?

No webview has been detected after 13926ms thinks me that it was client independent behavior. It would be helpful to attach working Appium server log as well.

Aug 10 '24 20:08 KazuCocoa

Rolled back to Appium 5.0.0 and Selenium 4.22.0 and even less works. (I had originally skipped Selenium 4.22.0.)

https://github.com/Sauceforge/Saucery/commit/32ef411ac19b2c68a5215240caaf76d3c864ca31

NUnit Virtual: Tests launch (except IOS Virtual) XUnit Virtual: Tests don't launch NUnit Real Devices: Tests don't launch XUnit Real Devices: Tests don't launch

On Appium 5.1.0 and Selenium 4.23.0:

NUnit Virtual: Tests launch XUnit Virtual: Tests launch NUnit Real Devices: Tests launch (only Android works) XUnit Real Devices: Tests launch (only Android works)

Aug 11 '24 00:08 agray

Could you attache the full Appium server log?

No webview has been detected after 13926ms thinks me that it was client independent behavior. It would be helpful to attach working Appium server log as well.

https://app.saucelabs.com/tests/5988caa82b3d4cbb858bf745f5ca7d24#0

Aug 11 '24 00:08 agray

It last worked on August 1st: https://app.saucelabs.com/builds/rdc/ccadaa7715f33b228031f806c6f56087

IPhone 15 Pro Max (IOS 17.4.1) https://app.saucelabs.com/tests/13c99c1b6c674651af2f952ec2113e2d

I'll even provide the video from SauceLabs: https://github.com/user-attachments/assets/bae2ea75-58b7-4333-b885-a558926f034d

Aug 11 '24 00:08 agray

So what was the last version that worked for you properly?

Aug 11 '24 01:08 Dor-bl

So what was the last version that worked for you properly?

I think it was pre 5.0.0 (or one of the 5 betas) and Selenium 4.21.0

Aug 11 '24 01:08 agray

Does other webViews works for you? I have no major changes in mind that can cause this.

Aug 11 '24 01:08 Dor-bl

Does other webViews works for you? I have no major changes in mind that can cause this.

Webviews?

Aug 11 '24 01:08 agray

No available web pages after 10 retries: Could not connect to a valid webapp. Make sure it is debuggable and has at least one active page.

This is appium log from the failed one, 5988ca.... It looks like the appium xcuitest failed to get webviews. I haven't checked logs well yet but it is client independent.

Aug 11 '24 03:08 KazuCocoa

It was working on 1st August and I have only updated nuget packages. This code base has worked for many many years.

Aug 11 '24 04:08 agray

Does other webViews works for you? I have no major changes in mind that can cause this.

Webviews?

https://developer.apple.com/design/human-interface-guidelines/web-views

Aug 11 '24 05:08 Dor-bl

worked one:

{"level":"VERBOSE","message":"[Appium] Requiring driver at /root/appium/appium2-202408010003/node_modules/appium-xcuitest-driver/build/index.js","timestamp":"2024-08-01 10:18:19.926"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Appium v2.11.2 creating new XCUITestDriver (v7.24.6) session","timestamp":"2024-08-01 10:18:19.927"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Checking BaseDriver versions for Appium and XCUITestDriver","timestamp":"2024-08-01 10:18:19.928"}
{"level":"VERBOSE","message":"[AppiumDriver@c1ff] Appium's BaseDriver version is 9.11.2","timestamp":"2024-08-01 10:18:19.928"}
...
ebugger] Notified that an application has been updated","timestamp":"2024-08-01 10:18:22.677"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Current applications available:","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]     Application: \"PID:504\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         id: \"PID:504\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isProxy: false","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         name: \"Safari\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         bundleId: \"com.apple.mobilesafari\"","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isActive: true","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isAutomationEnabled: true","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         pageArray:","timestamp":"2024-08-01 10:18:22.678"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]           - id: 1","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             title: \"\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             url: \"about:blank\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]             isKey: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]     Application: \"PID:209\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         id: \"PID:209\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isProxy: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         name: \"amsengagementd\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         bundleId: \"com.apple.amsengagementd\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isActive: false","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger]         isAutomationEnabled: \"Unknown\"","timestamp":"2024-08-01 10:18:22.679"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Finally selecting app PID:504","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Selected app after 10ms","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][XCUITestDriver@4505] Picking webview 'WEBVIEW_504.1' after 292ms","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][XCUITestDriver@4505] Attempting to set context to 'WEBVIEW_504.1' from 'NATIVE_APP'","timestamp":"2024-08-01 10:18:22.680"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Selecting page '1' on app 'PID:504' and forwarding socket setup","timestamp":"2024-08-01 10:18:22.681"}
{"level":"VERBOSE","message":"[368657c8][RemoteDebugger] Sending '_rpc_forwardIndicateWebView:' message to app 'PID:504', page '1' (id: 4): 'indic

issue's one:

{"level":"VERBOSE","message":"[AppiumDriver@45ad] Appium v2.11.3 creating new XCUITestDriver (v7.24.14) session","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] Checking BaseDriver versions for Appium and XCUITestDriver","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] Appium's BaseDriver version is 9.11.3","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[AppiumDriver@45ad] XCUITestDriver's BaseDriver version is 9.11.3","timestamp":"2024-08-11 00:51:07.668"}
{"level":"VERBOSE","message":"[XCUITestDriver@52b5] ","timestamp":"2024-08-11 00:51:07.671"}
...
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Received response from send (id: 2): '[\"PID:606\",{}]'","timestamp":"2024-08-11 00:51:14.406"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Sending to Web Inspector took 5ms","timestamp":"2024-08-11 00:51:14.407"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Skipping app 'PID:606'. Original error: Empty page dictionary received","timestamp":"2024-08-11 00:51:14.414"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.415"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.533"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.535"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Notified that an application has been updated","timestamp":"2024-08-11 00:51:14.537"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger] Current applications available:","timestamp":"2024-08-11 00:51:14.915"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]     Application: \"PID:606\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         id: \"PID:606\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isProxy: false","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         name: \"Safari\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         bundleId: \"com.apple.mobilesafari\"","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         hostId: undefined","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isActive: true","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         isAutomationEnabled: true","timestamp":"2024-08-11 00:51:14.916"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]         pageArray: undefined","timestamp":"2024-08-11 00:51:14.917"}
{"level":"VERBOSE","message":"[996e6eec][RemoteDebugger]     Application: \"PID:398\"","timestamp":"2024-08-11 00:51:14.917"}

Your automation used different version of xcuitest driver. I wondered if recent refactoring in remote debugger affected pageArray selection

cc @mykola-mokhnach (just to share the remote debugger's thing)

Or... https://github.com/appium/appium-xcuitest-driver/blob/master/CHANGELOG.md#7249-2024-08-01 affected the behavior? My recent local test with browserName safari worked though. Need to debug

Aug 11 '24 06:08 KazuCocoa

Could you test again with XCUITest driver v7.24.6 once? @agray

Aug 11 '24 06:08 KazuCocoa

I have never specified the version of XCUITest

See SauceLabs platform configurator documentation https://app.saucelabs.com/platform-configurator

SUT is page hosted by SauceLabs: http://saucelabs.com/test/guinea-pig

Aug 11 '24 06:08 agray

@agray Is the issue persistent, e. g. does it happen always or rather randomly? Are you able to reproduce the issue locally?

Could you please provide an example of a failed log with appium:safariLogAllCommunication capability set to true?

Aug 11 '24 07:08 mykola-mokhnach

Also please review logs attached above - they contain your user name and the access token. I don't think you want these values being shared

Aug 11 '24 07:08 mykola-mokhnach

so please review logs attached above - they contain your user name and the access token. I don't think you want these values being shared

deleted

Aug 11 '24 07:08 agray

@agray Is the issue persistent, e. g. does it happen always or rather randomly? Are you able to reproduce the issue locally?

Could you please provide an example of a failed log with appium:safariLogAllCommunication capability set to true?

Yes its persistent and I get the same exception locally while trying to initialise the driver on line 93 of SauceryBase.cs trying to instantiate a new IOSDriver. This is an integration test so it has to talk to SauceLabs.

Aug 11 '24 07:08 agray

Does it also happen on a single device or multiple devices? Have you tried to reboot the device if it is always the same and run the test again?

Eventually, there is also a possibility on Sauce to explicitly provide the Appium snapshot version to run on (https://docs.saucelabs.com/mobile-apps/automated-testing/appium/appium-versions/). You may try to run with an older version to verify the assumption whether the issue has to do with Appium-specific changes

Aug 11 '24 07:08 mykola-mokhnach

Does it also happen on a single device or multiple devices? Have you tried to reboot the device if it is always the same and run the test again?

The "device" is the one requested via DesiredOptions on SauceLabs. SauceLabs spins up a new instance for EVERY test.

Aug 11 '24 07:08 agray

The "device" is the one requested via DesiredOptions on SauceLabs. SauceLabs spins up a new instance for EVERY test.

Sorry, my question was a bit different. Do you always request the same device name from your test or every time the test is executed it runs on a different device from the cloud?

Aug 11 '24 07:08 mykola-mokhnach

Sorry, my question was a bit different. Do you always request the same device name from your test or every time the test is executed it runs on a different device from the cloud?

Same type of device, yes. How the selenium grid (which is what SauceLabs is) provides that device it up to the grid and part of the grid "service".

Aug 11 '24 07:08 agray

https://github.com/Sauceforge/Saucery/commit/54da2c8d95ef1f5d6622e52c2aff68164cf0f79f

https://github.com/Sauceforge/Saucery/actions/runs/10338099541

appiumRequests.json appium.log device.log

Aug 11 '24 07:08 agray

Lets sum up what needs to be done step by step:

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)
Run the same test with an older appium snapshot via appiumVersion sauce option
Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session)
Run the same test on a different device

Aug 11 '24 08:08 mykola-mokhnach

Hm, I don't have 17.4.1 but have 17.5.1 iPhone. It returned an empty safari view as below with browserName: safari, so I was not able to see pageArray: undefined case as attached in this ticket. I changed web inspector config as well though. The base xcuitest was 7.24.14. https://gist.github.com/KazuCocoa/4e2b14918e57cc1e3df932cc1cf789b3

Possibly the device has a specific condtion

Aug 13 '24 02:08 KazuCocoa

Any updates on this?

Aug 18 '24 23:08 agray

Any updates on this?

Did you performed the below steps?

Run the same test on the same device with safariLogAllCommunication enabled. Attach the resulting log here (do not forget to delete a private data from there before sharing it)
Run the same test with an older appium snapshot via appiumVersion sauce option
Run the same test on the same device, but reboot it before doing that (I believe this could be done via a Live Session) Run the same test on a different device

Aug 19 '24 04:08 Dor-bl