WinAppDriver icon indicating copy to clipboard operation
WinAppDriver copied to clipboard
verified publisher icon microsoft

Element not found when running test in GitHub actions but found when running test locally

Open JanTrnavskyNovanta opened this issue 3 years ago • 1 comments

Hi, I have an issue with the different behavior of WinAppDriver when running tests on my local machine (Windows 11 Pro 22000.856) and in GitHub Actions using Windows server 2022 runner. Local WinAppDriver version: 1.2.2009.02003 GitHubActions WinAppDriver version: 1.2.2009.02003 (based on https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md)

Test code: https://github.com/JanTrnavskyNovanta/win-app-driver-issue-repro/blob/a4a2a7c7fea1adb95385bcc901b7423d47457f6a/tests/test_bug_demo.py

How to replicate this issue:

Code: JanTrnavskyNovanta/win-app-driver-issue-repro#1 It is just a simple app using PySide2. To run it locally (I would recommend using a virtual environment)

  1. install dependencies from requirements.txt file pip install -r requirements.txt
  2. build the app using PyInstaller (if you are in the main directory go to a subdirectory with the app under test)
cd tested_app
pyinstaller __main__.spec -y

  1. run WinAppDriver

  2. go to tests folder, run test via pytest pytest

Locally it works well for me and reports 16 elements (and it clicks on button and test passes): https://github.com/JanTrnavskyNovanta/win-app-driver-issue-repro/blob/a4a2a7c7fea1adb95385bcc901b7423d47457f6a/tests/elements_report_computer.txt

But when running test in GH actions it finds only 7 elements (just window control ones) and fails to click the button

selenium.common.exceptions.NoSuchElementException: Message: An element could not be located on the page using the given search parameters.

Things I tried to investigate this issue

I tried to take a screenshot of the app under test and the buttons are there. (in fact, the screenshots are almost identical, there are just small differences in how the button and checkbox are rendered)

Screenshot of app under test and log of elements and handles in txt from server is stored as build artifact here

I tried to check if I do not use wrong handle (but there is only one), force full HD resolution, change Windows Server version to 2019. Nothing helped.

Any other idea what I should try or what could cause the issue?

Thanks, Jan

JanTrnavskyNovanta avatar Sep 13 '22 11:09 JanTrnavskyNovanta

Does the server have an actual physical display?

pdisser8 avatar Sep 21 '22 21:09 pdisser8

Hi @pdisser8, the display is:

DeviceID : DesktopMonitor1 DisplayType : MonitorManufacturer : (Standard monitor types) Name : Generic PnP Monitor ScreenHeight : 1080 ScreenWidth : 1

response of Run Get-WmiObject -Class Win32_DesktopMonitor or should I use some other PowerShell command to get it? :}

JanTrnavskyNovanta avatar Sep 26 '22 12:09 JanTrnavskyNovanta

ScreenWidth : 1

If that is true, then that's very likely the reason.

ChristoWolf avatar Oct 22 '22 09:10 ChristoWolf

Oh my bad, I did not copy whole response, the width is 1920 ;)

DeviceID : DesktopMonitor1 DisplayType : MonitorManufacturer : (Standard monitor types) Name : Generic PnP Monitor ScreenHeight : 1080 ScreenWidth : 1920

you can check it here https://github.com/JanTrnavskyNovanta/win-app-driver-issue-repro/actions/runs/3127796916/jobs/5074837002

JanTrnavskyNovanta avatar Oct 24 '22 05:10 JanTrnavskyNovanta

Ah ok yeah that sounds better :)

Have you tried taking a screenshot and dumping the page source when the issue occurs?

ChristoWolf avatar Oct 24 '22 06:10 ChristoWolf

I tried to take screenshot and the screenshot is fine. When I compared it with the screenshot taken locally it is almost the same. Except some anti-aliasing and slight differences in rounded corners (Win 11 vs Win server).

Well I am not sure how to dump the page source, the only thing I tried was after taking screenshot I found all elements by xpath //* - this matches all elements and locally works well. If you have some other idea how to dump all elements I will try it.

and I write to text file All window handles (to make sure I do not work with different window) and for all elements I write down - element Name and automation IDs

Screenshot and File with elements created on GH actions server:

Screenshot app_screenshot

text file

Window handles: ['0x00010202']
Amount of elements found: 7
Element name: MainWindow |  automation id: None
Element name: None |  automation id: None
Element name: System |  automation id: MenuBar
Element name: System |  automation id: None
Element name: Minimize |  automation id: None
Element name: Maximize |  automation id: None
Element name: Close |  automation id: None

Local run:

Screenshot: app_screenshot

text file

Window handles: ['0x00FE1162']
Amount of elements found: 16
Element name: MainWindow |  automation id: MainWindow
Element name: None |  automation id: None
Element name: System |  automation id: MenuBar
Element name: System |  automation id: None
Element name: Minimize |  automation id: None
Element name: Maximize |  automation id: None
Element name: Close |  automation id: None
Element name: None |  automation id: MainWindow.centralwidget
Element name: TestButton |  automation id: MainWindow.centralwidget.pushButton
Element name: TestRadioButton |  automation id: MainWindow.centralwidget.radioButton
Element name: TestCheckBox |  automation id: MainWindow.centralwidget.checkBox
Element name: None |  automation id: MainWindow.menubar
Element name: File |  automation id: None
Element name: Another Option |  automation id: None
Element name: None |  automation id: MainWindow.statusbar
Element name: None |  automation id: None

I hope this answers your question 🙂

Thank you

JanTrnavskyNovanta avatar Oct 24 '22 06:10 JanTrnavskyNovanta

You get the page source via self.driver.page_source. It shows the whole available UI automation tree.

Other than that, your stuff seems fine, so I'm also a bit puzzled. I have another suspicion though, but let's first check the page source.

ChristoWolf avatar Oct 24 '22 07:10 ChristoWolf

The page source from GH actions run:

<?xml version="1.0" encoding="utf-16"?><Window AcceleratorKey="" AccessKey="" AutomationId="" ClassName="Qt5152QWindowIcon" FrameworkId="Win32" HasKeyboardFocus="True" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="window" Name="MainWindow" Orientation="None" ProcessId="6460" RuntimeId="42.262624" x="0" y="0" width="802" height="632" CanMaximize="True" CanMinimize="True" IsModal="False" WindowVisualState="Normal" WindowInteractionState="ReadyForUserInteraction" IsTopmost="False" CanRotate="False" CanResize="True" CanMove="True" IsAvailable="True"><TitleBar AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="title bar" Orientation="None" ProcessId="6460" RuntimeId="42.262624.3.-2147483647.262624.-2.0" x="17" y="3" width="784" height="28"><MenuBar AcceleratorKey="" AccessKey="Alt+Space" AutomationId="MenuBar" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="menu bar" Name="System" Orientation="None" ProcessId="6460" RuntimeId="42.262624.3.-2147483646.6460.262489.0" x="1" y="8" width="22" height="22"><MenuItem AcceleratorKey="" AccessKey="Space" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="menu item" Name="System" Orientation="None" ProcessId="6460" RuntimeId="42.262624.3.-2147483646.6460.262489.1" x="1" y="8" width="22" height="22" ExpandCollapseState="Collapsed" IsAvailable="True" /></MenuBar><Button AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="button" Name="Minimize" Orientation="None" ProcessId="6460" RuntimeId="42.262624.3.-2147483647.262624.-2.2" x="662" y="1" width="47" height="30" /><Button AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="button" Name="Maximize" Orientation="None" ProcessId="6460" RuntimeId="42.262624.3.-2147483647.262624.-2.3" x="709" y="1" width="46" height="30" /><Button AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="button" Name="Close" Orientation="None" ProcessId="6460" RuntimeId="42.262624.3.-2147483647.262624.-2.5" x="755" y="1" width="47" height="30" /></TitleBar></Window>

and source from local run:

<?xml version="1.0" encoding="utf-16"?><Window AcceleratorKey="" AccessKey="" AutomationId="MainWindow" ClassName="MainWindow" FrameworkId="Qt" HasKeyboardFocus="True" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="window" Name="MainWindow" Orientation="None" ProcessId="1428" RuntimeId="42.2889010" x="0" y="0" width="802" height="632" CanMaximize="True" CanMinimize="True" IsModal="False" WindowVisualState="Normal" WindowInteractionState="Running" IsTopmost="False" CanRotate="False" CanResize="True" CanMove="True" IsAvailable="True"><TitleBar AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="title bar" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.3.-2147483647.2889010.-2.0" x="17" y="3" width="784" height="28"><MenuBar AcceleratorKey="" AccessKey="Alt+Space" AutomationId="MenuBar" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="menu bar" Name="System" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.3.-2147483646.1428.2109344463.0" x="1" y="8" width="22" height="22"><MenuItem AcceleratorKey="" AccessKey="Space" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="menu item" Name="System" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.3.-2147483646.1428.2109344463.1" x="1" y="8" width="22" height="22" ExpandCollapseState="Collapsed" IsAvailable="True" /></MenuBar><Button AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="button" Name="Minimize" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.3.-2147483647.2889010.-2.2" x="662" y="1" width="47" height="30" /><Button AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="button" Name="Maximize" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.3.-2147483647.2889010.-2.3" x="709" y="1" width="46" height="30" /><Button AcceleratorKey="" AutomationId="" ClassName="" FrameworkId="" HasKeyboardFocus="False" IsContentElement="False" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="button" Name="Close" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.3.-2147483647.2889010.-2.5" x="755" y="1" width="47" height="30" /></TitleBar><Group AcceleratorKey="" AccessKey="" AutomationId="MainWindow.centralwidget" ClassName="QWidget" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="group" Name="" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483645" x="1" y="52" width="800" height="559"><Button AcceleratorKey="" AccessKey="" AutomationId="MainWindow.centralwidget.pushButton" ClassName="QPushButton" FrameworkId="Qt" HasKeyboardFocus="True" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="button" Name="TestButton" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483648" x="31" y="112" width="80" height="21" /><RadioButton AcceleratorKey="" AccessKey="" AutomationId="MainWindow.centralwidget.radioButton" ClassName="QRadioButton" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="radio button" Name="TestRadioButton" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483647" x="31" y="153" width="81" height="17" IsSelected="False" IsAvailable="True" /><CheckBox AcceleratorKey="" AccessKey="" AutomationId="MainWindow.centralwidget.checkBox" ClassName="QCheckBox" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="True" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="check box" Name="TestCheckBox" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483646" x="31" y="83" width="68" height="17" ToggleState="Off" IsAvailable="True" /></Group><MenuBar AcceleratorKey="" AccessKey="" AutomationId="MainWindow.menubar" ClassName="QMenuBar" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="menu bar" Name="" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483644" x="1" y="31" width="800" height="21"><MenuItem AcceleratorKey="" AccessKey="" AutomationId="" ClassName="QAction" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="menu item" Name="File" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483639" x="1" y="31" width="34" height="20" ExpandCollapseState="Collapsed" IsAvailable="True" /><MenuItem AcceleratorKey="" AccessKey="" AutomationId="" ClassName="QAction" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="menu item" Name="Another Option" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483638" x="35" y="31" width="99" height="20" ExpandCollapseState="Collapsed" IsAvailable="True" /></MenuBar><StatusBar AcceleratorKey="" AccessKey="" AutomationId="MainWindow.statusbar" ClassName="QStatusBar" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="status bar" Name="" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483642" x="1" y="611" width="800" height="20"><Thumb AcceleratorKey="" AccessKey="" AutomationId="" ClassName="QSizeGrip" FrameworkId="Qt" HasKeyboardFocus="False" HelpText="" IsContentElement="True" IsControlElement="True" IsEnabled="True" IsKeyboardFocusable="False" IsOffscreen="False" IsPassword="False" IsRequiredForForm="False" ItemStatus="" ItemType="" LocalizedControlType="thumb" Name="" Orientation="None" ProcessId="1428" RuntimeId="42.2889010.4.-2147483643" x="788" y="618" width="13" height="13" /></StatusBar></Window>

The one running in GH actions is really missing elements 😕 but I have no idea why.

JanTrnavskyNovanta avatar Oct 24 '22 07:10 JanTrnavskyNovanta

Unfortunately, I can't use my machine currently, so I can't easily diff the XMLs. But my suspicion is that, for Windows Server, PySide might render the automation tree differently.

I don't have any experience with PySide, but usually elements are not exposed to the relevant automation tree if their IsControlElement is false. However, I don't have an idea how to easily check those without doing it via inspect.exe. Microsoft's UIAutomation.dll might be useful for this.

The culprit might be something very obvious, but I cannot see it currently.

ChristoWolf avatar Oct 24 '22 09:10 ChristoWolf

Hi @ChristoWolf , thanks for all your help. Today I observed the same issue on the laptop of a newcomer that joined our project. The laptop was running Windows 10. So we investigated it.

inspect.exe sometimes reports 7 and sometimes 16 elements. and it is dependent on the order in which we start the apps 😞

if we first open the tested app and inspect.exe after - we can access all 16 elements if we open inspect.exe before the tested app - it does report only 7 elements from the title bar.

For now, we implemented a workaround: it simply opens the app via the python library We do use winappdriver to find NativeWindowHandle And now we finally attach it to the running app. It is not as elegant as opening the app via WinAppDriver but it works

https://github.com/JanTrnavskyNovanta/win-app-driver-issue-repro/pull/1/commits/8c871cc6fd0a26089630cc51bfc4020f3c71b81e

passing GH action https://github.com/JanTrnavskyNovanta/win-app-driver-issue-repro/actions/runs/3469923701/jobs/5797497471

The reason why the order matters is a total mystery to me. But maybe this could help someone else with a similar issue.

Thanks Have a great day

JanTrnavskyNovanta avatar Nov 15 '22 11:11 JanTrnavskyNovanta

Yeah that sounds really weird.

if we open inspect.exe before the tested app - it does report only 7 elements from the title bar.

You then of course also clicked the Refresh button in inspect.exe, right?

ChristoWolf avatar Nov 15 '22 11:11 ChristoWolf

Yes, I clicked refresh button 🙂 without clicking refresh there isn't even the title bar visible 😉 I will even record video later to show this mystery. But for now important thing is it finally works for us.

JanTrnavskyNovanta avatar Nov 15 '22 12:11 JanTrnavskyNovanta

Flow you can see in the video: Tested app is running, we start inspect.exe - it shows all elements. We close the app, click the refresh button in inspect.exe - the window "MainWindow" disappears from inspect.exe We start tested app again and click the refresh button in inspect.exe - only title bar elements are shown there We close and open the inspect.exe again - all elements are present there

https://user-images.githubusercontent.com/99248083/201934432-c0724ea2-4cbd-417d-8c71-673a4d9433bb.mp4

I do not think it is a correct behavior (I might be missing something and it can be desired one), but it explains even why attaching to running app works well but starting it using winappdriver causes issues.

@ChristoWolf still big thanks to you, the page source command is really useful one I was not aware of.

JanTrnavskyNovanta avatar Nov 15 '22 13:11 JanTrnavskyNovanta