mozilla
Background updates for ar locale fail
We ran a background update test from 55.0b4 to 67.0b19 and we got a fail: FindError: Background update hamburger menu icon notification did not appear, aborting.
Here you can find the logfile for it and a screenshot from iris: https://drive.google.com/drive/u/1/folders/1PsQiOJmREanXQ8cn-drgmdT0yqexWYqx (note that you have to use the mozilla account to see the log and screenshot).
I'll investigate why the hamgurger menu is not found with Arabic Firefox.
Ok, it's not the hamburger menu. It's the upward pointing arrow in the notification that isn't matching. From the screenshot, the arrow looks more like icon we have captured for OSX.
Bogdan, Is this only failing on Windows 10 in AR? Or are other locales and OS's failing?
In any case, we must update the image captures for background_update_menu_notification.png in each OS directory in the update tests. The new captures should be small enough as to only pick up the arrow and no other stuff that might surround the icon. If this has been changed to a common resource in Fx, let's maintain just one image in the common directory of the update tests.
Ioana, this will likely fail in Iris2. The test cases have already been ported to Iris 2. Can you checkout the patterns as I discussed above and update them in Iris2, then port the new images back to Iris so this test can be run successfully in Iris by QA.
Also, while your in there. I just made a one line change to each of the update tests in Iris1. It's a pattern similarity change in both manual_update.py and background_update.py. In both tests, as part of your work here with the images, please update the pattern declaration to
update_restart_pattern = Pattern('background_update_menu_notification.png').similar(0.5)
From what we tested until now only "ar" locale failed on multiple platforms (just checked on Ubuntu 16 as well, don't have a mac available with iris at this time).
@tracywalker I've investigated this issue in iris 2 on MAC and the problem here is not the update_restart_pattern that does not match.
I think that the update tests will work in 3 days. From this link https://whattrainisitnow.com/ we can see that the latest Beta is 67.0 (which is not a normal beta version it is like a release version) and not 67.0b19.
Running the test with debug we can see that the 67.0b19 version is downloaded.
Serving HTTP on 127.0.0.1 port 2000. 2019-05-17 12:06:39 [DEBUG] Starting new HTTPS connection (1): archive.mozilla.org:443 2019-05-17 12:06:39 [DEBUG] https://archive.mozilla.org:443 "GET /pub/firefox/releases/ HTTP/1.1" 200 None 2019-05-17 12:06:39 [INFO] Retrieving list of candidate builds from https://archive.mozilla.org/pub/firefox/candidates/67.0b19-candidates/ 2019-05-17 12:06:39 [DEBUG] https://archive.mozilla.org:443 "GET /pub/firefox/candidates/67.0b19-candidates/ HTTP/1.1" 200 638 2019-05-17 12:06:39 [INFO] Found 1 build: build1 2019-05-17 12:06:39 [INFO] Selected build: build1 2019-05-17 12:06:39 [DEBUG] attempt 1/1 2019-05-17 12:06:39 [DEBUG] retry: calling _get_binary, attempt #1 2019-05-17 12:06:39 [DEBUG] https://archive.mozilla.org:443 "GET /pub/firefox/candidates/67.0b19-candidates/build1/mac/ar/ HTTP/1.1" 200 765 2019-05-17 12:06:39 [INFO] File has already been downloaded: /Users/ioanacrisan/.iris/cache/firefox-67.0b19-build1.ar.mac.dmg
So the test downloads the 67.0b19 version but after restart the version displayed is 67.0. This can be seen both by running the test manually and automatically.
Our automated tests get the latest version from path(in our case 67.0b19) but until 2019-05-20 the latest version is 67.0 and not 67.0b19 as we expect so the test fails. Below is the stack trace
assert update_step in FirefoxUtils.get_firefox_version(firefox.application.path),
'Incorrect Firefox update.'
E AssertionError: Incorrect Firefox update.
E assert '67.0b19' in '67.0'
E + where '67.0' = <function FirefoxUtils.get_firefox_version at 0x127155d08>('/private/var/folders/qd/b48hz7gs3cs2gmckmlwc_nq80000gn/T/iris2_6ynfryni/firefoxlatest_betaar/Firefox.app/Contents/MacOS/firefox')
E + where <function FirefoxUtils.get_firefox_version at 0x127155d08> = FirefoxUtils.get_firefox_version
E + and '/private/var/folders/qd/b48hz7gs3cs2gmckmlwc_nq80000gn/T/iris2_6ynfryni/firefoxlatest_betaar/Firefox.app/Contents/MacOS/firefox' = <targets.firefox.firefox_app.fx_browser.FirefoxApp object at 0x1271efe10>.path
E + where <targets.firefox.firefox_app.fx_browser.FirefoxApp object at 0x1271efe10> = <targets.firefox.firefox_app.fx_browser.FXRunner object at 0x12733dd68>.application
After 2019-05-20 the version will probably change to 68.0b... and the test will probably work.
After 2019-05-20 the version will probably change to 68.0b... and the test will probably work.
That may be what you're running into at this moment. If you look at the screen shots in the original report here, You'll see that it's the up arrow of background_update_menu_notification.png that is not found.
I just run the background_update.py test case and it is still FAILing. The arrow up pattern does not appear at all. As a reminder please don't use Firefox versions smaller that 57 when running the update tests.
I just run both the background_update.py and the manual_update.py test cases on my MAC in iris 2 and the tests PASSed. Below are the results:
Executing 1: - [update/manual_update.py]: This is a test for Firefox manual update. Current version: 57.0, updating to version: 68.0b5.
------------------------------------------------------------------------------------------------Test Report------------------------------------------------------------------------------------------------- Application: Firefox, Platform: osx, fx_version: 57.0, fx_build_id: 20171016185129, channel: beta Passed: 1, Failed: 0, Skipped: 0, Errors 0 -- Total: 1
Executing 1: - [update/background_update.py]: This is a test for Firefox background update. Current version: 57.0, updating to version: 68.0b5.
------------------------------------------------------------------------------------------------Test Report------------------------------------------------------------------------------------------------- Application: Firefox, Platform: osx, fx_version: 57.0, fx_build_id: 20171016185129, channel: beta
Passed: 1, Failed: 0, Skipped: 0, Errors 0 -- Total: 1
Just finished to run these tests on the other OSes(Win10, Win7 and LINUX) and the tests are working in iris 2.
I ran again the background update: 68.0b3_ar and en-US (Using iris 1.0) and it failed with the following errors:
Here are the images for both locales:
What happens is that Iris opens Fx the first time and it automatically closes it in about 1 second (it does not wait for the update to get triggered). Then Fx is opened the second time and About Firefox window is opened (which actually triggers the update). At this moment Iris looks for "Firefox is up to date" string which will not be there because Fx just completed the download and it needs a restart to apply them