elixir-wallaby
Wallaby tests started to fail suddenly both on CI and locally
Elixir and Erlang/OTP versions
Erlang/OTP 27 [erts-15.1] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]
Elixir 1.17.3 (compiled with Erlang/OTP 25)
Operating system
Mac & Linux box from CI
Browser
Chrome v133
Driver
ChromeDriver v133
Correct Configuration
- [x] I confirm that I have Wallaby configured correctly.
Current behavior
Hey folks!
Last month, on Feb. 21, our wallaby tests suddenly started intermittently failing both locally for all of our team members and on CI with Stale Reference and various race-condition errors. Example:
1) test *
test/*
** (Wallaby.StaleReferenceError) The element you are trying to reference is stale or no longer attached to the
DOM. The most likely reason is that it has been removed with JavaScript.
You can typically solve this problem by using find to block until the DOM is in a
stable state.
code: |> Browser.text()
stacktrace:
(wallaby 0.30.10) lib/wallaby/element.ex:267: Wallaby.Element.raise_error/1
Approximately 60 out of 180 tests were failing each run. We get all sorts of different errors that look like race conditions, like HTML not being updated when it should.
On many occasions, adding something like Process.sleep(1000) fixes the tests. I also noticed that some tests might benefit from waiting until the page is properly rendered. But they were working flawlessly for 4+ years, so it didn't make much sense to update them without a solid reason.
After some digging, we found that locking Chrome and ChromeDriver versions to <=v127 resolves the issues completely. The thing is, nobody on the team upgraded Chrome or ChromeDriver. I was using v133 locally (v134 was already available), and it suddenly started to fail a bunch of tests for everyone both locally and on CI.
So we wondered if it's something like a date-based hard deprecation on the Chrome side or a security patch applied to previous versions, but I've looked through every release note from Chrome for Developers 127 to 134 (here), but couldn't find anything that caught my attention (I'm definitely not a pro when it comes to browser tests 😅)
I only wanted to open this issue after exhausting my options, but here I am 😓 – Let me know if you need more info. Thanks for your time!
Expected behavior
Although wallaby is working fine with Chrome & ChromeDriver on v127, we want to understand what happened and be able to unlock the version in the future.
Test Code & HTML
Some simple tests that were always passing started to randomly break:
test "Required financial info notice" do
tenant = insert(:tenant, name: "Test Tenant")
user = insert(:user, tenant_id: tenant.id)
session =
tenant.subdomain
|> build_session()
|> log_in(email: user.email)
step "shows a notice" do
assert has_text?(session, "Financial Information Required")
end
end
def build_session(host, opts) do
custom_headers = %{"Host" => host}
metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(*.Repo, self())
{:ok, session} =
Wallaby.start_session(
custom_headers: custom_headers,
metadata: metadata,
readiness_timeout: 60_000,
startup_timeout: 60_000,
window_size:
if System.get_env("MOBILE") do
[width: 500, height: 1000]
else
[width: 1440, height: 1000]
end
)
session = Browser.visit(session, "http://#{host}")
if opts[:with_extras?] == true do
{session, metadata, custom_headers}
else
session
end
end
Error:
3) test Required financial info notice (*.*Test)
test/*
** (Wallaby.StaleReferenceError) The element you are trying to reference is stale or no longer attached to the
DOM. The most likely reason is that it has been removed with JavaScript.
You can typically solve this problem by using `find` to block until the DOM is in a
stable state.
code: assert has_text?(session, "Financial Information Required")
stacktrace:
(wallaby 0.30.10) lib/wallaby/element.ex:267: Wallaby.Element.raise_error/1
(wallaby 0.30.10) lib/wallaby/browser.ex:1095: anonymous fn/2 in Wallaby.Browser.has_text?/2
(wallaby 0.30.10) lib/wallaby/browser.ex:148: Wallaby.Browser.retry/2
(wallaby 0.30.10) lib/wallaby/browser.ex:1094: Wallaby.Browser.has_text?/2
*:23: (test)
Demonstration Project
No response
Could you try and clone the wallaby repo and run the test suite using the suspected Chrome and Chromdriver versions?
I ran them locally with 134 and they all passed.
Should be able to clone, install deps, then run WALLABY_DRIVER=chrome mix test
........................
Finished in 114.5 seconds (89.1s async, 25.4s sync)
4 features, 255 tests, 0 failures
No failures at all. Same version that fails the tests for our app.
Very interesting
This is going to be hard to narrow down without a minimal reproduction case that i can run, if you could somehow create one, that would be awesome.
yeah, looks like it's going to be tough to understand what's happening without a good example indeed.
I'll see with the team here if we can cook something or other alternatives, since I don't think it's going to be trivial to reproduce it, and I'll update the thread here too.
Thanks, Mitch!
@mhanberg hey!
we'll have some time to try to replicate it in a demo app in roughly 30 days (2 cycle/sprints), so this will be sitting for a while.
Feel free to close it or leave it open in case anyone else experiences this 🙂
I can confirm I'm seeing the same. My CI uses latest chromedriver and that latest is failing locally for me as well, whereas I was green before the update.
Looks like this is related
https://github.com/teamcapybara/capybara/issues/2800
Which leads me to this
https://issues.chromium.org/issues/402796660
A workaround here could be to set your CI (and local env) to use a custom version of Chrome/Chromedriver, rather than latest. This seems to get the tests into the regular flaky state for me :)
- id: chrome
uses: browser-actions/setup-chrome@v1
with:
chrome-version: 134
install-chromedriver: true
- name: Override default chrome
run: sudo ln -sf ${{ steps.chrome.outputs.chrome-path }} /usr/bin/google-chrome