[test-runner-browserstack] Error testing on Safari 15.x.x and later

[Sarin-Udompanish]

Summary:

When attempting to run a simple test on Safari 15.x.x and later using @web/test-runner-browserstack, the tests fail to start and result in a timeout error.

Steps to Reproduce:

1. Create a test file with the following code inside a __test__ folder:

describe('Simple test', () => {
  it('Should pass', function () {});
});

2. Configure web-test-runner.config.mjs as follows:

import { browserstackLauncher } from '@web/test-runner-browserstack';

const sharedCapabilities = {
  'browserstack.user': 'xxxx',
  'browserstack.key': 'xxxx',
  build: `build mos`,
  project: 'my project',
  name: 'my test',
};
export default {
  browsers: [
    browserstackLauncher({
      capabilities: {
        ...sharedCapabilities,
        browserName: 'Safari',
        os: 'OS X',
        os_version: 'Monterey',
      },
    }),
  ],
};

3. Run the tests using @web/test-runner with the configuration provided.

Expected Results:

The tests should start successfully and pass on Safari 15.x.x and later.

Actual Results:

The tests fail to start and result in a timeout error with the following message: "Browser tests did not start after 20000ms. You can increase this timeout with the testsStartTimeout option. Check the browser logs or open the browser in debug mode for more information."

Additional Information:

• If the os_version is changed to Big Sur or an older version, the tests work correctly.
• Increasing the timeout does not resolve the issue; the tests remain stuck until the timeout is reached.
• The issue is specific to Safari 15.x.x and later versions.

[sfloam]

Our team is also experiencing the same issue. If we run a single "it" statement in a test file, it will pass. If we run multiple "it" statements, the session will timeout even if the tests all pass according to the UI.

Our team also noticed a long delay when this line of code was executed in SessionManager.js when using Safari. This is not present when Chrome executes. We are wondering if this massive delay may be the culprit or related to the issue.

 const returnValue = await this.driver.execute(`
            return (function() {
                return { testCoverage: window.__coverage__ };
            })();
            `)

Note, our tests will all pass with Safari 14 like @Sarin-Udompanish.

Can we please push this ticket up in the queue? It is preventing us from deploying our code on BrowserStack.

[Westbrook]

We'd be happy to accept a PR if you have a path towards a resolution here!

[koddsson]

It would also be helpful if anyone has any ideas on how to run tests with a specific safari version. I just have version 17 installed on my machine and have no idea how I could even start working on this since I don't have Safari 15.