Browser provider fails to execute tests

[tooreht]

node: v10.7.0 testcafe: 0.23.1 testcafe-browser-provider-browserstack: 1.5.0

code

testcafe.ts

import { Selector } from 'testcafe'; // first import testcafe selectors

fixture `Getting Started`// declare the fixture
    .page `https://devexpress.github.io/testcafe/example`;  // specify the start page


//then create a test and place your code there
test('My first test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')
        .click('#submit-button')

        // Use the assertion to check if the actual header text is equal to the expected one
        .expect(Selector('#article-header').innerText).eql('Thank you, John Smith!');
});

cli

$ env BROWSERSTACK_USERNAME=xyz BROWSERSTACK_ACCESS_KEY='xyz' testcafe "browserstack:[email protected]:Windows" testcafe.ts
 Running tests in:
 - Chrome 53.0.2785 / Windows 10.0.0
 (https://automate.browserstack.com/builds/30c7750a2605f9ef5aa501a5df15d7650ab040ee/sessions/531194f8c80870e005d7d6a4a6ae64a56255ce8a)

 Getting Started
ERROR The Chrome 53.0.2785 / Windows 10.0.0 (https://automate.browserstack.com/builds/30c7750a2605f9ef5aa501a5df15d7650ab040ee/sessions/868726f36f8460ab5b6feec0288082f3cd2413be) browser disconnected. This problem may appear when a browser hangs or is closed, or due to network issues.

Browser provider fails to execute tests on browserstack. It loads the fixture page and then hangs for two minutes and finishes the test run with status "completed". Then it opens another test run with the same behavior. After that it executes a third test run which finally finishes with status "failed" with the following error message on browserstack:

(Marked via REST API : The Chrome 53.0.2785 / Windows 10.0.0 (https://automate.browserstack.com/builds/30c7750a2605f9ef5aa501a5df15d7650ab040ee/sessions/946b7232a1637a0b74b05b6f092b2abbc0e60f05) browser disconnected. This problem may appear when a browser hangs or is closed, or) 

The same happens when using browserstack automate backend:

$ env BROWSERSTACK_USERNAME=xyz BROWSERSTACK_ACCESS_KEY='xyz' BROWSERSTACK_USE_AUTOMATE=1 testcafe "browserstack:[email protected]:Windows" testcafe.ts

Tests run flawlessly when running locally:

$ testcafe chromium,firefox testcafe.ts             Mon 12 Nov 2018 12:07:58 PM CET
 Running tests in:
 - Chrome 70.0.3538 / Linux 0.0.0
 - Firefox 63.0.0 / Linux 0.0.0

 Getting Started
 ✓ My first test


 1 passed (3s)

[davidnaas]

Experiencing the same issue. Fixture page loads but then nothing happens

[steffenmllr]

Same here. Local tests run great, remote tests do not run. I'm only testing publicly available websites, so I also tried using the BROWSERSTACK_NO_LOCAL Env - no luck it always hangs with a res://ieframe.dll/dnserror.htm

[Nyeng]

Is this still a problem for you guys in the latest version of testcafe and testcafe-browser-provider-browserstack?

I'm trying to figure out how to run tests for non publicly available websites, but not sure if I should keep trying if it doesn't work for publicly available apps..

[davidnaas]

I haven't gone forward with this and shelved the browserstack part in my project for now. But I don't think pages behind auth should be a problem. I've had no problem running normal testcafe against sites behind basic auth. It's something else at work here related to browserstack.

[AndreyBelym]

@Nyeng, I've encountered no provider-specific problems with HTTP authentication. I guess it can be caused by the TestCafe core. Do your tests work when you run them locally? I think it's better to submit a bug report to TestCafe and provide a sample page that can be used to reproduce the issue.

[Nyeng]

It's not related to authentication. The site is only available on our local network, but works fine on Browserstack out of the box due to this module's implementation of forcing local testing to true(?). It works fine running tests locally on Browserstack, but when running from Jenkins I see the test starts running on Browserstack with the exact same settings and everything, but it won't load the dome (only thing I see is the header on the browser's tab window which means it's able to fetch some of the html?

.

I've never seen anyone dealing with this exact problem. I can post it as a bug report, but not really sure of how to reproduce for a public website.

edit: Used Browserstack's debug tool. Found out this only happens using our Wiremock instance, it works for our local "normal" endpoints. Might be too farfetched for anyone else to try to reproduce, but it's doable.

[AndreyBelym]

@Nyeng thank you for the provided information. I will try to reproduce it, but since I have no experience with Wiremock, it will be very helpful if you provide an example.

[Aaronius]

@Nyeng, is this something you were able to fix in your own code? We're running into similar browser disconnection problems and not using WireMock. We're still trying to pin down what the cause is.

Here's an example test run that failed: https://github.com/adobe/alloy/runs/2143251498?check_suite_focus=true

Although this specific run failed when IE disconnected, other browsers also intermittently disconnect.

[Aaronius]

BTW, we seem to only run into this issue when running tests from a github workflow. If we run tests from our local machine, we don't have the issue. So...it may be a Github workflow problem, a combination of the github workflow and this provider, or something else.

[wentwrong]

@Aaronius, I ran tests in IE11 using the BrowserStack provider and Github Actions to reproduce the "browser disconnected" problem, but it seems that everything works fine in my case: https://github.com/wentwrong/test-bs-on-gh-bug/runs/2164057520/.

Can you please clarify how I need to modify my setup in order to reproduce the issue?

[Aaronius]

I have the same question. The problem is intermittent and I don't know what's causing the failure. It could be the number of tests we're running, what we're doing within tests, the time we're running the tests, the particular sequence of tests, etc. I have yet to pin down the answers. In other news, we decided to switch to SauceLabs and have been having better (but not perfect) success.

[wentwrong]

We would investigate it further. However, it seems that the problem occurs from time to time and only under certain conditions that are still unknown to us. If someone encounters this issue, please give us a reproducible example and the recorded BrowserStackLocal log file (see the logFile option).

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

[github-actions[bot]]

We're closing this issue after a prolonged period of inactivity. If it still affects you, please add a comment to this issue with up-to-date information. Thank you.