cypress-coverage/coverage-final.json is empty object when running cypress-and-jest

[ghost]

I'm following Gleb Bahmutov's instructions for getting combined code coverage reports from cypress and jest (https://github.com/bahmutov/cypress-and-jest), and ran into this issue. Gleb asked me to post the issue here.

When I run "npm t" on my own project, my jest tests display an accurate coverage report, the cypress tests all properly pass, and .nyc-output/out.json file contains a lot of entries for my source files.

However, cypress-coverage/coverage-final.json only contains "{}".

Two things I'm doing a bit differently that I think might have an effect:

(1) I already had cypress running so I did not run "npx @bahmutov/cly init". However, I did check the contents of Gleb's cypress/(plugins|support)/index.js, jest.config.js and package.json to make sure I got everything from them. In package.json, I have these slightly-different versions:

"@cypress/code-coverage": "^3.2.1",
"babel-plugin-istanbul": "^6.0.0",
"check-code-coverage": "^1.0.1",
"cypress": "^4.4.0",
"jest": "^25.3.0",

(2) Under my main project folder, I divided the code into two subfolders, each with its own package.json, because I want the tests to be in a separate subproject than the source (to make it easier to distribute the source without the tests):

project ├── source │ ├── index.js │ ├── lib/ │ ├── package.json │ └── styles.css └── tests ├── babel.config.js ├── coverage/ ├── cypress/ ├── cypress-coverage/ ├── cypress.json ├── fdnCreate.html ├── fUpdate.html ├── jest.config.js ├── jest-coverage/ ├── node_modules/ ├── .nyc_output/ ├── package.json ├── package-lock.json ├── reports/ └── _ _ tests_ _/

I'll greatly appreciate any suggestions!

[bahmutov]

1. Can you run tests with DEBUG=code-coverage to see what is says when generating the report https://github.com/cypress-io/code-coverage#debugging?
2. Can you paste a few first lines from .nyc_output/out.json so we see the paths there?

[ghost]

debug-npm-t.txt is the result of running DEBUG=code-coverage npm t

Note that "OpplaudSaas" is the actual name of the project, "OpplaudDial" is the name of the source code subproject and "OpplaudDialTests" is the name of the tests subproject.

Here is the complete out.json: out.json.txt

Thank you, again, for taking the time to address this!

[bahmutov]

Can you please install code-coverage plugin v3.3.0 and run JUST the Cypress test with DEBUG=code-coverage npm run test:cy? I don't see any debug logs in the file you have sent (the debug logs are written into STDERR, maybe you are not redirecting it)

[ghost]

Sorry, I'm not sure where to redirect STDERR. I just:

• updated to code-coverage 3.3.0
• changed the package.json script to: ``"test:cy": "cypress run 2>&1",
• ran: ``$ DEBUG=code-coverage npm run test:cy 2>&1 > debug-test-cy.txt

debug-test-cy.txt is the result. I don't see anything that looks like a debug message(?).

I appreciate your help!

[bahmutov]

Hmm what is your support and plugins files? When you use cypress open does it show code coverage messages after each test?

Sent from my iPhone

On Apr 18, 2020, at 12:35, Andrew Andrews [email protected] wrote:

 Sorry, I'm not sure where to redirect STDERR. I just:

updated to code-coverage 3.3.0 changed the package.json script to: "test:cy": "cypress run 2>&1", ran: $ DEBUG=code-coverage npm run test:cy 2>&1 > debug-test-cy.txt debug-test-cy.txt is the result. I don't see anything that looks like a debug message(?).

I appreciate your help!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

Plugins & Support:

plugins-index.js.txt support-commands.js.txt support-index.js.txt

Here is the result of running both spec files in cypress open:

debug-cy-open.txt

Thank you kindly, Andrew

[ghost]

I should mention that, after the above, out.json was an empty object: {}

[bahmutov]

@Andrew-Opplaud how do you instrument your application when running Cypress tests? https://github.com/cypress-io/code-coverage#instrument-your-application

[ghost]

Thanks for your reply, Gleb. My cypress/plugins/index.js includes:

on('file:preprocessor', require('@cypress/code-coverage/use-browserify-istanbul'));

Based on this I was under the impression that would work. Am I mistaken?

I checked and node_modules/@cypress/code-coverage/use-browserify-istanbul.js exists.

I'm not using Babel (except for what Jest does automatically), but I also tried creating .babelrc containing:

{ "plugins": ["istanbul"] }

and added:

on('file:preprocessor', require('@cypress/code-coverage/use-babelrc'));

to plugins/index.js. When I ran npm run cy:test, out.json was still an empty object and the coverage report index.html was empty (except for the boilerplate, of course).

I tried changing the order of the on('file:preprocessor',...) lines in case that mattered, but it didn't. I also tried commenting-out the use-browserify-istanbul preprocessor when including use-babelrc, but the results were the same.

I appreciate your continued assistance. What should I try next?

[bahmutov]

Gotcha - so the "Alternative" way still applies only to unit test instrumentation. Let me add it to README, but in essence:

if your code does cy.visit(index.html) then anything index.html is loading has to be instrumented by YOU.

if your code is loaded directly from spec file, like this - Cypress can instrument it for you, this is called "unit test instrumentation".

import {add} from '../../src/utils'
it('works', () => {
  expect(add(2,3)).to.equal(5)
})

Thus in your case, YOUR web application was not instrumented, and thus output code coverage is empty.

[bahmutov]

Can you take a look at https://on.cypress.io/code-coverage#Instrumenting-code we give more advice there

[ghost]

Thanks for the explanation. I had hoped that combining Jest and Cypress (as per your cypress-and-jest example) automated the instrumentation for me, but I see now that I didn't notice the subtlety that the instrumentation was automated for Cypress unit tests, not the E2E tests.

So firstly, it sounds like Jest isn't really helping me in this situation: if I use Cypress for the unit tests as well as the E2E tests, then I can take Jest out of the equation. Is that correct?

Secondly, I've added a script to package.json:

`"nyc": "cd ~/OpplaudSaas && nyc instrument --compact=false OpplaudDial OpplaudDialTests/instrumented",`

Now, when I run npm run nyc, it quickly generates the files in the OpplaudDialTests\instrumented folder that correspond to the files in OpplaudDial (project source) folder but does not generate an .nyc_output/out.json file.

Is this to be expected?

Also, I've modified the test:cy script to:

`"test:cy": "npm run nyc && cypress run",`

and modified the import statements in my

Now, when I run npm run test:cy, it runs all of the tests and they pass, and I get out.json.txt with references back to the original sources instead of the instrumented versions.

Is this also to be expected?

Lastly, when I look in cypress-coverage\lcov-report\index.html, it's still essentially empty. :(

I feel like I'm getting closer... I hope I'm getting closer!?

Thanks again for any advice you can provide!

[bahmutov]

Can you please run with DEBUG=code-coverage and have the logs? With version 3.4.1 of the plugin.

[ghost]

Done... this is interesting: after running, there is now a coverage folder instead of a cypress-coverage folder, even though package.json still contains "nyc":{"report-dir": "cypress-coverage"}.

That probably explains the error messages in the output.

out.json looks the same as usual.

Thanks again!

[bahmutov]

Try v3.4.2 - I have fixed the passing of the report dir to NYC in release https://github.com/cypress-io/code-coverage/releases/tag/v3.4.2

[ghost]

Thanks, Gleb. The error messages are gone (debug342.txt) but out.json and (now) cypress-coverage/lcov-report/index.html look the same.

During npm install I got a lot of warnings about skipping various optional dependencies. I removed node_modules and installed again from scratch, but got the same warnings.

FWIW, coverage-final.json contains an empty object.

I appreciate your continued assistance!

[ghost]

Thanks, Gleb. The error messages are gone (debug342.txt) but out.json and (now) cypress-coverage/lcov-report/index.html look the same.

During npm install I got a lot of warnings about skipping various optional dependencies. I removed node_modules and installed again from scratch, but got the same warnings.

FWIW, coverage-final.json contains an empty object.

I appreciate your continued assistance!

[ghost]

Thanks, Gleb. The error messages are gone (debug342.txt) but out.json and (now) cypress-coverage/lcov-report/index.html look the same.

During npm install I got a lot of warnings about skipping various optional dependencies. I removed node_modules and installed again from scratch, but got the same warnings.

FWIW, coverage-final.json contains an empty object.

I appreciate your continued assistance!

[bahmutov]

seems due to GitHub problems the latest log fil you have linked is not found

[ghost]

Reattaching the output before and after purging package.json. Thanks!

[bahmutov]

@Andrew-Opplaud I really cannot do anything UNLESS you run and capture full logs with DEBUG=code-coverage turned on. Also running with latest version 3.5.1 would help

[ghost]

I'm sorry, it seems I don't understand where I am supposed to add DEBUG=code-coverage. I have tried many variations, including the following, but none seem to generate the additional output you're looking for:

DEBUG=code-coverage node node_modules/cypress/bin/cypress run 2>&1

I also removed node_modules and installed:

"devDependencies": {
  "@cypress/code-coverage": "^3.5.1",
  "babel-plugin-istanbul": "^6.0.0",
  "check-code-coverage": "^1.2.0",
  "cypress": "^4.4.1",
  "cypress-visual-regression": "^1.2.0"
}

But I get the same results (debug351.txt).

My shell is bash on Ubuntu. If you could please tell me what command I need to use, I will gladly try it!

I recognize that this is frustrating, apologize for my incompetence and, most of all, appreciate your help!

[bahmutov]

This is weird, but you have to figure out why the output is not generated. Is the plugin registered and working inside the plugins file? Do you see its messages in the command log?

[ghost]

No, I'm not seeing those messages in the command log.

While digging around trying to figure out why they're not there, I realized that, although I had instrumented the code in the project source folder, I did not instrument the code from the embedded <script> tag in the test page. So I moved that code to a separate x.js file, instrumented it, then replaced the <script>...</script> with <script src="instrumented/x.js"></script>.

Now, when I run cypress.run, the lcov-report/index.html file DOES include coverage information for that one file, but not for any of the files that it imports (or any of the files that the imports import).

I still do not see the messages in the log, but obviously the coverage is working, just not for the imported files.

Any idea why that would be? Thank you.

[ghost]

BTW the files are being imported, because the code runs and the tests pass!

[bahmutov]

Well, without debug logs, we cannot help much, unfortunately, sorry about that.

[ghost]

After discovering this and this, I came up with a solution... but it's not pretty:

The problem is that running nyc on the command line generates filesystem paths, both in the out.json file and in all of the files in the instrumented folder. But during the cypress run command, when it cy.visit()s the page and loads the JS files, they all have relative paths to the web server (in which . indicates the relative path to the instrumented folder.

The solution I found is: between running nyc instrument ..., and cypress run, replace all of the filesystem paths in both out.json and the instrumented files with relative webserver paths; essentially:

sed -i -e 's/\/path\/to\/project\/source/./g' .nyc_output/out.json
find instrumented -name "*.js" -exec sed -i -e 's/\/path\/to\/project\/source/./g' {} \;`

It would be Nice® if instrumentation happened automatically when Cypress visits a page. Barring that, it would be Almost As Nice® if Cypress included a simple command that would automate the process of instrumenting the files.

All the same, thank you for your attention to this problem and your attempts to help, which eventually led to the solution! Much appreciated!

[ghost]

On further investigation, it turns out that solution is not quite perfect: when I look at the coverage report, it shows the tables for the various files including those under the ./lib folder, but when I click on any file name except for the highest level file, instead of the source code in that file, I see a stack trace such as:

Unable to lookup source: ./lib/fLayout.js (ENOENT: no such file or directory, open './lib/fLayout.js')

Anyway, I think I'm at least on the right track toward a final solution. If I discover more, I'll post it here, and if you think of anything that might help, I'll appreciate your suggestions! Thanks again.

[dsacramone]

Did you end up sorting this out. I am running into the same issue as you have.

On further investigation, it turns out that solution is not quite perfect: when I look at the coverage report, it shows the tables for the various files including those under the ./lib folder, but when I click on any file name except for the highest level file, instead of the source code in that file, I see a stack trace such as:

Unable to lookup source: ./lib/fLayout.js (ENOENT: no such file or directory, open './lib/fLayout.js')

Anyway, I think I'm at least on the right track toward a final solution. If I discover more, I'll post it here, and if you think of anything that might help, I'll appreciate your suggestions! Thanks again.

[ghost]

Unfortunately, no. This was over a year ago so I don't remember the details, but I had to give up to finish the project on time. I've always planned to revisit it but haven't had the opportunity. If you discover anything new, please post here!