Issues using React components in test environment

[deerob4]

We've recently replaced our hand-rolled components with the ones provided by our toolkit. All works well when running the extension normally, but we get the following exception when running our tests:

/Users/dee/Programming/work/nordic/vscode-nrf-connect/node_modules/@vscode/webview-ui-toolkit/react/index.js:3
import { provideReactWrapper } from '@microsoft/fast-react-wrapper';
^^^^^^

SyntaxError: Cannot use import statement outside a module

This is being hit when our test imports the React component that makes use of the toolkit components. For context, our source code and tests are compiled using tsc -p ., and are run through the runTests utility provided by @vscode/test-electron. This means there's no bundling stage involved, which I guess is why we're not seeing this when the extension is running normally.

Is it possible that there's something misconfigured in the library, i.e. a missing "type": "module" in the package.json file?

[hawkticehurst]

Thanks for the bug report @deerob4!

Been a bit busy with other pressing work lately but will put it on the top of my todo list once I have some free cycles

Also thank you so much for the call out of a missing "type": "module"! I thought that was something I included long ago, but adding it might be a solution to some other problems I've been tackling with the toolkit

[hawkticehurst]

Hey @deerob4!

I finally found some time to look at this and have updated the toolkit to use the type: module!

But before I go off to create a custom test setup to match your configuration I thought I'd double-check to see if your extension is publicly available by any chance? Would love to be able to directly test the changes in your source code as a verification that I'm solving your problem (if possible)

[deerob4]

Hey, thanks for looking into this!

Our extension isn't open source unfortunately, but I created a minimal example that reproduces the issue. It implements a simple counter webview that uses the VSCodeButton component, as well as a couple of tests, and includes a rough approximation of our test environment. I hope it's useful!

[hawkticehurst]

Thank you so much! This is immensely helpful!

[hawkticehurst]

Hey @deerob4!

Finally circling back again with some updates. So I just made a PR with the type: module changes added (I'll make publish a new toolkit release shortly), but I thought I would test the changes with your example and didn't seem to have any luck running npm run test 😕

With that said, I'm also getting an "Unable to read file" error while trying to run the sample extension normally. It looks like you are getting a reference to a webview.js file (line 17 in src/extension.ts):

const scriptUri = panel.webview.asWebviewUri(
  vscode.Uri.joinPath(webviewRoot, "webview.js")
);

But once the extension is built, there is no webview.js file present in the dist directory –– thus the error. So wondering if that could perhaps be a contributing factor to this issue as well??

Like I said I'll go ahead and publish a new release of the toolkit so you can test this all on your end. If there's still an issue, let me know if there's something I can do to get the sample extension running correctly so we can continue to troubleshoot whatever is going on 😊

[deerob4]

Hey, thanks for getting back on this! You're completely right, there was an error in the sample - sorry for wasting your time with that. I pushed a commit that changes the reference to test.js, and it now runs as expected.

However, I'm still seeing an error with the tests after updating to v1.2.0, though this time it's slightly different:

Error [ERR_REQUIRE_ESM]: require() of ES Module /path/to/vscode-webview-ui-toolkit-test/node_modules/@vscode/webview-ui-toolkit/react/index.js from /path/to/vscode-webview-ui-toolkit-test/out/src/webview/Counter.js not supported.

Some searching suggested that adding {"type": "module"} to the extension itself and changing all the files to ESM modules might have an impact, but in the end the same error threw in VS Code's own loader.js. It also seems like exclusively using ESM modules shouldn't be necessary to use the toolkit, so that seems like a red herring.

[hawkticehurst]

No worries at all! Happy to have been able to help unblock that, at least!

I also pulled down your updates and was able to reproduce the ERR_REQUIRE_ESM error and updated the extension to use ESM modules and eventually produced the error from loader.js.

I also dove into the generated build code and definitely see what the issue is.

At its core the original error message you typed out above is exactly the issue at hand (go figure lol), there is a line of code in the generated build output that equates to require('an-esm-module-file').

More specifically it's line 27 in out/src/webview/Counter.js:

const react_2 = require("@vscode/webview-ui-toolkit/react");

This is (as I'm sure you know) a no-no since you can't use CommonJS import syntax (i.e. require) to import an ESM file (i.e. @vscode/webview-ui-toolkit/react or technically node_modules/@vscode/webview-ui-toolkit/react/index.js once Node's module resolution algorithm kicks in at runtime).

Like you, I then tried to fix this problem by changing the generated extension build code to be ESM based. But eventually hit the inverse problem where inside VS Code core the loader.js file uses CommonJS syntax and tries (unsuccessfully) to require your now ESM-based extension files 😪

A solution?

I then reverted back to the starting state and noticed that there is another error message below the one you typed out above giving a suggestion on how to fix the problem.

Instead change the `require` of index.js in `{insert-long-path}/out/src/webview/Counter.js` to a dynamic import() which is available in all CommonJS modules.

So back on line 27 in out/src/webview/Counter.js I manually changed the line to:

const react_2 = await import("@vscode/webview-ui-toolkit/react");

And then just explicitly ran the command node ./out/test/runTests.js to avoid the pretest NPM command you have setup that rebuilds the output.

This didn't work because there is no global await in the version of node that I (or this project?) was using, so I changed the line once more to:

const react_2 = (async () => {await import("@vscode/webview-ui-toolkit/react")})();

^Aka an async anonymous function that gets called immediately.

This actually seems to fix the immediate problem! 🎊 However....

1. This is a manual override to the build output files (I don't know how to get this code to be generated via build configs)
2. There are still more errors after this error gets resolved
3. Some of those errors actually seem to come from the fact that because the async code takes a while to resolve/return, the react_2 variable gets used before it has a value and contains undefined when used (I think?)

At this point, I have to transition back to some other projects I'm working on so I hope you can pick up the trail from here and see if you can figure out something I haven't

I might actually suggest revisiting the topic of trying to use ESM modules inside VS Code extensions, there is a long thread in the VS Code repo where people are discussing this exact topic because the webview toolkit react components are not the only ESM-only libraries that people want to use. From a casual glance at the thread, it seems like another cause of this problem could be how TypeScript treats code generation of dynamic imports?

Hope this all helps, good luck, and let me know what else you discover!

[hawkticehurst]

Also, seeing now that this seems to be an issue with ESM code being used in a CommonJS project and to a greater extent how VS Code itself is handling ESM-based extension code (versus something that is specifically caused by the React toolkit components) I'm going to go ahead and close this issue if that's cool?

If you feel strongly otherwise (or discover something that points back to the React toolkit components being a source of the issue) please open the issue again and we can discuss 😊

Also please do continue to send updates and I'm more than happy to help debug as I have time (I definitely want to know if there's a solution to this in case others run into the same problem)!