Module Not Found errors when upgrading from v3.2.5 to latest

[soutarm]

I've been tasked with updating an old repo from HTLEngine v3.2.5 to latest (v6.4.22 at the time of writing this) as we have dependency vulnerabilities. A number of templates are using a module to inline SVGs into the output markup via <sly data-sly-use... syntax.

At Webpack-based build time the custom Sightly plugin relies on the removed withUseDirectory() method (see sample code below) to direct the compiler to the module file location. The code is identical to this old issue https://github.com/adobe/htlengine/issues/114. I've followed the recommendations in that issue (i.e. the example given here https://github.com/adobe/htlengine/pull/117 but updated to use module.createRequire() which was added in Node v12.2.0) to no avail.

I have also attempted to use withResourceLoader(fsResourceLoader..., also but still no luck.

I've tried various paths in the templates themselves, e.g. <sly data-sly-use.logo="${'/sightly.SightlyStringUtils' or <sly data-sly-use.logo="${'./sightly.SightlyStringUtils' and even then full path name <sly data-sly-use.logo="${'/Users/xxxx/Documents/projects/my-project/src/modules/sightly.SightlyStringUtils' each with and without the .js file extension. No luck.

I also copied the module into a component folder to see if it could be found there. Nope.

I tried dropping the module into the project root, still nope.

Our Adobe vendors and their support team looked into it but they also weren't able to solve the issue, recommending I open an issue in the repo.

The project directory structure is:

-config
  |-plugins
     |-sightly-webpack-plugin
       |-index.js
 -src
  |-components
     |-component-name
       |-component.html (HTL Template)
  |-modules
     |-sightly.SightlyStringUtils.js

Expected Behaviour

During build the module should be found and inline the SVG file into the output HTML.

Actual Behaviour

At build time each instance of the module name in the template leads to a "module not found" error.

Reproduce Scenario (including but not limited to)

Attempt to run Webpack build which includes the following (working) plugin config:

const SightlyWebpackPlugin = require('./plugins/sightly-webpack-plugin');

...

plugins: [
    ...
    new SightlyWebpackPlugin({
      ...
      modulePath: './src/modules',
    }),
]

Steps to Reproduce

Run a Webpack build to compile the templates

Platform and Version

NodeJs v18.17.1 HTLEngine v6.4.22 Webpack v5.91.0

Sample Code that illustrates the problem

Sample Template Code:

<sly data-sly-use.logo="${'sightly.SightlyStringUtils' @ svgPath=properties.logo}">
  ${logo.svgItem @ context='unsafe'}
</sly>

Existing Plugin Code where modulePath is set to "./src/modules" (working in v3.2.5):

const templateCompiler = new Compiler().withSourceMap(true).withRuntimeVar(Object.keys(templateData))
const templateRuntime = new Runtime().withUseDirectory(self.options.modulePath).setGlobal(templateData)
const templateTransformer = await templateCompiler.compileToFunction(templateSource)
const templateCode = await templateTransformer(templateRuntime)

Sample attempted fix using createRequire() and withDirectory():

const modulePath = path.resolve(self.options.modulePath, 'sightly.SightlyStringUtils.js');
const templateCompiler = new Compiler().withDirectory('.').withSourceMap(true).withRuntimeVar(Object.keys(templateData));
const templateRuntime = new Runtime().setGlobal(templateData);
const templateTransformer = await templateCompiler.compileToFunction(templateSource, '.', createRequire(modulePath));
const templateCode = await templateTransformer(templateRuntime);

Sample attempted fix using fsResourceLoader() and withResourceLoader():

const templateCompiler = new Compiler().withSourceMap(true).withRuntimeVar(Object.keys(templateData));
const templateRuntime = new Runtime().withResourceLoader(fsResourceLoader(self.options.modulePath)).setGlobal(templateData);
const templateTransformer = await templateCompiler.compileToFunction(templateSource, '.', require);
const templateCode = await templateTransformer(templateRuntime);

Logs taken while reproducing problem

Using existing code but removing withUseDirectory():

Error: Cannot find module 'sightly.SightlyStringUtils'
Require stack:
- /Users/xxxx/Documents/projects/project-name/node_modules/@adobe/htlengine/src/compiler/Compiler.js
- /Users/xxxx/Documents/projects/project-name/node_modules/@adobe/htlengine/src/index.js
- /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js
- /Users/xxxx/Documents/projects/project-name/config/webpack.common.js
- /Users/xxxx/Documents/projects/project-name/config/webpack.dev.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/webpack-cli.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/bootstrap.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/bin/cli.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack/bin/webpack.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at main (eval at compileToFunction (/Users/xxxx/Documents/projects/project-name/node_modules/@adobe/htlengine/src/compiler/Compiler.js:279:22), <anonymous>:976:18)
    at /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js:317:44
    at async Promise.all (index 0)
    at async /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js:188:11 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/xxxx/Documents/projects/project-name/node_modules/@adobe/htlengine/src/compiler/Compiler.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/@adobe/htlengine/src/index.js',
    '/Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js',
    '/Users/xxxx/Documents/projects/project-name/config/webpack.common.js',
    '/Users/xxxx/Documents/projects/project-name/config/webpack.dev.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/webpack-cli.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/bootstrap.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/bin/cli.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack/bin/webpack.js'
  ]
}

When using createRequire():

Error: Cannot find module 'sightly.SightlyStringUtils'
Require stack:
- /Users/xxxx/Documents/projects/project-name/src/modules/sightly.SightlyStringUtils.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at main (eval at compileToFunction (/Users/xxxx/Documents/projects/project-name/node_modules/@adobe/htlengine/src/compiler/Compiler.js:279:22), <anonymous>:976:18)
    at /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js:312:44
    at async Promise.all (index 0)
    at async /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js:188:11 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/xxxx/Documents/projects/project-name/src/modules/sightly.SightlyStringUtils.js'
  ]
}

When using fsResourceLoader():

Error: Cannot find module 'sightly.SightlyStringUtils'
Require stack:
- /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js
- /Users/xxxx/Documents/projects/project-name/config/webpack.common.js
- /Users/xxxx/Documents/projects/project-name/config/webpack.dev.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/webpack-cli.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/bootstrap.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/bin/cli.js
- /Users/xxxx/Documents/projects/project-name/node_modules/webpack/bin/webpack.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:121:18)
    at main (eval at compileToFunction (/Users/xxxx/Documents/projects/project-name/node_modules/@adobe/htlengine/src/compiler/Compiler.js:279:22), <anonymous>:976:18)
    at /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js:290:44
    at async Promise.all (index 0)
    at async /Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js:187:11 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/xxxx/Documents/projects/project-name/config/plugins/sightly-webpack-plugin/index.js',
    '/Users/xxxx/Documents/projects/project-name/config/webpack.common.js',
    '/Users/xxxx/Documents/projects/project-name/config/webpack.dev.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/webpack-cli.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/lib/bootstrap.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack-cli/bin/cli.js',
    '/Users/xxxx/Documents/projects/project-name/node_modules/webpack/bin/webpack.js'
  ]
}

[tripodsan]

hi @soutarm , thank you for reporting. do you have a public repository where this can be reproduced?

[soutarm]

Hi @tripodsan, unfortunately I can't make the code public but I've cloned a private repo and invited you to collaborate. The included code is what we currently have (i.e. with the removed withUseDirectory still in place), the two packages that are updated (see below) plus I've also included node_modules as the package repo is not public either.

You should be able to just clone and run npm run build

Updated packages within the repo:

@adobe/htlengine: 3.2.5 -> 6.4.22
aem-clientlib-generator: 1.7.1 -> 1.8.0

Thanks heaps!

[tripodsan]

thanks, I'll have a look soon

[tripodsan]

hi @soutarm . I looked into the issue but haven't found a solution yet. it requires some more time....sorry.