oxsecurity
npm install not work in gitlab ci with mega-linter V7
Describe the bug
when use eslint, need to install the dependencies first, because in the soure code, some lib needs:
in nextjs layout.tsx
import type { Metadata } from "next";
export const metadata: Metadata = {
title: "Next.js",
description: "The React Framework for the Web",
};
export default function RootLayout({
children,
}: {
children: React.ReactNode;
}) {
return (
<html lang="en">
<meta
name="viewport"
content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no"
/>
<body>{children}</body>
</html>
);
}
when run eslint, it needs not just eslint plugins, but also lib deps. Thus I need to install the full deps in package.json
To Reproduce
Here is the .mega-linter.yml
---
APPLY_FIXES: all
ENABLE_LINTERS:
- TSX_ESLINT
- JSON_PRETTIER
- HTML_HTMLHINT
- GRAPHQL_GRAPHQL_SCHEMA_LINTER
- ENV_DOTENV_LINTER
- CSS_STYLELINT
- CSS_SCSS_LINT
- YAML_PRETTIER
- YAML_YAMLLINT
- DOCKERFILE_HADOLINT
- EDITORCONFIG_EDITORCONFIG_CHECKER
- COPYPASTE_JSCPD
- REPOSITORY_GIT_DIFF
PRE_COMMANDS:
- command: npm install
cwd: workspace
secured_env: true
TSX_ESLINT_PRE_COMMANDS:
- command: pwd
cwd: workspace
- command: npm install
cwd: workspace
SHOW_ELAPSED_TIME: true
FILEIO_REPORTER: false
FILTER_REGEX_EXCLUDE: '(\.git/|\.husky/|\.vscode/|node_modules/|tsconfig\.json|\.d\.ts)'
CSS_STYLELINT_FILE_EXTENSIONS:
- .css
- .scss
- .less
- .sass
- .postcss
TSX_ESLINT_FILE_EXTENSIONS:
- .js
- .cjs
- .mjs
- .jsx
- .ts
- .mts
- .cts
- .tsx
EDITORCONFIG_EDITORCONFIG_CHECKER_FILTER_REGEX_EXCLUDE: (\.json)
TEXT_REPORTER: true
TEXT_REPORTER_SUB_FOLDER: linters_logs
CONSOLE_REPORTER: true
CONSOLE_REPORTER_SECTIONS: true
OUTPUT_DETAIL: detailed
GITLAB_COMMENT_REPORTER: true
GITHUB_STATUS_REPORTER: true
EDITORCONFIG_EDITORCONFIG_CHECKER_DISABLE_ERRORS: true
TSX_ESLINT_CLI_LINT_MODE: project
As you can see, here is the PRE_COMMANDS
PRE_COMMANDS:
- command: npm install
cwd: workspace
secured_env: true
TSX_ESLINT_PRE_COMMANDS:
- command: pwd
cwd: workspace
- command: npm install
cwd: workspace
It's failed:
here is the
package.json file in project
{
"name": "next-template",
"version": "1.0.0",
"scripts": {
"dev": "cross-env BUILD_ENV=development next dev --turbo",
"build-test": "cross-env BUILD_ENV=test next build",
"build-prod": "cross-env BUILD_ENV=prod next build",
"start": "cross-env NODE_ENV=production next start -p 8888",
"lint": "eslint --fix . "
},
"author": "",
"license": "MIT",
"devDependencies": {
"@commitlint/cli": "^18.4.0",
"@commitlint/config-conventional": "^18.4.0",
"@next/eslint-plugin-next": "^14.0.2",
"@styled/typescript-styled-plugin": "^1.0.0",
"@types/node": "^20.9.0",
"@types/react": "^18.2.37",
"@types/react-dom": "^18.2.15",
"@typescript-eslint/eslint-plugin": "^6.10.0",
"@typescript-eslint/parser": "^6.10.0",
"cross-env": "^7.0.3",
"eslint": "^8.53.0",
"eslint-config-prettier": "^9.0.0",
"eslint-import-resolver-typescript": "^3.6.1",
"eslint-plugin-import": "^2.29.0",
"eslint-plugin-react": "^7.33.2",
"eslint-plugin-react-hooks": "^4.6.0",
"eslint-plugin-yml": "^1.10.0",
"husky": "^8.0.3",
"is-ci": "^3.0.1",
"jscpd": "^3.5.10",
"lint-staged": "^15.0.2",
"postcss": "^8.4.31",
"postcss-html": "^1.5.0",
"postcss-less": "^6.0.0",
"prettier": "^3.1.0",
"stylelint": "^15.11.0",
"stylelint-config-rational-order": "^0.1.2",
"stylelint-config-sass-guidelines": "^10.0.0",
"stylelint-config-standard": "^34.0.0",
"stylelint-config-standard-scss": "^11.1.0",
"typescript": "^5.2.2",
"typescript-eslint-language-service": "^5.0.5",
"validate-branch-name": "^1.3.0",
"yaml-eslint-parser": "^1.2.2"
},
"config": {
"commitizen": {
"path": "@commitlint/cz-commitlint"
}
},
"dependencies": {
"next": "^14.0.2",
"react": "^18.2.0",
"react-dom": "^18.2.0"
}
}
All the deps are declared and worked well in local. Expected behavior
After npm install the linter can work well when use eslint
Additional context When i use the sample config to install single plugin, it works.
PRE_COMMANDS:
- command: npm install @next/eslint-plugin-next
cwd: "root" # Will be run at the root of MegaLinter docker image
It seems that ESLINT works in root, not work well in workspace.
How ever, there is a lot of deps in package.json, if all list in PRE_COMMANDS like above, that is silly. And when you add new deps in package,json, you need to update PRE_COMMANDS,
log info
Running with gitlab-runner 16.3.0 (8ec04662)
on
Preparing the "docker" executor
Using Docker executor with image oxsecurity/megalinter-javascript:v7 ...
Pulling docker image oxsecurity/megalinter-javascript:v7 ...
Using docker image sha256:870e90b8ed799e8c82c266dfd80b8fda862577f9085ddc1dbb9540fc8922[3]
Preparing environment
00:01
Running on runner-43ylhtju-project-913-concurrent-0 via it-sonarqube...
Getting source from Git repository
00:00
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/fe/template/next/.git/
Checking out 326f41b7 as detached HEAD (ref is master)...
Removing megalinter-reports/
Removing node_modules/
Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:870e90b8ed799e8c82c266dfd80b8fda862577f9085ddc1dbb9540fc89223abe for oxsecurity/megalinter-javascript:v7 with digest oxsecurity/megalinter-javascript@sha256:f5758a6eb9f439ea974258f35741d6f977b5ed7ccd31f805c6490d3177f864a5 ...
Setting git safe.directory DEFAULT_WORKSPACE: /builds/fe/template/next ...
Setting git safe.directory default: /github/workspace ...
Setting git safe.directory to /tmp/lint ...
[MegaLinter init] ONE-SHOT RUN
[config] /builds/fe/template/next/.mega-linter.yml + Environment variables
.:oool' ,looo;
.xNXNXl .dXNNXo.
lXXXX0c. 'oKXXN0;
.oKNXNX0kxdddddddoc,. .;lodddddddxk0XXXX0c
.:kKXXXXXXXXXXXXNXX0dllx0XXXXXXXXXXXXXXXKd,
.,cdkOOOOOOOO0KXXXXXXXXXXK0OOOOOOOkxo:'
'ckKXNNNXkc'
':::::;. .c0XX0l. .;::::;.
'xXXXXXx' :kx: ;OXXXXKd.
.dKNNXXO; .. :0XXXXKl.
.lKXXXX0: .lKXXXX0:
:0XXXXKl. .dXXXXXk,
;kXXXXKd:cxXXXXXx'
'xXNXXXXXXXXXKo.
.oKXXXXNXXX0l.
.lKNNXNNXO:
,looool'
==========================================================
============= MegaLinter, by OX.security =============
========= https://ox.security?ref=megalinter ===========
==========================================================
----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
- Image Creation Date: 2023-10-28T23:01:51Z
- Image Revision: b48455a119cc28045eee8f1e9d0a542a85e[7]
- Image Version: v7.5.0
----------------------------------------------------------------------------------------------------
The MegaLinter documentation can be found at:
- https://megalinter.io/7.5.0
----------------------------------------------------------------------------------------------------
MegaLinter initialization (expand for details)
MegaLinter will analyze workspace [/builds/fe/template/next]
[Pre] run: [npm install] in cwd [/builds/fe/template/next]
[Pre] result:
added 23 packages, and audited 24 packages in 1m
3 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
...
...
...
✅ Linted [COPYPASTE] files with [jscpd] successfully - (0.97s) (expand for details)
- Using [jscpd v3.5.10] https://megalinter.io/7.5.0/descriptors/copypaste_jscpd
- MegaLinter key: [COPYPASTE_JSCPD]
- Rules config: [/builds/fe/template/next/.jscpd.json]
--Log detail:
┌────────────┬────────────────┬─────────────┬──────────────┬──────────────┬──────────────────┬───────────────────┐
│ Format │ Files analyzed │ Total lines │ Total tokens │ Clones found │ Duplicated lines │ Duplicated tokens │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ javascript │ 6 │ 251 │ 1151 │ 0 │ 0 (0%) │ 0 (0%) │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ tsx │ 1 │ 26 │ 157 │ 0 │ 0 (0%) │ 0 (0%) │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ Total: │ 7 │ 277 │ 1308 │ 0 │ 0 (0%) │ 0 (0%) │
└────────────┴────────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴───────────────────┘
Found 0 clones.
HTML report saved to /builds/fe/template/next/megalinter-reports/copy-paste/html/
Detection time:: 245.622ms
Unable to get number of errors with regex_sum and ✖ ([0-9]+) problem
❌ Linted [TSX] files with [eslint]: Found 1 error(s) - (4.08s) (expand for details)
- Using [eslint v8.52.0] https://megalinter.io/7.5.0/descriptors/tsx_eslint
- MegaLinter key: [TSX_ESLINT]
- Rules config: identified by [eslint]
[Pre][TSX_ESLINT] run: [pwd] in cwd [/builds/fe/template/next]
[Pre][TSX_ESLINT] result:
/builds/fe/template/next
[Pre][TSX_ESLINT] run: [npm install] in cwd [/builds/fe/template/next]
[Pre][TSX_ESLINT] result:
up to date, audited 24 packages in 2s
3 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
--Error detail:
Oops! Something went wrong! :(
ESLint: 8.52.0
ESLint couldn't find the plugin "@next/eslint-plugin-next".
(The package "@next/eslint-plugin-next" was not found when loaded as a Node module from the directory "/builds/fe/template/next".)
It's likely that the plugin isn't installed correctly. Try reinstalling by running the following:
npm install @next/eslint-plugin-next@latest --save-dev
The plugin "@next/eslint-plugin-next" was referenced from the config file in ".eslintrc.js".
If you still can't figure out the problem, please stop by https://eslint.org/chat/help to chat with the team.
+----SUMMARY------+----------------------+---------------+-------+-------+--------+--------------+
| Descriptor | Linter | Mode | Files | Fixed | Errors | Elapsed time |
+-----------------+----------------------+---------------+-------+-------+--------+--------------+
| ✅ COPYPASTE | jscpd | project | n/a | | 0 | 0.97s |
| ✅ CSS | stylelint | list_of_files | 1 | 0 | 0 | 0.71s |
| ⚠️ EDITORCONFIG | editorconfig-checker | list_of_files | 16 | | 1 | 0.06s |
| ✅ JSON | prettier | list_of_files | 3 | 0 | 0 | 0.84s |
| ✅ REPOSITORY | git_diff | project | n/a | | 0 | 0.0s |
| ❌ TSX | eslint | project | n/a | yes | 1 | 4.08s |
| ✅ YAML | prettier | list_of_files | 3 | 0 | 0 | 0.34s |
| ✅ YAML | yamllint | list_of_files | 3 | | 0 | 0.19s |
+-----------------+----------------------+---------------+-------+-------+--------+--------------+
[Gitlab Comment Reporter] No merge request has been found, so no comment has been posted
[Updated Sources Reporter] No source file has been formatted or fixed
❌ Error(s) have been found during linting
To disable linters or customize their checks, you can use a .mega-linter.yml file at the root of your repository
More info at https://megalinter.io/7.5.0/configuration/
Uploading artifacts for failed job
00:00
Uploading artifacts...
WARNING: report: no matching files. Ensure that the artifact path is relative to the working directory (/builds/fe/template/next)
ERROR: No files to upload
ERROR: Job failed: exit code 1
Here is the ESLint config file .eslintrc.js
/** @type {import('eslint').Linter.Config} */
module.exports = {
extends: [
// npm install --save-dev @typescript-eslint/eslint-plugin
"plugin:@typescript-eslint/recommended",
// npm install --save-dev eslint-config-prettier
"prettier",
// npm install --save-dev eslint-plugin-react
"plugin:react/jsx-runtime",
// npm install --save-dev @next/eslint-plugin-next
"plugin:@next/next/recommended",
// npm install --save-dev eslint-plugin-react-hooks
"plugin:react-hooks/recommended",
// npm install eslint-plugin-import --save-dev
"plugin:import/recommended",
// 支持ts
"plugin:import/typescript",
// npm install eslint-plugin-yml --save-dev
"plugin:yml/recommended",
],
globals: {
wx: "readonly",
},
settings: {
react: {
version: "detect",
},
next: {
rootDir: ".",
},
"import/parsers": {
"@typescript-eslint/parser": [".ts", ".tsx", ".mdx"],
},
"import/resolver": {
typescript: true,
node: true,
},
},
// npm install --save-dev @typescript-eslint/parser
parser: "@typescript-eslint/parser",
plugins: ["@typescript-eslint"],
ignorePatterns: [
"megalinter-reports/**/*",
"!.lintstagedrc.cjs",
"!.stylelintrc.cjs",
"!.commitlintrc.cjs",
"!.eslintrc.js",
],
overrides: [
{
files: [".eslintrc.{js,cjs}", "**/*.tsx", "**/*.cjs", "**/*.ts", "**/*.mjs", "**/*.jsx", "**/*.mdx"],
rules: {
quotes: "off",
"@typescript-eslint/no-explicit-any": "warn",
"import/first": 2,
},
},
{
files: ["*.yaml", "*.yml"],
parser: "yaml-eslint-parser",
// Options used with yaml-eslint-parser.
parserOptions: {
defaultYAMLVersion: "1.2",
},
},
],
noInlineConfig: true,
};
@sharh as eslint is installed globally, I think you don't have other choice than installing plugins using the full command in PRE_COMMANDS, like npm install @next/eslint-plugin-next
But you remark is very valid... we could add an option to parse package.json and automatically build & run the dependencies commands to install :)
Thanks for repy.If that becomes true, that would be very nice. And when will it be true?
@sharh when someone makes a Pull Request to make it available :) Maybe you ? ^^ I my side I already have a big backlog ^^
I imagine some config like JAVASCRIPT_ESLINT_USE_NPM=true, and if found, it would run npm install then call eslint via npm run eslint :)
Wouldn't it make it really hard to support, on our side? Npm is pretty intertwined. Unless there is a kind of environment possible.
@echoix in that case would run the eslint installed by npm install and not the one contained in MegaLinter image... it would be something experimental and disabled by default, as like you i'm not 100% sure it wouldn't generate regressions
I was more thinking on the side that we can't officially offer support for something that we can't control. It's not really testable on our side, and their setup would have to work with what node versions we ship. Does the install of node_modules would like "uninstall" our linters (or downgrade them) to satisfy dependencies? Does an "virtualenv" concept exist in node? What happens if we change the way we package/install our linters? As always, precommands can be used to do anything users want..
@echoix our node_modules are installed in /node-deps , i assume what I propose would install them in working directory/node_modules, so it wouldn't mess with our installs ^^ But maybe I missed something ? :/
So basicaly I have similar problem. I'm trying to use TYPESCRIPT_ES with my dependencies to install in
PRE_COMMANDS:
- command: npm install @angular-eslint/eslint-plugin --save-dev -g
cwd: "root"
output is like this:
[Pre] run: [cd /node-deps && npm install @angular-eslint/eslint-plugin --save-dev -g] in cwd [/]
[Pre] result:
added 136 packages in 3s
31 packages are looking for funding
run `npm fund` for details
But later I get:
- Using [eslint v8.54.0] https://megalinter.io/7.6.0/descriptors/typescript_eslint
- MegaLinter key: [TYPESCRIPT_ES]
- Rules config: [/.eslintrc.json]
- Number of files analyzed: [797]
--Error detail:
Oops! Something went wrong! :(
ESLint: 8.54.0
ESLint couldn't find the plugin "@angular-eslint/eslint-plugin".
(The package "@angular-eslint/eslint-plugin" was not found when loaded as a Node module from the directory "/tmp/lint".)
It's likely that the plugin isn't installed correctly. Try reinstalling by running the following:
npm install @angular-eslint/eslint-plugin@latest --save-dev
The plugin "@angular-eslint/eslint-plugin" was referenced from the config file in "--config#overrides[0]".
I've tried in PRE_COMMANDS also cwd: workspace but with te same results.
Where I should install npm package to make it discoverable and work?
EDIT: for future reference, I managed to make it work with command: npm install --production=false
Regarding prettier plugins: They can only be discovered by MegaLinter's prettier if you point to the exact location in .prettierrc.js using require.resolve:
module.exports = {
plugins: ['prettier-plugin-svelte'], // does not work
}
module.exports = {
plugins: [require.resolve('prettier-plugin-svelte')], // works
}
Afterward, plugins can be installed using PRE_COMMANDS, as usual:
PRE_COMMANDS:
- command: npm install prettier-plugin-svelte
This might be relevant, if MegaLinter tries to install the required plugins based on the package.json or for anyone using prettier plugins in the meantime.
