bit Latest Bit version `1.6.73` causes build failures in Ripple CI / Bit Cloud

Describe the bug

Since the Ripple CI / Bit Cloud build process has started using Bit version 1.6.73 I've been seeing build failures on my React 18 environment (see Ripple CI logs here), which is based on @teambit/[email protected]:

the error occurs even if I don't update my components - just running bit tag --unmodified and exporting the results throws a build error for me on the latest Bit version
the bit build process still succeeds when I running it locally with any Bit version

Screenshots, exceptions and logs - error details from Ripple CI

The full build logs and all errors are visible here (the env is in a public scope): https://bit.cloud/sitestudio/envs/~ripple-ci/job/sitestudio-sitestudio-envs-rebuilding-scope/component/sitestudio.envs/[email protected]

The main part of the error message says:

PnpmError: [email protected] install: prebuild-install || node-gyp rebuild --release Exit status 1

Steps to Reproduce

You can see an example when comparing the last 2 versions of the tina-react-env here: https://bit.cloud/sitestudio/envs/tina-react-env/~compare
- There are no code differences between the 2 versions
- Version 0.0.2 built perfectly 1 month ago, while version 0.0.3 is failing to build
You can import that tina-react-env and it will build fine locally
- run bit tag --unmodified and bit export and the Ripple CI throws the error detailed above

Expected Behavior

New Bit versions should not lead to build failures on Ripple CI / Bit Cloud for components that were previously building successfully.
- Or if there are errors on Ripple CI it should ideally be possible to recreate those same errors when running bit build locally
Alternatively a way to specify the Bit version that Ripple CI uses would provide an immediate resolution: #8627

Specifications

Bit version: 1.6.73
Workspace type: harmony
Node version: 21.2.0
pnpm version: 7.4.0
Platform: Bit Cloud / Ripple CI
Relevant env: The tina-react-env is based on the React 18 env: @teambit/[email protected]. It only has very minor customizations

Mar 06 '24 01:03 coreyaus

For some reason I'm unable to reproduce the issue locally with the same version. Screenshot 2024-03-05 at 9 16 32 PM

@guysaar223 , @GiladShoham , can you take a look?

Mar 06 '24 02:03 davidfirst

Does it consistently fail? Even if you retry? Looks related to our virtual fs but we have not changed anything in our ripple CI package manager. I have tried running it locally in a container and it worked.

Sometimes the build is skipped, when the build artifacts are found in the cache, so maybe that is why it worked the last time.

This is definitely a bug but finding the reason will be hard if I won't be able to reproduce it locally. In that case the only way to find the reason will be to add debug messages to the virtual fs, so that more info is printed in ripple.

Mar 06 '24 11:03 zkochan

I created a new simple component with [email protected], this one builds without issues: https://bit.cloud/zoltan/dogfooding/~ripple-ci/job/zoltan-main-Gb0BJcI1A

Mar 06 '24 12:03 zkochan

Thanks for the quick responses on this! I really appreciate your review, particular as you say because it's always hard to debug issues when it's not possible to reproduce them locally so thanks for your time so far 🙏

In answer to your first question, it is failing consistently at the moment, even when retrying. Happily it's a pretty simple env to recreate or replace so I'll run a test now to see if it builds successfully when recreated from scratch. The only customisations in that env are these 3 extra peer dependencies added to the host-dependencies.ts and env.jsonc files:

tinacms
@tinacms/mdx
@tinacms/schema-tools

The env doesn't depend on better-sqlite3 directly but that package is included indirectly due to the dependency on tinacms, via the following path:

└─ [email protected]
   ├─ @tinacms/[email protected]
      ├─ [email protected]
         └─ [email protected]

Let me know if those notes trigger any insights and I'll report back shortly once I've finished creating a fresh version of the env from scratch (I'm halfway through that process now) 👍

Mar 06 '24 13:03 coreyaus

The update is that I just finished recreating that env from scratch and ran into the same error: https://bit.cloud/sitestudio/envs/~ripple-ci/job/sitestudio-main-update-test-react-env-to-include-extra-peer-dependencies

Steps to reproduce:

Here are the steps I followed to re-create that env from scratch. Please let me know if there's something I'm missing or doing wrong (and apologies in advance if that's the case!):

Created a new workspace and test-react-env:

mkdir temp-new-bit-workspace && cd temp-new-bit-workspace
bit init --default-scope=sitestudio.envs
bit create react-env test-react-env --aspect teambit.react/react-env --scope=sitestudio.envs
bit install --add-missing-deps
bit tag --message "Create base test-react-env"
bit export

Manually added the latest versions of the 3 extra dependencies mentioned above (tinacms, @tinacms/mdx & @tinacms/schema-tools) to the host-dependencies.ts and env.jsonc files (see this code diff), then ran the following commands to tag and export those updates:
```
bit install --add-missing-deps
bit tag --message "Update test-react-env to include extra peer dependencies"
bit export
```

Please let me know if I missed something in that process, or if there are any other changes you'd like me to test. Would it be helpful to manually configure better-sqlite3 as a dependency somewhere within the workspace or env (or manually override the version as a test)?

Thanks again!

Mar 06 '24 13:03 coreyaus

Thanks for the new details. I was able to reproduce the issue locally in a ripple container using a fork of sqlite that I used in my component. Using this command.

bit sign zoltan.dogfooding/[email protected] --multiple --rebuild

Mar 06 '24 14:03 zkochan

Thanks for your work on this @zkochan and great to hear you were able to reproduce the issue locally.

I wanted to check in with a couple of quick questions:

Are there are any tests I can run to help your work delivering a fix?
Have you found any workarounds in the meantime?

I'm currently unable to build or update an environment if it uses a package that includes better-sqlite3 in its dependencies. Thanks for any help you can offer!

Things I've tried

Setting better-sqlite3 as an explicit dependency of the env in the host-dependencies.ts and env.jsonc files

Trying to remove the dependency based on the docs here: https://bit.cloud/teambit/dependencies/aspect-docs/dependency-resolver/~compositions

"teambit.dependencies/dependency-resolver": {
  "packageManager": "teambit.dependencies/pnpm",
  "policy": {
    "dependencies": {
      "better-sqlite3": "-"
    },
    "peerDependencies": {
      "better-sqlite3": "-"
    }
  },
},

Adding better-sqlite3 to the neverBuiltDependencies array (again based on these docs)

Unfortunately none of those approaches worked (you can probably tell I was mostly just trying everything I could think of!)

PS: It looks like Ripple CI is now using a not-yet-released version of Bit (1.6.84) to run the build process - is that right?

Mar 14 '24 01:03 coreyaus

Right now the only solution would be to turn off fusepm (cc @guysaar223) or somehow build the package outside of the mounted directory.

I spent about 2 days debugging it but couldn't figure out the issue. The better_sqlite3.o file seems to have the wrong file type. It is data instead of ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped. Looks like the C compiler can't complete the compilation correctly on the Fuse filesystem.

removing better-sqlite3 or adding it to neverBuiltDependencies should work though. I should check this.

Mar 14 '24 11:03 zkochan

Same problem

✖ (9/10) env: teambit.react/react, task "teambit.preview/preview:GeneratePreview" has failed in 5s
- (10/10) [Preview: PreBundlePreview]
✔ (10/10) [Preview: PreBundlePreview] Completed successfully in 249μs
✔ running tasks. Succeeded in 18s
  running post-build for all tasks 
✔ running post-build for all tasks. Succeeded in 502μs
✖ Running build pipeline using 1 environment(s), total 10 tasks. Failed in 18s
- all objects were persisted successfully!
The following errors were found while running the build pipeline
Failed task 1: "teambit.preview/preview:GeneratePreview" of env "teambit.react/react"
component: esui.lib/[email protected]
ModuleNotFoundError: Module not found: Error: Can't resolve '@emotion/styled' in '/tmp/capsules-root/sign-capsules/2024-4-15/1lqkiw6/node_modules/.pnpm/@[email protected][email protected]/node_modules/@mui/styled-engine'
ModuleNotFoundError: Module not found: Error: Can't resolve '@emotion/styled' in '/tmp/capsules-root/sign-capsules/2024-4-15/1lqkiw6/node_modules/.pnpm/@[email protected][email protected]/node_modules/@mui/styled-engine'
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/Compilation.js:2022:28
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:817:13
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:10:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:275:22
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:448:22
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:118:11
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:689:25
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:893:8
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:1013:5
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/neo-async/async.js:6883:13
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/webpack/lib/NormalModuleFactory.js:996:45
    at finishWithoutResolve (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:567:11)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:656:15
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:15:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:89:43
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/forEachBail.js:39:13
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/AliasPlugin.js:148:14
    at next (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/forEachBail.js:35:3)
    at forEachBail (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/forEachBail.js:49:9)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/AliasPlugin.js:61:5
    at _next0 (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:8:1)
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:30:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:89:43
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/Resolver.js:714:5
    at eval (eval at create (/root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:15:1)
    at /root/.bvm/versions/1.6.151/bit-1.6.151/node_modules/enhanced-resolve/lib/DirectoryExistsPlugin.js:41:15
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)
Found 1 errors in 1 component

May 15 '24 12:05 nardote

resolved on rippleci

Aug 12 '24 08:08 itaymendel