Incompatible with Next.js / Webpack

[BobDickinson]

Environment details

• OS: MacOS 14.6.1
• Node.js version: 20.10.0
• npm version: 10.2.3
• @google-cloud/spanner version: 7.14.0

Steps to reproduce

1. Create a new Next.js project, import @google-cloud/spanner
2. Run the project in dev mode via next dev. Experience:

Error: ENOENT: no such file or directory, open 'google/longrunning/operations.proto'

3. After fixing the above, run next build and next start. Experience:

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type number (11363)

Analysis

The Node Spanner lib (this project), like some other Google libs using gax, require .proto files to be loaded from the package at runtime. This presents challenges for packaging solutions like webpack. If the first case above, webpack doesn't know that it needs to make the .proto files available in the final runtime files, which is why it doesn't package them and they can't be found. The solution to this is the explicitly copy the .proto files to an appropriate location in the webpack config, as follows (in next.config.js):

webpack: (config, { isServer }) => {
  if (isServer) {
    // Copy the google-gax protos to the .next/server/protos directory so that the Spanner client library can find them.
    //
    config.plugins.push(
      new CopyPlugin({
        patterns: [
          {
            from: "node_modules/@google-cloud/spanner/node_modules/google-gax/build/protos",
            to: path.resolve(__dirname, '.next/server/protos'),
          },
        ],
      })
    );
  }
  return config;
}

Once that is fixed, when doing build/start, the Spanner module code attempts to resolve the directory for the .proto files using require.resolve:

https://github.com/googleapis/nodejs-spanner/blob/1f06871f7aca386756e8691013602b069697bb87/src/common-grpc/service.ts#L46

In webpack and similar environments, the behavior of require.resolve cannot be relied up to produce a valid path, and in the case of webpack, it doesn't produce a path (or even a string). In webpack, require.resolve produces a numeric module reference, which causes the above referenced code to fail and throw an exception, preventing the module from loading. The webpack workaround is as follows:

const resolvedGoogleGaxPath = require.resolve('google-gax');

webpack: (config, { isServer }) => {
  if (isServer) {
    config.module.rules.push({
      test: /common-grpc\/service\.js$/, 
      loader: 'string-replace-loader',
      options: { 
        search: 'require\.resolve\(\'google-gax\'\)', 
        replace: JSON.stringify(resolvedGoogleGaxPath),
      }
    });
  }
  return config;
};

There are several issues open between Next.js and webpack in this area, but I don't get the impression either project sees it as their responsibility to address them. This same problem exists with other Google libs that use the grpc common code.

I don't necessarily expect that anyone is going to fix this on the Node Spanner side either. I'm opening this issue mainly because I blew about 12 hours digging into this and coming up with the above solution (finding no real credible/workable solutions when looking in the usual places based on the error messages I was seeing). I would expect the dev team will just close this, but at least future Next.js travellers may find it and save themselves some effort.

Here is the stack trace for the first issue (proto files not packaged):

 ⨯ Error: ENOENT: no such file or directory, open 'google/longrunning/operations.proto'
    at Object.open (node:internal/fs/sync:78:18)
    at Object.openSync (node:fs:565:17)
    at Object.readFileSync (node:fs:445:35)
    at fetch (webpack-internal:///(rsc)/./node_modules/protobufjs/src/root.js:126:34)
    at Root.load (webpack-internal:///(rsc)/./node_modules/protobufjs/src/root.js:152:105)
    at Root.loadSync (webpack-internal:///(rsc)/./node_modules/protobufjs/src/root.js:183:17)
    at loadProtosWithOptionsSync (webpack-internal:///(rsc)/./node_modules/@grpc/proto-loader/build/src/util.js:67:29)
    at loadSync (webpack-internal:///(rsc)/./node_modules/@grpc/proto-loader/build/src/index.js:216:61)
    at Spanner.loadProtoFile (webpack-internal:///(rsc)/./node_modules/@google-cloud/spanner/build/src/common-grpc/service.js:767:58)
    at eval (webpack-internal:///(rsc)/./node_modules/@google-cloud/spanner/build/src/common-grpc/service.js:304:35)
    at Array.forEach (<anonymous>)
    at new GrpcService (webpack-internal:///(rsc)/./node_modules/@google-cloud/spanner/build/src/common-grpc/service.js:302:36)
    at new Spanner (webpack-internal:///(rsc)/./node_modules/@google-cloud/spanner/build/src/index.js:277:9)
    at getSpanner (webpack-internal:///(rsc)/./libs/spanner.ts:20:21)
    at getSpannerInstance (webpack-internal:///(rsc)/./libs/spanner.ts:35:23)
    at getDatabase (webpack-internal:///(rsc)/./libs/spanner.ts:81:26)
    at doesDatabaseExist (webpack-internal:///(rsc)/./libs/spanner.ts:117:26)
    at SpannerTenantModel.get (webpack-internal:///(rsc)/./models/spanner/tenant.ts:115:84)
    at GET (webpack-internal:///(rsc)/./app/api/auth/route.ts:175:30)
    at async /Users/bob/Documents/GitHub/teamspark-ai-api/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:62591 {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'google/longrunning/operations.proto'
}
(node:7911) Warning: google/longrunning/operations.proto not found in any of the include paths /Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/protos,(rsc)/node_modules/google-gax/build/protos

And here's the stack trace for the second issue (require.resolve doesn't work as expected in webpack):

 ⨯ TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type number (11363)
    at new NodeError (node:internal/errors:406:5)
    at validateString (node:internal/validators:162:11)
    at Object.dirname (node:path:1279:5)
    at 61492 (/Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/chunks/3020.js:421:129)
    at t (/Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/webpack-runtime.js:1:143)
    at 50753 (/Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/chunks/3020.js:476:311)
    at t (/Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/webpack-runtime.js:1:143)
    at 86838 (/Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/chunks/3020.js:1:124)
    at t (/Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/webpack-runtime.js:1:143)
    at 76711 (/Users/bob/Documents/GitHub/teamspark-ai-api/.next/server/chunks/3020.js:317:1758) {
  code: 'ERR_INVALID_ARG_TYPE'

[Flux159]

I believe that this is also causing Bun's single file executable to fail at runtime when trying to load "google-gax" in a spanner request: https://github.com/oven-sh/bun/issues/19908

Is there a way to patch this behavior to ensure that google-gax is packaged correctly for a single executable? I can fix this by just doing a bun install google-gax after moving the executable, but it defeats the purpose of having a single binary that I can give to someone else without needing to install dependencies.

[surbhigarg92]

Spanner Client Library is not compatible with client-side JavaScript (e.g., browsers, Webpack, Next.js frontend).

[Macil]

Next.js uses Webpack on backend code too. I believe it's needed for its React Server Component support.

[surbhigarg92]

@Macil In my quick POC , nextjs worked with Spanner Client for backend. Can you please share steps/sample app with me to reproduce the issue ?