jest_test doesn't find Google Auth default credentials when using Google cloud libraries

[damiansan239]

🐞 bug report

Affected Rule

The issue is caused by the rule:

jest_test

Is this a regression?

Yes, the previous version in which this bug was not present was: ....

Description

A clear and concise description of the problem...

I'm working with Google Cloud Storage libraries in my package. Running the jest command on my test works well but doesn't in the jest_test rule.

🔬 Minimal Reproduction

https://github.com/nerdraven/google_cloud_storage

🔥 Exception or Error

==================== Test output for //apps/data-importer:test:
 FAIL  apps/data-importer/image_upload.test.js
  Test Download of image
    ✓ Image gets downloaded (808 ms)
  Google Cloud Storage testsuite
    ✕ It should upload files to Cloud Storage (1205 ms)

  ● Google Cloud Storage testsuite › It should upload files to Cloud Storage

    Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.

      at GoogleAuth.getApplicationDefaultAsync (node_modules/google-auth-library/build/src/auth/googleauth.js:183:19)
      at GoogleAuth.getClient (node_modules/google-auth-library/build/src/auth/googleauth.js:568:17)
      at GoogleAuth.request (node_modules/google-auth-library/build/src/auth/googleauth.js:621:24)
      at Upload.makeRequest (node_modules/@google-cloud/storage/build/src/gcs-resumable-upload.js:605:21)
      at retry.retries (node_modules/@google-cloud/storage/build/src/gcs-resumable-upload.js:312:29)
      at Upload.createURIAsync (node_modules/@google-cloud/storage/build/src/gcs-resumable-upload.js:309:21)

  ● Google Cloud Storage testsuite › It should upload files to Cloud Storage

    Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.

      at GoogleAuth.getApplicationDefaultAsync (node_modules/google-auth-library/build/src/auth/googleauth.js:183:19)
      at GoogleAuth.getClient (node_modules/google-auth-library/build/src/auth/googleauth.js:568:17)
      at GoogleAuth.authorizeRequest (node_modules/google-auth-library/build/src/auth/googleauth.js:609:24)
          at async Promise.all (index 1)

  ● Google Cloud Storage testsuite › It should upload files to Cloud Storage

    expect.assertions(1)

    Expected one assertion to be called but received zero assertion calls.

      at Object.assertions (../../../apps/data-importer/image_upload.test.ts:32:12)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 passed, 2 total
Snapshots:   0 total
Time:        3.872 s
Ran all test suites.
Jest did not exit one second after the test run has completed.

This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.

🌍 Your Environment

Operating System:

Linux (WSL2)

Output of bazel version:

5.1.0

Rules_nodejs version:

(Please check that you have matching versions between WORKSPACE file and @bazel/* npm packages.)

5.4.2

Anything else relevant?

[damiansan239]

After some search in Github, I was told I should set the DEBUG_AUTH env to true. With that I get this INFO

==================== Test output for //apps/data-importer:test:
  console.info
    FetchError {
      message: 'request to http://metadata.google.internal./computeMetadata/v1/instance failed, reason: getaddrinfo ENOTFOUND metadata.google.internal.',
      type: 'system',
      errno: 'ENOTFOUND',
      code: 'ENOTFOUND',
      config: {
        url: 'http://metadata.google.internal./computeMetadata/v1/instance',
        headers: { 'Metadata-Flavor': 'Google' },
        retryConfig: {
          noResponseRetries: 0,
          currentRetryAttempt: 0,
          retry: 3,
          httpMethodsToRetry: [Array],
          statusCodesToRetry: [Array]
        },
        responseType: 'text',
        timeout: 3000,
        paramsSerializer: [Function: paramsSerializer],
        validateStatus: [Function: validateStatus],
        method: 'GET'
      }
    }

      at Object.isAvailable (node_modules/gcp-metadata/src/index.ts:220:15)
          at async Promise.all (index 1)

  console.info
    FetchError {
      message: 'request to http://metadata.google.internal./computeMetadata/v1/instance failed, reason: getaddrinfo ENOTFOUND metadata.google.internal.',
      type: 'system',
      errno: 'ENOTFOUND',
      code: 'ENOTFOUND',
      config: {
        url: 'http://metadata.google.internal./computeMetadata/v1/instance',
        headers: { 'Metadata-Flavor': 'Google' },
        retryConfig: {
          noResponseRetries: 0,
          currentRetryAttempt: 0,
          retry: 3,
          httpMethodsToRetry: [Array],
          statusCodesToRetry: [Array]
        },
        responseType: 'text',
        timeout: 3000,
        paramsSerializer: [Function: paramsSerializer],
        validateStatus: [Function: validateStatus],
        method: 'GET'
      }
    }

      at Object.isAvailable (node_modules/gcp-metadata/src/index.ts:220:15)

 FAIL  apps/data-importer/image_upload.test.js
  Test Download of image
    ✓ Image gets downloaded (868 ms)
  Google Cloud Storage testsuite
    ✕ It should upload files to Cloud Storage (1410 ms)

  ● Google Cloud Storage testsuite › It should upload files to Cloud Storage

    Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.

      at GoogleAuth.getApplicationDefaultAsync (node_modules/@google-cloud/storage/node_modules/google-auth-library/build/src/auth/googleauth.js:183:19)
      at GoogleAuth.getClient (node_modules/@google-cloud/storage/node_modules/google-auth-library/build/src/auth/googleauth.js:565:17)
      at GoogleAuth.request (node_modules/@google-cloud/storage/node_modules/google-auth-library/build/src/auth/googleauth.js:618:24)
      at Upload.makeRequest (node_modules/@google-cloud/storage/build/src/gcs-resumable-upload.js:605:21)
      at retry.retries (node_modules/@google-cloud/storage/build/src/gcs-resumable-upload.js:312:29)
      at Upload.createURIAsync (node_modules/@google-cloud/storage/build/src/gcs-resumable-upload.js:309:21)

  ● Google Cloud Storage testsuite › It should upload files to Cloud Storage

    Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.

      at GoogleAuth.getApplicationDefaultAsync (node_modules/@google-cloud/storage/node_modules/google-auth-library/build/src/auth/googleauth.js:183:19)
      at GoogleAuth.getClient (node_modules/@google-cloud/storage/node_modules/google-auth-library/build/src/auth/googleauth.js:565:17)
      at GoogleAuth.authorizeRequest (node_modules/@google-cloud/storage/node_modules/google-auth-library/build/src/auth/googleauth.js:606:24)
          at async Promise.all (index 1)

  ● Google Cloud Storage testsuite › It should upload files to Cloud Storage

    expect.assertions(1)

    Expected one assertion to be called but received zero assertion calls.

      at Object.assertions (../../../apps/data-importer/image_upload.test.ts:25:12)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 passed, 2 total
Snapshots:   0 total
Time:        3.069 s
Ran all test suites.
Jest did not exit one second after the test run has completed.

This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.
================================================================================
Target //apps/data-importer:test up-to-date:
  bazel-bin/apps/data-importer/test.sh
  bazel-bin/apps/data-importer/test_loader.cjs
  bazel-bin/apps/data-importer/test_require_patch.cjs
INFO: Elapsed time: 16.004s, Critical Path: 15.65s
INFO: 5 processes: 1 internal, 4 linux-sandbox.
INFO: Build completed, 1 test FAILED, 5 total actions
//apps/data-importer:test                                                FAILED in 6.0s
  /home/nerdraven/.cache/bazel/_bazel_nerdraven/8747b2477eee1cf6dbfd36fcd8e894af/execroot/closera/bazel-out/k8-fastbuild/testlogs/apps/data-importer/test/test.log

INFO: Build completed, 1 test FAILED, 5 total actions

I'm not sure, but it seems to be GCE environment mode or so

[punkch]

I had a similar issue, where jest_test (I am using the macro from the examples) couldn't properly setup the google credentials provided by the environment. On my local machine, am setting these with gcloud auth application-default login and then all the google libraries are supposed to work automagically. They did work from the nodejs_binary, but not from jest_test, where it was throwing some error that the auth client couldn't figure the project id. What I did to workaround this, was to hardcode process.env.GCLOUD_PROJECT='my-gcp-project' in my google related tests, but am really confused why it behaves like this.

Hopefully the above gives you some pointers.

[damiansan239]

Am really grateful for this.

Thanks very much

[damiansan239]

What was able to work for me now was hardcoding the service account credentials. I can work with his. Thank was again

  public static client: Storage = new Storage({
    projectId: "my-gcp-project",
    credentials: {
      type: "service_account",
      private_key:"-----BEGIN PRIVATE KEY-----",
      client_email: "service-account-email",
      client_id: "keyfile-client-id",
    },
  });

[punkch]

Well, if you have to use service account credentials, better save them somewhere as json and do it like this (to avoid accidental commit):

  public static client: Storage = new Storage({
    projectId: "my-gcp-project",
    keyFilename: "/path/to/keyfile.json"  
  });

docs here.

If you have to use a specific service account, the recommended practice is to not use service account credentials, but to impersonate it. Steps to achieve this:

• install gcloud
• Do gcloud auth application-default login and authenticate with your user service account
• Grant your user the iam.serviceAccountTokenCreator role on the google storage service account.
• enable the IAM Credentials API
• follow the example here here

Then, if you run your application on GCP (cloud run, gke, app engine, etc) everything has application default credentials, so you just grant these service accounts the iam.serviceAccountTokenCreator and they will be able to impersonate the google storage service account as well.

Hope this helps, and I still don't understand why jest_test can't figure the project id, but nodejs_binary can (new to Bazel)

[damiansan239]

I should have used the method you highlighted here but, my case for using it was that even using the default application credentials from gcloud or even setting GCLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS env vars as test_env attributes or in the code, it still doesn't work. From my observation, using the DEBUG_AUTH env var, it seems like the bazel makes the SDK think it is in a GCE environment and tries to query the metadata server for an auth token which fails. Using the credentials file overrides those assumptions, I think.

Also, where the keyFilename attribute is in the constructor, I don't know a definitive way of telling the absolute path to keyfile in a bazel workspace. Also, I think bazel doesn't look past the workspace in the sandbox environment.

For the snippet, I gave I didn't actually implement it in my code that way though, It was just a proof of concept.

Thanks anyways for your attention to this issue.

[punkch]

Do you pass the --google_default_credentials flag to your bazel command?

[damiansan239]

I tried the flag. It still doesn't help

[alexeagle]

This seems like an issue with Bazel, not with rules_nodejs or with the jest_test rule. I imagine you could reproduce the same problem with an sh_test, right?

Since Bazel is maintained by Google and you're using their cloud, it would be nice for them to provide some support for this issue if you file under bazelbuild/bazel repo instead.

[punkch]

Just to elaborate, for the sake of clarity, the --google_default_credentials flag, i.e. bazel run --google_default_credentials //path/to/my:target works as it should for me, both when developing locally and when using Google Cloud Build. My nodejs_binary bazel targets can use google client libraries and consume google apis without problem. jest_test finds the default credentials as well, but can not figure the GCP project Id so I've just hardcoded process.env.GCLOUD_PROJECT = 'my-gcp-project' in my test files to workaround this. The code consuming the google apis is the same for the nodejs_binary and the jest_test targets so I can't figure why the difference in behaviour.

[alexeagle]

I think you can repro that with sh_test - the problem is likely how Bazel scrubs the environment variables when running tests - https://bazel.build/reference/test-encyclopedia you can use --test_env=GCLOUD_PROJECT to tell Bazel to propagate that one environment variable from the context where bazel is running into the subprocess (run bazel with --subcommands to see what the test process spawn looks like)

[punkch]

Thank you very much for the pointers. It is most likely something wrong with my setup. I'll try all this, but I have to do a lot of learning beforehand.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 6 months. It will be closed if no further activity occurs in 30 days. Collaborators can add a "cleanup" or "need: discussion" label to keep it open indefinitely. Thanks for your contributions to rules_nodejs!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without any activity since it was labeled "Can Close?"