Sourcemaps repeatedly uploading with esbuild plugin

[ancheetah]

Environment

"@sentry/aws-serverless": "^8.26.0" "@sentry/esbuild-plugin": "^2.22.2" "serverless": "^3.26.0" "serverless-esbuild": "^1.52.1"

Description

Sourcemaps should be uploaded for aws lambda functions when packaging (sls package). I've tried two different configurations:

1. Loading the sentry esbuild-plugin as a serverless-esbuild plugin as suggested here. This results in good stack traces but the sourcemaps are uploaded repeatedly when run in a Github Action. It finds something like 4 files then uploads then 8 files then uploads and repeats until it finds all the files. How often it's uploaded and the number of files found is different every time.

serverless.yml

package:
  individually: true

custom:
  esbuild:
    bundle: true
    minify: false
    sourcemap: true
    platform: 'node'
    outputBuildFolder: '.build'
    keepOutputDirectory: true
    skipRebuild: true
    logLevel: debug
    plugins: './esbuild.plugins.js'

esbuild.plugin.js

const { sentryEsbuildPlugin } = require('@sentry/esbuild-plugin');

module.exports = (serverless) => {
  return [
      sentryEsbuildPlugin({
        org: my_org,
        project: my_project,
        authToken:
          process.env.SOURCEMAPS === 'true'
            ? serverless.configurationInput.provider.environment.SENTRY_AUTH_TOKEN
            : undefined,
        sourcemaps: {
          assets: ['./.esbuild/.build/**/*'],
          filesToDeleteAfterUpload: ['./.esbuild/.build/**/*.js.map'],
        },
        telemetry: false,
        debug: true,
      }),
    ]
  }

2. Loading the sentry plugin within the esbuild.config.js as per the documentation. This uploads the sourcemaps only once however the sourcemap debug IDs are injected into the output esbuild folder but not the .serverless folder which gets deployed so we get no stack trace in Sentry.

serverless.yml

package:
  individually: true

custom:
  esbuild:
    config: './esbuild.config.js'

esbuild.config.js

const { sentryEsbuildPlugin } = require("@sentry/esbuild-plugin");
const esbuild = require("esbuild");
const glob = require("glob");

module.exports = async (serverless) => {
  const sources = glob.sync('./src/**/index.ts');

  await esbuild.build({
    entryPoints: sources,
    bundle: true,
    minify: false,
    sourcemap: true,
    platform: 'node',
    outdir: '.dist-esbuild',
    logLevel: 'debug',
    plugins: [
    sentryEsbuildPlugin({
      org: my_org,
      project: my_project,
      authToken: process.env.SOURCEMAPS === 'true'
        ? serverless.configurationInput.provider.environment.SENTRY_AUTH_TOKEN 
        : undefined,
      sourcemaps: {
        assets: './.dist-esbuild/**/*',
        filesToDeleteAfterUpload: ['./.dist-esbuild/**/*.js.map'],
      },
      telemetry: false,
      debug: true,
    }),
    ]
  })
}

[lforst]

Hi, can you verify how often other plugins are called with sls? I fear this is up to sls and nothing we can control.

[ancheetah]

Hi @lforst, with the first method, according to the logs the esbuild is run only once before the sentry plugin. It is when the sentry plugin starts that it repeatedly uploads. Here is a sample of logs that keep repeating (some info redacted). Happy to share more with you privately.

> Found 10 files
> Analyzing 10 sources
> Adding source map references
> Bundled 6 files for upload
> Bundle ID: 12345
> Uploaded files to Sentry
> Found 12 files
> File upload complete (processing pending on server)
> Organization: my-org
> Project: my-project
> Release: 12345
> Dist: None
> Upload type: artifact bundle

[lforst]

Nothing sticks out to me as what may cause this. Would you mind providing a reproduction example so we can investigate further? Thanks!

[ancheetah]

@lforst here is a reproduction with a Github action that you can use to test

https://github.com/ancheetah/serverless-sentry-esbuild

[lforst]

This rang a bell and I dug out this issue: https://github.com/getsentry/sentry-javascript-bundler-plugins/issues/245

It seems like sls is repeatedly calling esbuild. Unfortunately, from the plugin's perspective, there is nothing (reasonable) we can do to prevent the plugin being invoked multiple times. You can potentially narrow down the upload scope of the plugin with sourcemaps.assets.

As an alternative, you can also use Sentry CLI to upload sourcemaps which you can invoke once for all of your bundles: https://docs.sentry.io/platforms/javascript/sourcemaps/uploading/cli/

[ancheetah]

Thanks for tracking that down. I think using the Sentry CLI after building and packaging will be a good workaround. However, the sentry-esbuild plugin is still needed to inject the debug IDs at the time of packaging with sls since it is those packaged files that will get deployed. Could you add an option to the sentry-esbuild plugin that injects the debug IDs but skips uploads?

[mydea]

You should not need the esbuild plugin when you use sentry-cli - see https://docs.sentry.io/platforms/javascript/sourcemaps/uploading/cli/#3-inject-debug-ids-into-artifacts for how to inject the debug IDs directly in this scenario! Let us know if that does not work for you.

[ancheetah]

This works for us. Thanks @mydea!

[markuspoerschke]

@ancheetah Can you show how you solved the problem? I see that you created a repo showing the problem. It would be nice to see the solution. I came from Google and am faced with “I solved the problem…”, but no explanation. It would be awesome if you can provide some details what changes to the serverless YAML file would be needed.

[ancheetah]

Hey @markuspoerschke we call the sentry-cli commands to upload sourcemaps in a script that does some re-packaging after sls package. First, in your serverless.yml turn the sourcemaps option on for the esbuild plugin:

custom:
  esbuild:
    sourcemap: true

Then after you call sls package you can use these sentry-cli commands to inject the debug IDs into the sourcemaps from esbuild and upload them to Sentry:

    sentry-cli sourcemaps inject $SOURCEMAP_DIR
    sentry-cli sourcemaps upload $SOURCEMAP_DIR

You can either set the Sentry auth token in a .sentryclirc or load them into your re-packaging script like we do here: https://github.com/masslight/ottehr/blob/412c6ced1227e67f4174889494ca00484c410cd7/packages/intake/zambdas/package.json#L29-L30 https://github.com/masslight/ottehr/blob/412c6ced1227e67f4174889494ca00484c410cd7/packages/intake/zambdas/scripts/package-for-release.sh#L25-L29

Last, you can delete the sourcemaps and re-zip your functions before deploying them.