Expose an `id` for concurrent test runners (like `JEST_WORKER_ID`)

[blimmer]

What is the problem this feature will solve?

When running tests concurrently (the default setting in the Node test runner), it's common practice to split concurrent test runs across multiple local resources (like locally running servers, databases, etc).

Many other popular test runners offer this feature via an environment variable:

• jest: JEST_WORKER_ID
• mocha: MOCHA_WORKER_ID
• vitest: VITEST_POOL_ID

This makes it very easy to set up the proper database connection. For example, here's a snippet from my codebase:

if (process.env.STAGE === "test" && process.env.JEST_WORKER_ID) {
  process.env.DB_DATABASE = `myapp_tests_${process.env.JEST_WORKER_ID}`;
}

Then, in my database docker container, I create n databases, one for each worker. For example, if I run jest with --maxWorkers 8, I create myapp_tests_1 -> myapp_tests_8. During tests, the parallel test suites talk to a different database to avoid deadlocks.

What is the feature you are proposing to solve the problem?

The simplest solution for users coming from other frameworks would be to provide an environment variable just like the other test runners (e.g., NODE_TEST_WORKER_ID).

However, if that's not feasible, adding a workerId to the TestContext could also be a good solution. This solution is not as ideal as the environment variable because people would need to pass the test context to whatever establishes their db/server/etc connection. In my case, at least, this would require refactoring of code. Right now, I rely on that variable being set at file import time.

What alternatives have you considered?

I considered trying to find a solution based on pid, but it's not reliable enough. @RedYetiDev mentioned in Slack that there might be a solution using hooks. I have not dug into this yet.

[cjihrig]

We could add an environment variable fairly easily, but I have a question:

if (process.env.STAGE === "test" && process.env.JEST_WORKER_ID) {
  process.env.DB_DATABASE = `myapp_tests_${process.env.JEST_WORKER_ID}`;
}

There is already a NODE_TEST_CONTEXT environment variable that can be used to determine that the test runner is being used (if you actually need something like that). Why not use something like crypto.randomUUID() to create a unique ID for each file?

[blimmer]

The problem is that, as an external step before the test runner is launched, I:

• Start up the postgres instance via docker compose
• Run an init script to create n databases (based on the parallelism I run the tests with)
• Run my migration logic (node-pg-migrate) against all the test databases
• Create a flush_database() function in each test database to run beforeEach test

I don't want to do this for each test file because the migration logic is quite slow. It's much faster to migrate once and have a fast flush_database function to reset database state between tests.

So a UUID, for my use case, would not be appropriate. Having a sequential integer between 1 and n (where n is the max concurrency) is required for the database setup logic that occurs outside the test runner context.

[cjihrig]

Got it. Thanks for the explanation. I think we should add support for this.

[0hmX]

I can work on this! will pick this up!

[cjihrig]

Thanks @cu8code. Please be sure to handle both cases of test isolation. When --experimental-test-isolation=process, each child process should get an environment variable from 1 to N. When --experimental-test-isolation=none, only the value of 1 should be used.

[hpatel292-seneca]

Hi @cu8code, are you working on this? If not I can give it a try.

[0hmX]

I am working on this 👍

[JakobJingleheimer]

Is worker.threadId possibly what you're looking for?

[blimmer]

We discussed threadId in Slack. It's not quite equivalent to what's available in other test runners:

[JakobJingleheimer]

Ahh. The same discussion in multiple places 😅

Is that truly a requirement though? Why is sequentiality actually required? Do you need to be able to guess ids relative to each other? If so, my spidy-senses are tingling. It seems merely having a unique and consistently available id to derive the table name is all you need:

myapp_tests_${threadId} should be fine within a test.

[blimmer]

I described this in the issue description. TLDR;

My test suite interacts with a database. Recreating the database (with all tables) for each test suite is very slow because I'd need to re-run all migrations, which takes several seconds. Therefore, before the test suite runs, I prepare n test databases (where n is the maximum concurrency) with a flush_database() method that truncates the database between tests. Before the test suite starts running, I need to have static values to name the test databases.

The presence of this feature in Jest, Mocha, and Vtest suggests that many users interested in switching to the node native test runner will want this.

[JakobJingleheimer]

You can use a "global" before to handle this; it will run within the thread, so threadId will be available (and other tests run within that same thread will have the same threadId for deriving the name).

[blimmer]

I suppose that could work; it'd just require some refactoring of how I'm doing things now. With this strategy, though, it seems possible to end up with many orphaned test databases.

For example, if I was using a debugging session for a test (very common) and "CTRL+C"-ed out of the process, skipping the global after to clean up the my_test_db_{thread_id}, the database would be orphaned.

In general, to make it easy for people to switch from the top three most popular other test runners, we should provide the static 1->n environment vairable. It's a feature that I, and many others, will likely expect.

[JakobJingleheimer]

It definitely does work. I've done it.

process 'beforeExit' event

[blimmer]

I appreciate you providing alternatives. If you all decide not to match the feature from Mocha, Jest and Vite, I suppose I can go this direction.

However, if it's really as simple as what @cu8code has proposed with #56091, I don't see why Node shouldn't match the feature.

[stephenh]

@JakobJingleheimer adding more data points from our use case, our test suite has 1,250 individual test files (*.test.ts), the majority of which use a database connection.

Today with Jest, we run yarn db-setup to pre-configure 8 our_database_${n} schemas, so we can run have yarn test execute 8 tests in parallel, and finish in "only" ~5-10 minutes, depending on the machine specs.

(Granted, 5-10 minutes is not great--ofc faster is always better, which is why we're looking into node:test and specifically avoiding Jest's per-test-file isolation cost.)

If we did not have determinism in the our_database_${n} assignment, then every invocation of yarn test would pay a set up (and tear down) cost of a new/dedicated our_database_${random pid} which is going to add another ~30-60 seconds of setup time (we have 450 tables in our db schema, so it takes awhile even to init a clean/new db schema), per worker, per yarn test invocation.

Granted, this would be only a ~10-20% cost increase in overall yarn test time, but we're looking for every speed up we can get--as I imagine you would as well, when facing a ~5-10 minute yarn test time. :sweat_smile:

As Ben has said, being able to have a deterministic/logical worker id (vs. the physical thread id) is key to letting our yarn db-setup and yarn test commands coordinate with each other.

Per your musings in #56091 about your beforeExit suggestion being a "complementary [...] and better" design than our approach, do these new data points change your mind?

I'm always happy to use the latest/best idea for a problem instead of "whatever we did before", but I'm not seeing how "create a new ${our_database_random id} per worker" for every yarn test invocation, at our scale, is a good/better idea.

[JakobJingleheimer]

Unless I'm missing something, I think this is not new beyond what I understood (or assumed).

You must pay a setup cost at some point. At the start of each thread seems an appropriate time (my suggestion).

I did something similar at my last job (mongodb in memory) with the same seeding and migrations. For ~450 tests, total run-time was 7.8s, of which ~2.5s was setup.

I expect your use-case is not uncommon, so I'd be happy to lay out how to do this in an article.

[stephenh]

Right, the difference is whether the setup cost is once per "change to the migrations/schema" (after which the engineer manually runs yarn db-setup) or once for every single invocation to yarn test.

Granted, in CI this doesn't matter, because CI has to db-setup + test for every run, but if I'm an engineer doing local iterations/a TDD loop, I might make ~a few changes to the migrations/schema (and run yarn db-setup once), but then run the tests ~10-20 times w/o changing the migrations, and w/stable our_database_${n}s, we can avoid re-paying the db/schema setup cost.

Appreciate the offer to write up an article, but we don't need to be shown how to do per-worker schema setup/tear down--we could do that if we wanted, we just don't want to, for the reasons we've outlined. :-)

[JakobJingleheimer]

Do you have some time this week to jump on a quick call? Specifically regarding "change to the migrations/schema".

[stephenh]

Hi @JakobJingleheimer ; we genuinely appreciate the offer to VC! If we were stuck on some super-intricate Node internals issue, it would be invaluable.

That said, I don't think there's a lot of unknowns/ambiguities at this point to resolve via discussion; we've described our use case (avoiding per-test/per-worker setup costs), and as Ben lightly alluded to, we think so many other major test frameworks supporting this feature makes it unlikely the ~2-3 of us will come up with a novel solution to something that many others have attempted to solve, and seem to have settled on "stable worker ids" as a suitable solution. It just doesn't seem controversial to us. :shrug:

Granted, it's the node:test maintainers decision either way, so we ofc defer. We will likely have to use mocha in the short-term anyway, until this feature (should it get accepted) hits a node LTS release, but would love to switch over when/if that happens. Thanks!

[JakobJingleheimer]

The reason I suggested the call is to understand "change to the migrations/schema", which I don't and I think the VC would be significantly easier than trying to explain over text. As it is, I do not believe this change is necessary. That could change my mind.

[stephenh]

change to the migrations/schema

We use node-pg-migrate to manage our db schema, so each time we need a new column / new table / etc, we write a migrations/(timestamp)-add-author-first-time.ts that, when executed by yarn db-setup, issues a ALTER TABLE ADD COLUMN to alter the db.

So what yarn db-setup does is: start from a clean/empty pg schema (or technically a ~recently new pg_dump of the schema), and apply all pending migrations in the migrations/ directory.

If the engineer hasn't changed any migrations/*.ts files (since their last yarn db-setup invocation), then the database schema itself will not have changed, and we don't have any reason to redo the CREATE TABLEs / ALTER TABLEs / etc (which, per prior comments, can take ~30 seconds on large db schemas), because each of the our_database_${n}s is already setup.

While we happen to use node-pg-migrate, this flow will generally/likely look extremely similar to apps that use postgres/likely any relational db. I.e. anyone that uses prisma will rerun the equivalent yarn db-setup each time they change their prisma schema file. Anyone that uses an "use annotations to define the db schema" (TypeORM, maybe MikroORM iirc) will rerun their own yarn db-setup each time they change their annotations.

But fundamentally engineers "change the schema" (the migration SQL files, the prisma model file, their entity's annotations) an order-or-two-magnitude less often than they run yarn test.

So we'd prefer for the test workers to just assume the our_database_${n} schema has been put in place for them, but they can only do that if they have a deterministic way to know ${n}.

[mtnt]

What`s the state now?

I run node --test --experimental-test-isolation=process **/*.spec.js and in each file I get a threadId and they all are equal. I guess it happens because there are several child processes and each of them has own main thread. So threadId can not define the test DB for me...

[sejalkailashyadav]

Hi! I'd like to work on this issue as my first contribution. Can I take it?

[0hmX]

@sejalkailashyadav, this is not a very beginner-friendly issue. Please take a look at userland-migrations. There is a lot of good first

[ghst-0]

There is already a NODE_TEST_CONTEXT environment variable that can be used to determine that the test runner is being used (if you actually need something like that). Why not use something like crypto.randomUUID() to create a unique ID for each file?

I recently ran into this and I found NODE_TEST_CONTEXT to be unreliable.

Using my setup (described below) when run through the command line

db.js hit
NODE_TEST_CONTEXT undefined
db.js hit
NODE_TEST_CONTEXT child-v8
▶ Tests function.js
...

Here db.js is loaded once with undefined causing one of the database connections to use the production db instead of the testing one. For each test suite subsequently db.js is hit (loaded) once with NODE_TEST_CONTEXT being child-v8.

My workaround is to check for the existence of the --test flag instead, this seems to work consistently both through the command line and through my IDE (Idea based).

---

For context it might help if I provide a complete example for database testing using node.js native test runner when using postgres.js. If nothing else it might help others that end up here looking for a workaround to this issue of how to test databases using the native test runner.

It solves these problems

• Only empties/recreates the database once per test run
• Collect test coverage for all database tests combined
• Initiating the test runner through the command line
• Running with --experimental-test-isolation=process works
• Individual tests can be run through the IDE
• Individual test suites can be run through the IDE
• All tests can be run through the IDE (if the appropriate flag is set in the configuration, i.e. --test-global-setup=test/global-setup.js)
• In the normal case the database gets released and the process exits after the tests complete

All these somehow uses the same context and the database does not get emptied when it shouldn't/the database connection stays up.

What does not work (I do not know how fix these)

• Running all test suites in a particular directory. This is likely due to how the tests are started by the IDE in this case, using multiple process invocations (which will not work with my code).
• Processes hang and does not exit cleanly in some cases when tests fail. This is a known but currently unsolved problem with postgres.js. Maybe some introduction of process 'beforeExit' can help with this? If so how should it be implemented in this case?(The root problem is that with postgres.js sql.end() absolutely have to be called otherwise the process is kept waiting https://github.com/porsager/postgres/issues/869.)

---

Tests are started like this, specifying the global setup:

node --test --experimental-test-coverage --test-global-setup=test/global-setup.js 'test/**/*.test.js'

db.js

import postgres from 'postgres'
// TODO: Remove debug code
console.log('db.js hit')
console.log('NODE_TEST_CONTEXT', process.env.NODE_TEST_CONTEXT)
const sql = (process.execArgv.some(substr => substr.startsWith('--test')))
  ? postgres(POSTGRES_URL_TEST, { debug: true, onnotice: () => {/* empty */}})
  : postgres(POSTGRES_URL)

global-setup.js

import sql from './db.js'
import { initDb, emptyDb, isEmptyDb } from './init.js'

/**
 * Runs before all test suites, creates the empty testing database
 * @returns {Promise<void>}
 */
const globalSetup = async () => {
  // Create testing database
  const dbIsEmpty = await isEmptyDb()

  // Check if the database already have tables (if so delete them)
  if (!dbIsEmpty)
    await emptyDb()
  await initDb()
}

/**
 * Runs after all test suites, removes all tables from the testing database and the database connection used
 * @returns {Promise<void>}
 */
const globalTeardown = async () => {
  // Clean up the testing database
  await emptyDb()
  // Clean up the database connection (required for a clean process exit)
  await sql.end()
}

export { globalSetup, globalTeardown }

init.js

import sql from './db.js'

/**
 * Cleans out all tables from database
 * @returns {Promise<void>}
 */
const emptyDb = async () => {
  await sql.begin(sql => [
    sql`DROP SCHEMA public CASCADE`,
    sql`CREATE SCHEMA public`
  ])
}

/**
 * Returns true if the database contains no tables, false otherwise
 * @returns {Promise<boolean>}
 */
const isEmptyDb = async () => {
  const [tableCount] = await sql`
      SELECT count(*)
      FROM information_schema.tables
      WHERE table_schema = 'public'`
  return (tableCount.count === '0')
}


/**
 * Creates the standard database structure
 * @returns {Promise<*>}
 */
const initDb = async () => {
  await sql.begin(sql => [
    /* database creation */
  ])
}

suite-setup.js

import { globalSetup, globalTeardown } from './global-setup.js'

/**
 * Used to set up the database when running individual suites
 * @returns {Promise<void>}
 */
const suiteSetup = async () => {
  // Check if the process was started with the test-global-setup flag, if so database setup is handled globally
  if (!process.execArgv.some(substr => substr.startsWith('--test-global-setup')))
    await globalSetup()
}

/**
 * Used to tear down the database when running individual suites
 * @returns {Promise<void>}
 */
const suiteTeardown = async () => {
  // Check if the process was started with the test-global-setup flag, if so database teardown is handled globally
  if (!process.execArgv.some(substr => substr.startsWith('--test-global-setup')))
    await globalTeardown()
}

export { suiteSetup, suiteTeardown }

function.test.js

import { test, describe, after, before } from 'node:test'

import { suiteSetup, suiteTeardown } from './suite-setup.js'
import sql from './db.js'

describe('Tests function', () => {

  // Sets up things for suite (if needed)
  before(suiteSetup)

  test('function()', { todo: true }, () => {})

  after(async () => {
    // Tear down suite only things (if needed)
    await suiteTeardown()
    // Clean up the database connection (required for a clean process exit)
    await sql.end()
  })
})

[thisalihassan]

I've opened PR #61394 to address this issue.

The PR adds NODE_TEST_WORKER_ID environment variable and context.workerId property, which are set based on the test isolation mode:

• --test-isolation=process (default): Each test file gets a unique worker ID from 1 to N, where N is the concurrency level
• --test-isolation=none: All tests get worker ID 1 (since they run in the same process)

Worker IDs are managed through a pool that allocates and reuses IDs as test files complete, so with 16 test files and concurrency of 8, IDs 1-8 get reused twice.

Why Sequential IDs (1, 2, 3...) vs UUIDs:

I considered using crypto.randomUUID() as suggested in the comments, but went with sequential IDs for consistency with other test frameworks:

• Jest uses JEST_WORKER_ID (sequential)
• Vitest uses VITEST_POOL_ID (sequential)
• Mocha uses MOCHA_WORKER_ID (sequential)

Sequential IDs also make it easier to pre-allocate resources.

This is available at import time, so it works for module-level initialization without needing to pass context around. Looking forward to feedback on the implementation!