Features: Batching and graphile-worker upgrade to `0.16.0-rc.1`

[nicktrn]

Closes #396 Closes #348

✅ Checklist

• [x] I have followed every step in the contributing guide
• [x] The PR title follows the convention.
• [x] I ran and tested the code works
• [x] More testing just to be safe
• [x] Create migration docs

---

Testing

Migration

1. Create this docker-compose.yml

version: "3"

volumes:
  data:

networks:
  app_network:
    external: false

x-db-url: &db-url postgresql://postgres:postgres@db:5432/postgres

x-env: &env
  LOGIN_ORIGIN: http://localhost:3030
  APP_ORIGIN: http://localhost:3030
  PORT: 3030
  REMIX_APP_PORT: 3030
  MAGIC_LINK_SECRET: secret
  SESSION_SECRET: secret
  ENCRYPTION_KEY: ae13021afef0819c3a307ad487071c06
  DATABASE_URL: &db-url
  DIRECT_URL: &db-url
  GRACEFUL_SHUTDOWN_TIMEOUT: 5000
  NODE_ENV: production
  RUNTIME_PLATFORM: docker-compose
  TRIGGER_LOG_LEVEL: debug

services:
  old-webapp:
    image: ghcr.io/triggerdotdev/trigger.dev:v2.2.21
    restart: "no"
    environment:
      <<: *env
      WORKER_ENABLED: false
      EXECUTION_WORKER_ENABLED: false
      TASK_OPERATION_WORKER_ENABLED: false
    ports:
      - 3065:3030
    depends_on:
      - db
    networks:
      - app_network

  old-workers:
    image: ghcr.io/triggerdotdev/trigger.dev:v2.2.21
    restart: "no"
    environment:
      <<: *env
      HTTP_SERVER_DISABLED: true
    depends_on:
      - db
    networks:
      - app_network

  new-webapp:
    image: ghcr.io/triggerdotdev/trigger.dev:batching-rc.2
    restart: always
    environment:
      <<: *env
      WORKER_ENABLED: false
      EXECUTION_WORKER_ENABLED: false
      TASK_OPERATION_WORKER_ENABLED: false
    ports:
      - 3030:3030
    depends_on:
      - db
    networks:
      - app_network

  new-workers:
    image: ghcr.io/triggerdotdev/trigger.dev:batching-rc.2
    restart: always
    environment:
      <<: *env
      HTTP_SERVER_DISABLED: true
    depends_on:
      - db
    networks:
      - app_network
      
  db:
    image: postgres:14
    restart: always
    volumes:
      - data:/var/lib/postgresql/data/
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres
    networks:
      - app_network
    ports:
      - 3085:5432

2. Start up the DB

docker-compose up -d db

3. Start up the old webapp and workers

docker-compose up old-webapp old-workers

4. In another window, start up the new webapp and workers

docker-compose up new-webapp new-workers

5. To delete the DB and start again, maybe after altering some env vars

docker-compose down -v

Batching

https://github.com/triggerdotdev/trigger.dev/blob/2afa19995e1cee973d25a278256e66e667f304b2/references/job-catalog/src/events.ts#L156-L211

---

Changelog

• Upgrade to v0.16.0-rc.1
• Detect migration errors and point to docs
• Add migration delay to allow graceful shutdown to complete (rolling deploy)
• Insert custom add_batch_job function
• Add batched event triggers and webhook delivery
• Read replica support via optional DATABASE_READ_REPLICA_URL env var

---

Screenshots

💯

[changeset-bot[bot]]

🦋 Changeset detected

Latest commit: f130fbd67c2e9145d02d44662b3156eb03b3bbf8

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[nicktrn]

~~0.15.0 was released yesterday. The only change being that migration files are now embedded as strings. Just in case of unforeseen bugs, we should stay on 0.14.0-rc.0 for now.~~

[nicktrn]

Waiting for v0.16 release and #689 before finishing this up.

[matt-aitken]

We got the key Graphile updates in already, and since v2 is EOL in January I don't think this makes sense anymore.