temporalio
System Worker fails to authenticate after enabling JWT authorization
Summary
After enabling JWT authorization on Temporal cluster, system workers (like history scanner) fail to authenticate and cannot connect to the frontend service, resulting in "Request unauthorized" errors. Environment
Temporal Version: 1.24.3 Authentication Method: JWT with Keycloak/OIDC Cluster Configuration: Single frontend service with JWT authorization enabled
Problem Description
When JWT authorization is enabled on a Temporal cluster, internal system workers cannot authenticate with the frontend service and fail with authorization errors. This appears to be a common issue where system workers need authentication credentials but there's no clear documentation or configuration method to provide them.
Error Details
{
"level": "error",
"ts": "2025-06-03T01:04:50.770Z",
"msg": "error starting temporal-sys-history-scanner-workflow workflow",
"service": "worker",
"error": "Request unauthorized.",
"logging-call-at": "scanner.go:289",
"stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/worker/scanner.(*Scanner).startWorkflow\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:289\ngo.temporal.io/server/service/worker/scanner.(*Scanner).startWorkflowWithRetry.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:259\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:143\ngo.temporal.io/server/service/worker/scanner.(*Scanner).startWorkflowWithRetry\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:258"
}
Current Configuration
Authorization Configuration
authorization:
jwtKeyProvider:
keySourceURIs:
- https://myapp.domain.com/auth/realms/default/protocol/openid-connect/certs
refreshInterval: "1m"
permissionsClaimName: "permissions"
authorizer: default
claimMapper: default
Services Configuration
services:
frontend:
httpPort: 7243
membershipPort: 6933
port: 7233
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
history:
httpPort: 0
membershipPort: 6934
port: 7234
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
matching:
httpPort: 0
membershipPort: 6935
port: 7235
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
worker:
httpPort: 0
membershipPort: 6939
port: 7239
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
Expected Behavior
System workers should be able to operate normally when JWT authorization is enabled, either by:
Having a built-in bypass mechanism for system operations Supporting configuration of JWT credentials for system workers Clear documentation on how to configure system worker authentication
Actual Behavior
System workers fail with "Request unauthorized" errors and cannot perform essential operations like history scanning, workflow cleanup, etc.
You'll want to use internal-frontend for the easiest integration.
It is mentioned in the release notes here: https://github.com/temporalio/temporal/releases/tag/v1.20.0 under "Internal frontend" It was introduced exactly to support this purpose.
Another option is to use mTLS but this requires integration in the claim mapper, similar to https://github.com/temporalio/samples-server/tree/main/tls/tls-full
@1nu if we don't want to use mTLS for now, the only option is to use Internal frontend deployment.
internalFrontend:
enabled: true
Seems like this is working as intended by using an internal frontend role.
