Connection refused during Nexus Operation for Internal Frontend HTTP Port (7246)

[dran-dev]

Expected Behavior

When attempting to use Temporal Nexus endpoints for cross-namespace communication, requests fail because the internal frontend service is not listening on the expected HTTP port (7246). This occurs despite having Nexus enabled in the configuration.

Actual Behavior

{"message":"Post \"http://10.1.251.74:7246/nexus/endpoints/4ffdb2f8-5829-47fb-9242-8c353a730b64/services/testnexusservice/test-operation?callback=http%3A%2F%2Flocalhost:%3A7243%2Fnamespaces%2Ftest-namespace%2Fnexus%2Fcallback\": dial tcp 10.1.251.74:7246: connect: connection refused","applicationFailureInfo":{"type":"CallError"}}

Steps to Reproduce the Problem

1. created a temporal nexus endpoint for cross namespace call.

$ temporal operator nexus endpoint list
  ID                       4ffdb2f8-5829-47fb-9242-8c353a730b64
  Name                     testnexus-endpoint
  CreatedTime              <nil>
  LastModifiedTime         "2025-02-25T19:45:20.784Z"
  Target.External.URL
  Target.Worker.Namespace  test-worker-namespace
  Target.Worker.TaskQueue  test-task-queue
  Description

2. The internal frontend pod only listens on these ports:

$ netstat -plant | grep LISTEN
tcp        0      0 127.0.0.1:7936          0.0.0.0:*               LISTEN      1/temporal-server
tcp        0      0 :::9090                 :::*                    LISTEN      1/temporal-server
tcp        0      0 :::6936                 :::*                    LISTEN      1/temporal-server
tcp        0      0 :::7236                 :::*                    LISTEN      1/temporal-server

3. relevant config

services:
  frontend:
    rpc:
      grpcPort: 7233
      httpPort: 7243
      membershipPort: 6933
      bindOnIP: "0.0.0.0"
  internal-frontend:
    rpc:
      grpcPort: 7236
      httpPort: 7246
      membershipPort: 6936
      bindOnIP: "0.0.0.0"

4. dynamicConfig

dynamicConfig:
  system.enableNexus: true
  component.nexusoperations.callback.endpoint.template: "http://localhost:7243/namespaces/{{.NamespaceName}}/nexus/callback"

Specifications

• Version:
  • temporal-server - 1.26.2, 1.27.0
  • helm chart - v0.55.0, v0.56.0
• Platform: Kubernetes

Is there additional configuration needed to enable the HTTP port (7246) on the internal frontend? Is this a known issue with the current version? Are there any workarounds available?

Let me know if you need any additional information to help investigate this issue.

[yycptt]

cc @rodrigozhou @pdoerner Can you help take a look here?

[vitornp]

I have the same problem

[bergundy]

We are taking a look at this, thanks for reporting.