headplane icon indicating copy to clipboard operation
headplane copied to clipboard
verified publisher icon tale

Headplane agent exited with code undefined and signal SIGTERM

Open spatterIight opened this issue 6 months ago • 2 comments

Description

Hello !

I am having an issue with the Headplane agent, I am not sure why but it seems its heartbeat is timing out causing it to exit. Do you know why this may be?

Here is my configuration:

headscale:
  config_path: /etc/headscale/config.yaml
  config_strict: true
  dns_records_path: /var/lib/headplane/extra_records.json
  url: http://mash-headscale:8080
integration:
  agent:
    enabled: true
    pre_authkey: REDACTED
  docker:
    container_name: mash-headscale
    enabled: false
    socket: unix:///var/run/docker.sock
server:
  cookie_secret: REDACTED
  cookie_secure: true
  data_path: /var/lib/headplane
  host: 0.0.0.0
  port: 3000

Debug log:

Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.493Z [agent] INFO: Headplane agent started
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.497Z [agent] INFO: Enabling Debug logging for headplane-agent
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.497Z [agent] INFO: Be careful, this will spam a lot of information
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.497Z [agent] DEBUG: Received directive: START
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.498Z [agent] DEBUG: [v1] using fake (no-op) tun device
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.498Z [agent] DEBUG: [v1] using fake (no-op) OS network configurator
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.498Z [agent] DEBUG: [v1] using fake (no-op) DNS configurator
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.498Z [agent] DEBUG: dns: using dns.noopManager
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.498Z [agent] DEBUG: link state: interfaces.State{defaultRoute=eth1 ifs={eth0:[172.16.6.130/25] eth1:[172.16.0.130/25] eth2:[192.168.16.18/20]} v4=true v6=false}
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.498Z [agent] DEBUG: onPortUpdate(port=48901, network=udp6)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.498Z [agent] DEBUG: [v1] warning: fakeRouter.UpdateMagicsockPort: not implemented.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: magicsock: [warning] failed to force-set UDP read buffer size to 7340032: operation not permitted; using kernel default values (impacts throughput only)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: magicsock: [warning] failed to force-set UDP write buffer size to 7340032: operation not permitted; using kernel default values (impacts throughput only)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: onPortUpdate(port=51702, network=udp4)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: [v1] warning: fakeRouter.UpdateMagicsockPort: not implemented.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: magicsock: [warning] failed to force-set UDP read buffer size to 7340032: operation not permitted; using kernel default values (impacts throughput only)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: magicsock: [warning] failed to force-set UDP write buffer size to 7340032: operation not permitted; using kernel default values (impacts throughput only)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: [v1] magicsock: peermtu: peer MTU status is false
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: magicsock: disco key = d:18726543cd264ad4
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: Creating WireGuard device...
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: Bringing WireGuard device up...
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: wg: [v2] UDP bind has been updated
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: wg: [v2] Interface state was Down, requested Up, now Up
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: Bringing router up...
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: [v1] warning: fakeRouter.Up: not implemented.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: Clearing router settings...
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: [v1] warning: fakeRouter.Set: not implemented.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: Starting network monitor...
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: Engine created.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] INFO: tsnet running state path /var/lib/headplane/agent/tailscaled.state
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.499Z [agent] DEBUG: pm: migrating "_daemon" profile to new format
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.500Z [agent] DEBUG: logpolicy: using UserCacheDir, "/.cache/Tailscale"
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.500Z [agent] DEBUG: [v1] netmap packet filter: (not ready yet)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.500Z [agent] INFO: tsnet starting with hostname "headplane-agent", varRoot "/var/lib/headplane/agent"
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.500Z [agent] DEBUG: Start
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.500Z [agent] DEBUG: ipnext: active extensions:
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.500Z [agent] DEBUG: [v
Oct 28 13:52:02 smirnoff mash-headplane[480975]: JSON]1{"Hostinfo":{"IPNVersion":"1.88.2-ERR-BuildInfo","BackendLogID":"c7cf66838e3604293121d457241fa6de4831b581aabaa0c7ac1221e67392c979","OS":"linux","OSVersion":"6.12.55-1-lts","Container":true,"Distro":"debian","DistroVersion":"12.12","Desktop":false,"Package":"tsnet","Hostname":"4dc181f7ee3c","Machine":"x86_64","GoArch":"amd64","GoArchVar":"v1","GoVersion":"go1.25.1","Userspace":true,"UserspaceRouter":true,"AppConnector":false,"StateEncrypted":false}}
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.501Z [agent] DEBUG: wg: [v2] Routine: receive incoming mkReceiveFunc - started
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.501Z [agent] DEBUG: control: [v1] HostInfo: {"IPNVersion":"1.88.2-ERR-BuildInfo","BackendLogID":"c7cf66838e3604293121d457241fa6de4831b581aabaa0c7ac1221e67392c979","OS":"linux","OSVersion":"6.12.55-1-lts","Container":true,"Distro":"debian","DistroVersion":"12.12","Desktop":false,"Package":"tsnet","Hostname":"headplane-agent","Machine":"x86_64","GoArch":"amd64","GoArchVar":"v1","GoVersion":"go1.25.1","Userspace":true,"UserspaceRouter":true,"AppConnector":false,"StateEncrypted":false}
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.501Z [agent] DEBUG: Backend: logs: be:c7cf66838e3604293121d457241fa6de4831b581aabaa0c7ac1221e67392c979 fe:
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.501Z [agent] DEBUG: Switching ipn state NoState -> NeedsLogin (WantRunning=true, nm=false)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.501Z [agent] DEBUG: blockEngineUpdates(true)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.501Z [agent] DEBUG: wg: [v2] Routine: receive incoming receiveDERP - started
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.501Z [agent] DEBUG: control: [v1] authRoutine: state:new; goal=nil paused=false
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: control: [v1] mapRoutine: state:new
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: wg: [v2] Routine: receive incoming mkReceiveFunc - started
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: [v1] magicsock: peermtu: peer MTU status is false
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: wgengine: Reconfig: configuring userspace WireGuard config (with 0/0 peers)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: wgengine: Reconfig: configuring router
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: [v1] warning: fakeRouter.Set: not implemented.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: wgengine: Reconfig: user dialer
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: wgengine: Reconfig: configuring DNS
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: dns: Set: {DefaultResolvers:[] Routes:{} SearchDomains:[] Hosts:0}
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: dns: Resolvercfg: {Routes:{} Hosts:0 LocalDomains:[]}
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: dns: OScfg: {}
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: [v1] wgengine: Reconfig done
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] INFO: LocalBackend state is NeedsLogin; running StartLoginInteractive...
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: StartLoginInteractiveAs(""): url=false
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.502Z [agent] DEBUG: control: client.Login(10)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: control: [v1] mapRoutine: context done.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: control: [v1] mapRoutine: state:new
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: control: [v1] authRoutine: context done.
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: control: [v1] authRoutine: state:new; wantLoggedIn=true
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: control: [v1] direct.TryLogin(flags=10)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: control: LoginInteractive -> regen=true
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: control: doLogin(regen=true, hasUrl=false)
Oct 28 13:52:02 smirnoff mash-headplane[480975]: 2025-10-28T17:52:02.503Z [agent] DEBUG: health(warnable=warming-up): error: Tailscale is starting. Please wait.
Oct 28 13:52:07 smirnoff mash-headplane[480975]: 2025-10-28T17:52:07.504Z [agent] DEBUG: health(warnable=warming-up): ok
Oct 28 13:52:08 smirnoff mash-headplane[480975]: 2025-10-28T17:52:08.243Z [api] DEBUG: GET /api/v1/node
Oct 28 13:52:08 smirnoff mash-headplane[480975]: 2025-10-28T17:52:08.243Z [api] DEBUG: GET /api/v1/user
Oct 28 13:52:08 smirnoff mash-headplane[480975]: 2025-10-28T17:52:08.246Z [api] DEBUG: No ephemeral nodes to prune
Oct 28 13:52:08 smirnoff mash-headplane[480975]: 2025-10-28T17:52:08.246Z [api] DEBUG: GET /api/v1/apikey
Oct 28 13:52:10 smirnoff mash-headplane[480975]: 2025-10-28T17:52:10.509Z [agent] WARN: Headplane agent experienced an error: Agent heartbeat timeout
Oct 28 13:52:10 smirnoff mash-headplane[480975]: 2025-10-28T17:52:10.509Z [agent] DEBUG: Error details: Error: Agent heartbeat timeout
Oct 28 13:52:10 smirnoff mash-headplane[480975]:     at Timeout._onTimeout (file:///app/build/server/assets/hp-agent-B6M0PQ5u.js:203:41)
Oct 28 13:52:10 smirnoff mash-headplane[480975]:     at listOnTimeout (node:internal/timers:588:17)
Oct 28 13:52:10 smirnoff mash-headplane[480975]:     at process.processTimers (node:internal/timers:523:7) {
Oct 28 13:52:10 smirnoff mash-headplane[480975]:   [stack]: [Getter/Setter],
Oct 28 13:52:10 smirnoff mash-headplane[480975]:   [message]: 'Agent heartbeat timeout'
Oct 28 13:52:10 smirnoff mash-headplane[480975]: }
Oct 28 13:52:10 smirnoff mash-headplane[480975]: 2025-10-28T17:52:10.512Z [agent] WARN: Headplane agent exited with code undefined and signal SIGTERM
Oct 28 13:52:10 smirnoff mash-headplane[480975]: 2025-10-28T17:52:10.512Z [agent] WARN: Headplane agent will restart in 17.722 seconds (attempt 5)

SystemD service:

File: /etc/systemd/system/mash-headplane.service

[Unit]
Description=Headplane (mash-headplane)
Requires=docker.service
After=docker.service
Requires=mash-headscale.service
After=mash-headscale.service

[Service]
Type=simple
Environment="HOME=/root"
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker stop -t 30 mash-headplane 2>/dev/null || true'
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker rm mash-headplane 2>/dev/null || true'

ExecStartPre=/usr/bin/env docker create \
      --rm \
      --name=mash-headplane \
      --log-driver=none \
      --user=942:1001 \
      --cap-drop=ALL \
      --read-only \
      --network=mash-headplane \
            --env-file=/smirnoff/mash/headplane/env \
      --label-file=/smirnoff/mash/headplane/labels \
            --mount type=bind,src=/smirnoff/mash/headscale/config/config.yaml,dst=/etc/headscale/config.yaml \
            --mount type=bind,src=/smirnoff/mash/headplane/config.yaml,dst=/etc/headplane/config.yaml \
      --mount type=bind,src=/smirnoff/mash/headplane/data,dst=/var/lib/headplane \
            --mount type=bind,src=/smirnoff/mash/headscale/data/extra-records.json,dst=/var/lib/headplane/extra_records.json \
                        ghcr.io/tale/headplane:0.6.1

ExecStartPre=/usr/bin/env docker network connect traefik mash-headplane
ExecStartPre=/usr/bin/env docker network connect mash-headscale mash-headplane

ExecStart=/usr/bin/env docker start --attach mash-headplane

ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker stop -t 30 mash-headplane 2>/dev/null || true'
ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker rm mash-headplane 2>/dev/null || true'

Restart=always
RestartSec=30
SyslogIdentifier=mash-headplane

[Install]
WantedBy=multi-user.target

Headplane Version

0.6.1

Headscale Version

v0.27.0

spatterIight avatar Oct 28 '25 17:10 spatterIight

It seems like the issue is running the container with tightened security, I.E. running the container as a non-root user and dropping capabilities (It does not seem like running as read-only affects this error).

Only when both of these are disabled does the agent connect.

It would be ideal if it was not necessary, since running containers with these hardened settings is a best practice.

Thanks!

spatterIight avatar Oct 28 '25 18:10 spatterIight

Adding the following capability resolves the error when running as root: --cap-add=NET_RAW

This allows you run as root with all capabilities dropped, except adding back that one ^

Unfortunately adding this capability when running as a non-root user does not work, so running as root is still necessary...

spatterIight avatar Oct 28 '25 19:10 spatterIight