carbonapi icon indicating copy to clipboard operation
carbonapi copied to clipboard
verified publisher icon go-graphite

[BUG] Reloading via systemctl kills carbonapi process

Open ryangsteele opened this issue 5 years ago • 3 comments

Describe the bug Upon issuing a systemctl reload carbonapi.service, the carbonapi process dies instead of being gracefully restarted.

CarbonAPI Version 0.14.1-1 (latest stable Debian package from packagecloud)

Logs Running journalctl -u carbonapi.service after a systemctl reload carbonapi.service yields the following errors:

Jan 12 19:37:43 carbonapi.test.host systemd[1]: Reloading CarbonAPI - replacement for graphite-web written in Go.
Jan 12 19:37:43 carbonapi.test.host systemd[1]: Reloaded CarbonAPI - replacement for graphite-web written in Go.
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.426Z        INFO        main        will parse config as yaml        {"config_file": "/etc/carbonapi/carbonapi.yaml"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.426Z        INFO        main        will parse config as yaml        {"config_file": "/etc/carbonapi/carbonapi.yaml"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        cache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        backendCache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        cache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.433Z        INFO        main        backendCache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        starting carbonapi        {"build_version": "0.14.1", "config": "{\"ExtrapolateExperiment\":false,\"Logger\":[{\"logger\":\"\",\"file\":\"/var/log/carbonapi/carbonapi.log\",\"level\":\"error\",\"encoding\":\"console\",\"encoding-time\":\"iso8601
\",\"encoding-duration\":\"seconds\",\"sample-tick\":\"\",\"sample-initial\":0,\"sample-thereafter\":0}],\"Listen\":\"0.0.0.0:8082\",\"Buckets\":10,\"Concurency\":1000,\"ResponseCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":[],\"DefaultTimeoutSec\":60},\"BackendCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":null,\"DefaultTimeoutS
ec\":60},\"Cpus\":0,\"TimezoneString\":\"\",\"UnicodeRangeTables\":null,\"Graphite\":{\"Pattern\":\"{prefix}.{fqdn}\",\"Host\":\"127.0.0.1:2003\",\"Interval\":60000000000,\"Prefix\":\"carbon.api\"},\"IdleConnections\":1000,\"PidFile\":\"/var/run/carbonapi/carbonapi.pid\",\"SendGlobsAsIs\":null,\"AlwaysSendGlobsAsIs\":null,\"MaxBatchSize\":0,\"Zipper\":\"\",\"Upstr
eams\":{\"ConcurrencyLimitPerServer\":0,\"MaxIdleConnsPerHost\":100,\"Backends\":null,\"BackendsV2\":{\"Backends\":[{\"GroupName\":\"go-carbon\",\"Protocol\":\"carbonapi_v3_pb\",\"LBMethod\":\"broadcast\",\"Servers\":[\"http://127.0.0.1:8081\"],\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"ConcurrencyLimit\":0,\"KeepAliveInterva
l\":30000000000,\"MaxIdleConnsPerHost\":100,\"MaxTries\":3,\"MaxBatchSize\":0,\"BackendOptions\":null,\"ForceAttemptHTTP2\":false,\"DoMultipleRequestsIfSplit\":false}],\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0},\
"MaxBatchSize\":0,\"FallbackMaxBatchSize\":0,\"MaxTries\":0,\"DoMultipleRequestsIfSplit\":false,\"CarbonSearch\":{\"Backend\":\"\",\"Prefix\":\"virt.v1.*\"},\"CarbonSearchV2\":{\"Backends\":null,\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":0,\"Render\":0,\"Connect\":0},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0,\"
Prefix\":\"\"},\"ExpireDelaySec\":0,\"TLDCacheDisabled\":true,\"InternalRoutingCache\":600000000000,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":30000000000,\"ScaleToCommonStep\":true},\"ExpireDelaySec\":600,\"GraphiteWeb09Compatibility\":false,\"IgnoreClientTimeout\":false,\"DefaultColors\":null,\"GraphTemp
lates\":\"\",\"FunctionsConfigs\":{},\"HeadersToPass\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"HeadersToLog\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"Define\":null,\"Prefix\":\"\",\"Expvar\":{\"Listen\":\"\",\"Enabled\":true,\"PProfEnabled\":false},\"NotFoundStatusCode\":200,\"HTTPResponseStackTrace\":true}"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.434Z        INFO        main        expvar handler will listen on a separate address/port        {"expvar_listen": "", "pprof_enabled": false}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.434Z        FATAL        main        failed to start http server        {"error": "error inheriting socket fd 3: file file+net listener: fcntl: bad file descriptor"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.435Z        INFO        main        starting carbonapi        {"build_version": "0.14.1", "config": "{\"ExtrapolateExperiment\":false,\"Logger\":[{\"logger\":\"\",\"file\":\"/var/log/carbonapi/carbonapi.log\",\"level\":\"error\",\"encoding\":\"console\",\"encoding-time\":\"iso8601
\",\"encoding-duration\":\"seconds\",\"sample-tick\":\"\",\"sample-initial\":0,\"sample-thereafter\":0}],\"Listen\":\"0.0.0.0:8082\",\"Buckets\":10,\"Concurency\":1000,\"ResponseCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":[],\"DefaultTimeoutSec\":60},\"BackendCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":null,\"DefaultTimeoutS
ec\":60},\"Cpus\":0,\"TimezoneString\":\"\",\"UnicodeRangeTables\":null,\"Graphite\":{\"Pattern\":\"{prefix}.{fqdn}\",\"Host\":\"127.0.0.1:2003\",\"Interval\":60000000000,\"Prefix\":\"carbon.api\"},\"IdleConnections\":1000,\"PidFile\":\"/var/run/carbonapi/carbonapi.pid\",\"SendGlobsAsIs\":null,\"AlwaysSendGlobsAsIs\":null,\"MaxBatchSize\":0,\"Zipper\":\"\",\"Upstr
eams\":{\"ConcurrencyLimitPerServer\":0,\"MaxIdleConnsPerHost\":100,\"Backends\":null,\"BackendsV2\":{\"Backends\":[{\"GroupName\":\"go-carbon\",\"Protocol\":\"carbonapi_v3_pb\",\"LBMethod\":\"broadcast\",\"Servers\":[\"http://127.0.0.1:8081\"],\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"ConcurrencyLimit\":0,\"KeepAliveInterva
l\":30000000000,\"MaxIdleConnsPerHost\":100,\"MaxTries\":3,\"MaxBatchSize\":0,\"BackendOptions\":null,\"ForceAttemptHTTP2\":false,\"DoMultipleRequestsIfSplit\":false}],\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0},\
"MaxBatchSize\":0,\"FallbackMaxBatchSize\":0,\"MaxTries\":0,\"DoMultipleRequestsIfSplit\":false,\"CarbonSearch\":{\"Backend\":\"\",\"Prefix\":\"virt.v1.*\"},\"CarbonSearchV2\":{\"Backends\":null,\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":0,\"Render\":0,\"Connect\":0},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0,\"
Prefix\":\"\"},\"ExpireDelaySec\":0,\"TLDCacheDisabled\":true,\"InternalRoutingCache\":600000000000,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":30000000000,\"ScaleToCommonStep\":true},\"ExpireDelaySec\":600,\"GraphiteWeb09Compatibility\":false,\"IgnoreClientTimeout\":false,\"DefaultColors\":null,\"GraphTemp
lates\":\"\",\"FunctionsConfigs\":{},\"HeadersToPass\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"HeadersToLog\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"Define\":null,\"Prefix\":\"\",\"Expvar\":{\"Listen\":\"\",\"Enabled\":true,\"PProfEnabled\":false},\"NotFoundStatusCode\":200,\"HTTPResponseStackTrace\":true}"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.436Z        INFO        main        expvar handler will listen on a separate address/port        {"expvar_listen": "", "pprof_enabled": false}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.437Z        FATAL        main        failed to start http server        {"error": "error inheriting socket fd 3: file file+net listener: fcntl: bad file descriptor"}

CarbonAPI Configuration: Note: I have tried with pidFile set to the empty string ("") as well as having it set to the same pidfile defined in /etc/systemd/system/carbonapi.service (as suggested in #426), the latter of which is shown below:

listen: "0.0.0.0:8082"
pidFile: "/var/run/carbonapi/carbonapi.pid"
prefix: ""
expvar:
  enabled: true
  pprofEnabled: false
  listen: ""
graphite:
    host: "127.0.0.1:2003"
    interval: "60s"
    prefix: "carbon.api"
    pattern: "{prefix}.{fqdn}"
headersToPass:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
headersToLog:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
notFoundStatusCode: 200
concurency: 1000
cache:
   type: "mem"
   size_mb: 0
   defaultTimeoutSec: 60
backendCache:
   type: "mem"
   size_mb: 0
   defaultTimeoutSec: 60
cpus: 0
tz: ""
maxBatchSize: 0
idleConnections: 1000
upstreams:
    tldCacheDisabled: true
    buckets: 10
    timeouts:
        find: "2s"
        render: "10s"
        connect: "200ms"
    concurrencyLimitPerServer: 0
    keepAliveInterval: "30s"
    maxIdleConnsPerHost: 100
    doMultipleRequestsIfSplit: false
    backendsv2:
        backends:
          -
            groupName: "go-carbon"
            protocol: "carbonapi_v3_pb"
            lbMethod: "broadcast"
            maxTries: 3
            maxBatchSize: 0
            keepAliveInterval: "30s"
            concurrencyLimit: 0
            maxIdleConnsPerHost: 100
            forceAttemptHTTP2: false
            doMultipleRequestsIfSplit: false
            timeouts:
                find: "2s"
                render: "10s"
                connect: "200ms"
            servers:
                - "http://127.0.0.1:8081"
    graphite09compat: false
logger:
    - logger: ""
      file: "/var/log/carbonapi/carbonapi.log"
      level: "error"
      encoding: "console"
      encodingTime: "iso8601"
      encodingDuration: "seconds"

Simplified query (if applicable) N/A

Backend metric retention and aggregation schemas N/A

Backend response (if possible) N/A

Additional context Contents of /etc/systemd/system/carbonapi.service:

[Unit]
Description=CarbonAPI - replacement for graphite-web written in Go
After=syslog.target network.target

[Service]
Type=simple
EnvironmentFile=/etc/default/carbonapi
ExecStart=/usr/bin/carbonapi $CARBONAPI_ARGS
ExecReload=/bin/kill -USR2 $MAINPID
PIDFile=/var/run/carbonapi/carbonapi.pid
LimitNOFILE=200000
RuntimeDirectory=carbonapi
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

Contents of /etc/default/carbonapi:

CARBONAPI_ARGS="-config /etc/carbonapi/carbonapi.yaml"

If there is no desire to support reloads going forward, I can remove that capability as part of https://github.com/go-graphite/carbonapi/issues/551.

ryangsteele avatar Jan 12 '21 20:01 ryangsteele

Last time I've used it was about January 2018 I think. Back then it was working, however it is possible that it no longer works in recent versions of Go or in some specific combinations of Go and kernel (library that was used to support that functionality is archives by authors, without any reasons specified).

However no one ever complained before but I don't know if that's because no one used it or it's working in most of the cases.

I'll try to have a look and at least reproduce it. (but no promise on a timeline though, sorry).

Civil avatar Jan 12 '21 21:01 Civil

No worries @Civil - for now I can just use restart upon config changes.

I'll try and dig in to this more as time permits too. The kernels I tested on were all in the 4.15.0 branch (4.15.0-112.113 and 4.15.0-130.134). If there's any other information I've left out that would be useful, just let me know.

Thanks for the prompt reply!

ryangsteele avatar Jan 13 '21 16:01 ryangsteele

As reload feature was removed since the bug was open, I'll reclassify it as "enhancement" as I'm open to re-introduce that back, if there will be a reasonable way to support reload (by reasonable - that will still compile on OSs that doesn't have systemd, for example, as I think another way is to use sd-* helpers there and bindings for systemd, but I'm not sure it's easy to make that optional)

Civil avatar May 18 '21 00:05 Civil