dkron
dkron copied to clipboard
distribworks
No leader
My cluster consists of 3 server node (Dkron 3.1.8):
As you can see, there is no leader.
All nodes rolled up with this command:
docker run \
--name $INSTANCE_CONTAINER_NAME \
--hostname $INSTANCE_HOST_NAME \
--detach \
--volume /etc/localtime:/etc/localtime:ro --volume /etc/timezone:/etc/timezone:ro \
--volume $INSTANCE_CONTAINER_NAME-backup:/var/lib/dkron/backup \
--volume $INSTANCE_CONTAINER_NAME-data:/var/lib/dkron/data \
--volume $INSTANCE_CONTAINER_NAME-log:/var/log/dkron \
--publish $INSTANCE_HTTP_PORT:8080 \
--publish $INSTANCE_SERF_PORT:$INSTANCE_SERF_PORT \
--publish $INSTANCE_SERF_PORT:$INSTANCE_SERF_PORT/udp \
--publish $INSTANCE_GRPC_PORT:$INSTANCE_GRPC_PORT \
$DEFAULT_GO_SETTINGS \
$DEFAULT_HEALTH_SETTINGS \
$DEFAULT_LOG_SETTINGS \
$IMAGE_NAME:$VERSION \
agent \
--server \
--bind-addr "{{ GetPrivateIP }}":$INSTANCE_SERF_PORT \
--advertise-addr host.backpack.test:$INSTANCE_SERF_PORT \
--rpc-port $INSTANCE_GRPC_PORT \
--retry-join $SERVER_1_HOST_NAME:$SERVER_1_SERF_PORT \
--retry-join $SERVER_2_HOST_NAME:$SERVER_2_SERF_PORT \
--retry-join $SERVER_3_HOST_NAME:$SERVER_3_SERF_PORT \
--retry-interval 15s \
$INSTANCE_BOOTSTRAP \
--region test \
--datacenter backpack \
--node-name $INSTANCE_NODE_NAME \
--tag group=servers \
--tag host=$INSTANCE_HOST_NAME \
--data-dir /var/lib/dkron/data \
--log-level debug
Logs from server-1:
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-files]" node=server-1 path=/dkron/dkron-processor-files
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-processor-files pid=12
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-processor-files
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin223279704 network=unix node=server-1 subsystem_name=plugins.dkron-processor-files timestamp="2021-06-21T21:53:59.473+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-fluent]" node=server-1 path=/dkron/dkron-processor-fluent
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-processor-fluent pid=19
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-processor-fluent
time="2021-06-21T21:53:59+03:00" level=debug msg="time=\"2021-06-21T21:53:59+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_HOST\"" node=server-1 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:53:59+03:00" level=debug msg="time=\"2021-06-21T21:53:59+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_PORT\"" node=server-1 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:53:59+03:00" level=debug msg="time=\"2021-06-21T21:53:59+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_TAG\"" node=server-1 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin100965216 network=unix node=server-1 subsystem_name=plugins.dkron-processor-fluent timestamp="2021-06-21T21:53:59.524+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-log]" node=server-1 path=/dkron/dkron-processor-log
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-processor-log pid=26
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-processor-log
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin512259449 network=unix node=server-1 subsystem_name=plugins.dkron-processor-log timestamp="2021-06-21T21:53:59.536+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-syslog]" node=server-1 path=/dkron/dkron-processor-syslog
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-processor-syslog pid=33
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-processor-syslog
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin975111997 network=unix node=server-1 subsystem_name=plugins.dkron-processor-syslog timestamp="2021-06-21T21:53:59.544+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-gcppubsub]" node=server-1 path=/dkron/dkron-executor-gcppubsub
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-executor-gcppubsub pid=40
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-executor-gcppubsub
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin874004440 network=unix node=server-1 subsystem_name=plugins.dkron-executor-gcppubsub timestamp="2021-06-21T21:53:59.561+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-http]" node=server-1 path=/dkron/dkron-executor-http
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-executor-http pid=46
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-executor-http
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin292889508 network=unix node=server-1 subsystem_name=plugins.dkron-executor-http timestamp="2021-06-21T21:53:59.582+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-kafka]" node=server-1 path=/dkron/dkron-executor-kafka
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-executor-kafka pid=54
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-executor-kafka
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin123245755 network=unix node=server-1 subsystem_name=plugins.dkron-executor-kafka timestamp="2021-06-21T21:53:59.592+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-nats]" node=server-1 path=/dkron/dkron-executor-nats
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-executor-nats pid=61
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-executor-nats
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin995465403 network=unix node=server-1 subsystem_name=plugins.dkron-executor-nats timestamp="2021-06-21T21:53:59.599+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-rabbitmq]" node=server-1 path=/dkron/dkron-executor-rabbitmq
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-executor-rabbitmq pid=68
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-executor-rabbitmq
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin601745816 network=unix node=server-1 subsystem_name=plugins.dkron-executor-rabbitmq timestamp="2021-06-21T21:53:59.608+0300"
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-shell]" node=server-1 path=/dkron/dkron-executor-shell
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin started" node=server-1 path=/dkron/dkron-executor-shell pid=75
time="2021-06-21T21:53:59+03:00" level=debug msg="waiting for RPC address" node=server-1 path=/dkron/dkron-executor-shell
time="2021-06-21T21:53:59+03:00" level=debug msg="using plugin" node=server-1 version=1
time="2021-06-21T21:53:59+03:00" level=debug msg="plugin address" address=/tmp/plugin331092741 network=unix node=server-1 subsystem_name=plugins.dkron-executor-shell timestamp="2021-06-21T21:53:59.619+0300"
time="2021-06-21T21:53:59+03:00" level=info msg="agent: Dkron agent starting" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="2021/06/21 21:53:59 [INFO] serf: EventMemberJoin: server-1 192.168.1.20"
time="2021-06-21T21:53:59+03:00" level=info msg="agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="agent: Joining cluster..." cluster=LAN node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="2021/06/21 21:53:59 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.4:8901"
time="2021-06-21T21:53:59+03:00" level=info msg="2021/06/21 21:53:59 [DEBUG] memberlist: Stream connection from=172.17.0.4:58982"
time="2021-06-21T21:53:59+03:00" level=info msg="2021/06/21 21:53:59 [DEBUG] memberlist: Failed to join 192.168.1.20: dial tcp 192.168.1.20:8904: connect: connection refused"
time="2021-06-21T21:53:59+03:00" level=info msg="2021/06/21 21:53:59 [DEBUG] memberlist: Failed to join 192.168.1.20: dial tcp 192.168.1.20:8907: connect: connection refused"
time="2021-06-21T21:53:59+03:00" level=info msg="agent: Join LAN completed. Synced with 1 initial agents" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached." node=server-1
time="2021-06-21T21:53:59+03:00" level=info node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] [WARNING] Running in \"debug\" mode. Switch to \"release\" mode in production." node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg=" - using env:\texport GIN_MODE=release" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg=" - using code:\tgin.SetMode(gin.ReleaseMode)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /debug/vars --> github.com/gin-contrib/expvar.Handler.func1 (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /health --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).APIRoutes.func1 (4 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1 --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/ --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/members --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).membersHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/leader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaderHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/isleader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).isLeaderHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] POST /v1/leave --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaveHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] POST /v1/restore --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).restoreHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/busy --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).busyHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] POST /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] PATCH /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobsHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] DELETE /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobDeleteHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job/run --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job/toggle --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobToggleHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] PUT /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobGetHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /v1/jobs/:job/executions --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).executionsHandler-fm (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET / --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func1 (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] [WARNING] Since SetHTMLTemplate() is NOT thread-safe. It should only be called" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="at initialization. ie. before any route is registered or the router is listening in a socket:" node=server-1
time="2021-06-21T21:53:59+03:00" level=info node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="\trouter := gin.Default()" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="\trouter.SetHTMLTemplate(template) // << good place" node=server-1
time="2021-06-21T21:53:59+03:00" level=info node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="api: Running HTTP server" address=":8080" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] GET /ui/*filepath --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func2 (6 handlers)" node=server-1
time="2021-06-21T21:53:59+03:00" level=info msg="[GIN-debug] Listening and serving HTTP on :8080" node=server-1
time="2021-06-21T21:54:00+03:00" level=info msg="2021-06-21T21:54:00.859+0300 [INFO] raft: initial configuration: index=1 servers=\"[{Suffrage:Voter ID:server-1 Address:172.17.0.4:8902}]\""
time="2021-06-21T21:54:00+03:00" level=info msg="2021-06-21T21:54:00.860+0300 [INFO] raft: entering follower state: follower=\"Node at 172.17.0.4:8902 [Follower]\" leader="
time="2021-06-21T21:54:00+03:00" level=info msg="dkron: monitoring leadership" node=server-1
time="2021-06-21T21:54:00+03:00" level=info msg="2021/06/21 21:54:00 [INFO] serf: EventMemberUpdate: server-1"
time="2021-06-21T21:54:00+03:00" level=info msg="agent: registering usage stats for cluster ID 'kCPl0DBrEVfQMMjChZs16yW7kqkzU9hGmFSxiBH6kkY='" node=server-1
time="2021-06-21T21:54:00+03:00" level=info msg="agent: Listen for events" node=server-1
time="2021-06-21T21:54:00+03:00" level=info msg="agent: Received event" event=member-join node=server-1
time="2021-06-21T21:54:00+03:00" level=debug msg="agent: Member event" event=member-join member=server-1 node=server-1
time="2021-06-21T21:54:00+03:00" level=info msg="adding server" node=server-1 server=server-1
time="2021-06-21T21:54:01+03:00" level=info msg="agent: Received event" event=member-update node=server-1
time="2021-06-21T21:54:01+03:00" level=debug msg="agent: Member event" event=member-update member=server-1 node=server-1
time="2021-06-21T21:54:02+03:00" level=info msg="2021-06-21T21:54:02.385+0300 [WARN] raft: heartbeat timeout reached, starting election: last-leader="
time="2021-06-21T21:54:02+03:00" level=info msg="2021-06-21T21:54:02.385+0300 [INFO] raft: entering candidate state: node=\"Node at 172.17.0.4:8902 [Candidate]\" term=2"
time="2021-06-21T21:54:02+03:00" level=info msg="2021-06-21T21:54:02.434+0300 [DEBUG] raft: votes: needed=1"
time="2021-06-21T21:54:02+03:00" level=info msg="2021-06-21T21:54:02.435+0300 [DEBUG] raft: vote granted: from=server-1 term=2 tally=1"
time="2021-06-21T21:54:02+03:00" level=info msg="2021-06-21T21:54:02.435+0300 [INFO] raft: election won: tally=1"
time="2021-06-21T21:54:02+03:00" level=info msg="2021-06-21T21:54:02.435+0300 [INFO] raft: entering leader state: leader=\"Node at 172.17.0.4:8902 [Leader]\""
time="2021-06-21T21:54:02+03:00" level=info msg="dkron: cluster leadership acquired" node=server-1
time="2021-06-21T21:54:02+03:00" level=info msg="dkron: monitoring leadership" node=server-1
time="2021-06-21T21:54:02+03:00" level=info msg="agent: Starting scheduler" node=server-1
time="2021-06-21T21:54:02+03:00" level=debug msg="dkron: Skipping self join check since the cluster is too small" node=server-1 peer=server-1
time="2021-06-21T21:54:02+03:00" level=info msg="2021/06/21 21:54:02 [DEBUG] memberlist: Stream connection from=172.17.0.5:43642"
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [DEBUG] memberlist: Stream connection from=172.17.0.1:58668"
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [INFO] serf: EventMemberJoin: server-2 192.168.1.20"
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="agent: Received event" event=member-join node=server-1
time="2021-06-21T21:54:10+03:00" level=debug msg="agent: Member event" event=member-join member=server-2 node=server-1
time="2021-06-21T21:54:10+03:00" level=warning msg="non-server in gossip pool" member=server-2 node=server-1
time="2021-06-21T21:54:10+03:00" level=info msg="agent: Received event" event=member-update node=server-1
time="2021-06-21T21:54:10+03:00" level=debug msg="agent: Member event" event=member-update member=server-1 node=server-1
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [INFO] serf: EventMemberUpdate: server-2"
time="2021-06-21T21:54:11+03:00" level=info msg="agent: Received event" event=member-update node=server-1
time="2021-06-21T21:54:11+03:00" level=debug msg="agent: Member event" event=member-update member=server-1 node=server-1
time="2021-06-21T21:54:11+03:00" level=debug msg="agent: Member event" event=member-update member=server-2 node=server-1
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] memberlist: Stream connection from=172.17.0.1:58696"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [INFO] serf: EventMemberJoin: server-3 192.168.1.20"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:20+03:00" level=info msg="agent: Received event" event=member-update node=server-1
time="2021-06-21T21:54:20+03:00" level=debug msg="agent: Member event" event=member-update member=server-1 node=server-1
time="2021-06-21T21:54:20+03:00" level=debug msg="agent: Member event" event=member-update member=server-2 node=server-1
time="2021-06-21T21:54:20+03:00" level=info msg="agent: Received event" event=member-join node=server-1
time="2021-06-21T21:54:20+03:00" level=debug msg="agent: Member event" event=member-join member=server-3 node=server-1
time="2021-06-21T21:54:20+03:00" level=warning msg="non-server in gossip pool" member=server-3 node=server-1
time="2021-06-21T21:54:20+03:00" level=info msg="2021/06/21 21:54:20 [INFO] serf: EventMemberUpdate: server-3"
time="2021-06-21T21:54:21+03:00" level=info msg="agent: Received event" event=member-update node=server-1
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-update member=server-1 node=server-1
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-update member=server-2 node=server-1
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-update member=server-3 node=server-1
time="2021-06-21T21:54:42+03:00" level=info msg="2021/06/21 21:54:42 [DEBUG] memberlist: Initiating push/pull sync with: server-2 192.168.1.20:8904"
time="2021-06-21T21:55:00+03:00" level=info msg="2021/06/21 21:55:00 [DEBUG] memberlist: Stream connection from=172.17.0.1:58738"
time="2021-06-21T21:55:02+03:00" level=debug msg="dkron: Skipping self join check since the cluster is too small" node=server-1 peer=server-1
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.459+0300 [INFO] raft: updating configuration: command=AddStaging server-id=server-2 server-addr=192.168.1.20:8905 servers=\"[{Suffrage:Voter ID:server-1 Address:172.17.0.4:8902} {Suffrage:Voter ID:server-2 Address:192.168.1.20:8905}]\""
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.467+0300 [INFO] raft: added peer, starting replication: peer=server-2"
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.467+0300 [INFO] raft: updating configuration: command=AddStaging server-id=server-3 server-addr=192.168.1.20:8908 servers=\"[{Suffrage:Voter ID:server-1 Address:172.17.0.4:8902} {Suffrage:Voter ID:server-2 Address:192.168.1.20:8905} {Suffrage:Voter ID:server-3 Address:192.168.1.20:8908}]\""
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.492+0300 [INFO] raft: added peer, starting replication: peer=server-3"
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.509+0300 [WARN] raft: appendEntries rejected, sending older logs: peer=\"{Voter server-2 192.168.1.20:8905}\" next=1"
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.534+0300 [WARN] raft: appendEntries rejected, sending older logs: peer=\"{Voter server-3 192.168.1.20:8908}\" next=1"
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.542+0300 [INFO] raft: pipelining replication: peer=\"{Voter server-2 192.168.1.20:8905}\""
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.559+0300 [INFO] raft: pipelining replication: peer=\"{Voter server-3 192.168.1.20:8908}\""
time="2021-06-21T21:55:12+03:00" level=info msg="2021/06/21 21:55:12 [DEBUG] memberlist: Initiating push/pull sync with: server-3 192.168.1.20:8907"
time="2021-06-21T21:55:30+03:00" level=info msg="2021/06/21 21:55:30 [DEBUG] memberlist: Stream connection from=172.17.0.1:58770"
time="2021-06-21T21:55:39+03:00" level=info msg="2021/06/21 21:55:39 [DEBUG] memberlist: Stream connection from=172.17.0.1:58780"
time="2021-06-21T21:55:42+03:00" level=info msg="2021/06/21 21:55:42 [DEBUG] memberlist: Initiating push/pull sync with: server-2 192.168.1.20:8904"
time="2021-06-21T21:56:00+03:00" level=info msg="2021/06/21 21:56:00 [DEBUG] memberlist: Stream connection from=172.17.0.1:58794"
time="2021-06-21T21:56:02+03:00" level=info msg="2021-06-21T21:56:02.483+0300 [INFO] raft: updating configuration: command=RemoveServer server-id=server-1 server-addr= servers=\"[{Suffrage:Voter ID:server-2 Address:192.168.1.20:8905} {Suffrage:Voter ID:server-3 Address:192.168.1.20:8908}]\""
time="2021-06-21T21:56:02+03:00" level=info msg="2021-06-21T21:56:02.666+0300 [INFO] raft: removed ourself, shutting down"
time="2021-06-21T21:56:02+03:00" level=debug msg="dkron: shutting down leader loop" node=server-1
time="2021-06-21T21:56:02+03:00" level=info msg="dkron: removed server with duplicate ID" node=server-1 server=server-1
time="2021-06-21T21:56:02+03:00" level=info msg="2021-06-21T21:56:02.666+0300 [INFO] raft: aborting pipeline replication: peer=\"{Voter server-2 192.168.1.20:8905}\""
time="2021-06-21T21:56:02+03:00" level=error msg="dkron: failed to add raft peer" error="raft is already shutdown" node=server-1
time="2021-06-21T21:56:02+03:00" level=error msg="failed to reconcile member" error="raft is already shutdown" member="{server-1 192.168.1.20 8901 map[dc:backpack expect:1 group:servers host:dkron-server-1.backpack.test port:8902 region:test role:dkron rpc_addr:192.168.1.20:8902 server:true version:3.1.8] alive 1 5 2 2 5 4}" node=server-1
time="2021-06-21T21:56:02+03:00" level=info msg="2021-06-21T21:56:02.666+0300 [INFO] raft: aborting pipeline replication: peer=\"{Voter server-3 192.168.1.20:8908}\""
time="2021-06-21T21:56:02+03:00" level=error msg="dkron: failed to reconcile" error="raft is already shutdown" node=server-1
time="2021-06-21T21:56:02+03:00" level=debug msg="scheduler: Stopping scheduler" node=server-1
time="2021-06-21T21:56:02+03:00" level=info msg="dkron: cluster leadership lost" node=server-1
time="2021-06-21T21:56:02+03:00" level=info msg="dkron: monitoring leadership" node=server-1
Logs from server-2:
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-files]" node=server-2 path=/dkron/dkron-processor-files
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-processor-files pid=13
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-processor-files
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin973874045 network=unix node=server-2 subsystem_name=plugins.dkron-processor-files timestamp="2021-06-21T21:54:09.515+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-fluent]" node=server-2 path=/dkron/dkron-processor-fluent
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-processor-fluent pid=20
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-processor-fluent
time="2021-06-21T21:54:09+03:00" level=debug msg="time=\"2021-06-21T21:54:09+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_HOST\"" node=server-2 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:54:09+03:00" level=debug msg="time=\"2021-06-21T21:54:09+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_PORT\"" node=server-2 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:54:09+03:00" level=debug msg="time=\"2021-06-21T21:54:09+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_TAG\"" node=server-2 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin406444169 network=unix node=server-2 subsystem_name=plugins.dkron-processor-fluent timestamp="2021-06-21T21:54:09.547+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-log]" node=server-2 path=/dkron/dkron-processor-log
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-processor-log pid=27
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-processor-log
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin284985561 network=unix node=server-2 subsystem_name=plugins.dkron-processor-log timestamp="2021-06-21T21:54:09.554+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-syslog]" node=server-2 path=/dkron/dkron-processor-syslog
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-processor-syslog pid=34
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-processor-syslog
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin953028521 network=unix node=server-2 subsystem_name=plugins.dkron-processor-syslog timestamp="2021-06-21T21:54:09.564+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-gcppubsub]" node=server-2 path=/dkron/dkron-executor-gcppubsub
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-executor-gcppubsub pid=41
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-executor-gcppubsub
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin314043060 network=unix node=server-2 subsystem_name=plugins.dkron-executor-gcppubsub timestamp="2021-06-21T21:54:09.577+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-http]" node=server-2 path=/dkron/dkron-executor-http
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-executor-http pid=48
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-executor-http
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin687072465 network=unix node=server-2 subsystem_name=plugins.dkron-executor-http timestamp="2021-06-21T21:54:09.587+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-kafka]" node=server-2 path=/dkron/dkron-executor-kafka
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-executor-kafka pid=56
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-executor-kafka
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin055072684 network=unix node=server-2 subsystem_name=plugins.dkron-executor-kafka timestamp="2021-06-21T21:54:09.596+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-nats]" node=server-2 path=/dkron/dkron-executor-nats
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-executor-nats pid=63
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-executor-nats
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin937499942 network=unix node=server-2 subsystem_name=plugins.dkron-executor-nats timestamp="2021-06-21T21:54:09.607+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-rabbitmq]" node=server-2 path=/dkron/dkron-executor-rabbitmq
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-executor-rabbitmq pid=70
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-executor-rabbitmq
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin579262576 network=unix node=server-2 subsystem_name=plugins.dkron-executor-rabbitmq timestamp="2021-06-21T21:54:09.659+0300"
time="2021-06-21T21:54:09+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-shell]" node=server-2 path=/dkron/dkron-executor-shell
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin started" node=server-2 path=/dkron/dkron-executor-shell pid=77
time="2021-06-21T21:54:09+03:00" level=debug msg="waiting for RPC address" node=server-2 path=/dkron/dkron-executor-shell
time="2021-06-21T21:54:09+03:00" level=debug msg="using plugin" node=server-2 version=1
time="2021-06-21T21:54:09+03:00" level=debug msg="plugin address" address=/tmp/plugin189683034 network=unix node=server-2 subsystem_name=plugins.dkron-executor-shell timestamp="2021-06-21T21:54:09.671+0300"
time="2021-06-21T21:54:09+03:00" level=info msg="agent: Dkron agent starting" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [INFO] serf: EventMemberJoin: server-2 192.168.1.20"
time="2021-06-21T21:54:09+03:00" level=info msg="agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="agent: Joining cluster..." cluster=LAN node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.20:8901"
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [INFO] serf: EventMemberJoin: server-1 192.168.1.20"
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.5:8904"
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [DEBUG] memberlist: Stream connection from=172.17.0.5:39724"
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [DEBUG] memberlist: Failed to join 192.168.1.20: dial tcp 192.168.1.20:8907: connect: connection refused"
time="2021-06-21T21:54:09+03:00" level=info msg="agent: Join LAN completed. Synced with 2 initial agents" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached." node=server-2
time="2021-06-21T21:54:09+03:00" level=info node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] [WARNING] Running in \"debug\" mode. Switch to \"release\" mode in production." node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg=" - using env:\texport GIN_MODE=release" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg=" - using code:\tgin.SetMode(gin.ReleaseMode)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /debug/vars --> github.com/gin-contrib/expvar.Handler.func1 (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /health --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).APIRoutes.func1 (4 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1 --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/ --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/members --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).membersHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/leader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaderHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/isleader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).isLeaderHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] POST /v1/leave --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaveHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] POST /v1/restore --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).restoreHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/busy --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).busyHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] POST /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] PATCH /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobsHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] DELETE /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobDeleteHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job/run --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job/toggle --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobToggleHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] PUT /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobGetHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /v1/jobs/:job/executions --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).executionsHandler-fm (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET / --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func1 (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] [WARNING] Since SetHTMLTemplate() is NOT thread-safe. It should only be called" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="at initialization. ie. before any route is registered or the router is listening in a socket:" node=server-2
time="2021-06-21T21:54:09+03:00" level=info node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="\trouter := gin.Default()" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="\trouter.SetHTMLTemplate(template) // << good place" node=server-2
time="2021-06-21T21:54:09+03:00" level=info node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="api: Running HTTP server" address=":8080" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] GET /ui/*filepath --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func2 (6 handlers)" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="[GIN-debug] Listening and serving HTTP on :8080" node=server-2
time="2021-06-21T21:54:09+03:00" level=info msg="2021/06/21 21:54:09 [DEBUG] serf: messageJoinType: server-1"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-1"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-1"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-1"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [DEBUG] serf: messageJoinType: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="2021-06-21T21:54:10.826+0300 [INFO] raft: initial configuration: index=0 servers=[]"
time="2021-06-21T21:54:10+03:00" level=info msg="2021-06-21T21:54:10.826+0300 [INFO] raft: entering follower state: follower=\"Node at 172.17.0.5:8905 [Follower]\" leader="
time="2021-06-21T21:54:10+03:00" level=info msg="dkron: monitoring leadership" node=server-2
time="2021-06-21T21:54:10+03:00" level=info msg="2021/06/21 21:54:10 [INFO] serf: EventMemberUpdate: server-2"
time="2021-06-21T21:54:10+03:00" level=info msg="agent: registering usage stats for cluster ID 'mBq1W0Tg7l2C4B3hTxdkNQDhr5HdQhr1ZYv8xFNS0zU='" node=server-2
time="2021-06-21T21:54:11+03:00" level=info msg="agent: Listen for events" node=server-2
time="2021-06-21T21:54:11+03:00" level=info msg="agent: Received event" event=member-join node=server-2
time="2021-06-21T21:54:11+03:00" level=debug msg="agent: Member event" event=member-join member=server-2 node=server-2
time="2021-06-21T21:54:11+03:00" level=debug msg="agent: Member event" event=member-join member=server-1 node=server-2
time="2021-06-21T21:54:11+03:00" level=info msg="adding server" node=server-2 server=server-2
time="2021-06-21T21:54:11+03:00" level=info msg="adding server" node=server-2 server=server-1
time="2021-06-21T21:54:11+03:00" level=info msg="agent: Received event" event=member-update node=server-2
time="2021-06-21T21:54:11+03:00" level=debug msg="agent: Member event" event=member-update member=server-2 node=server-2
time="2021-06-21T21:54:12+03:00" level=info msg="2021/06/21 21:54:12 [DEBUG] memberlist: Stream connection from=172.17.0.6:44374"
time="2021-06-21T21:54:12+03:00" level=info msg="2021-06-21T21:54:12.747+0300 [WARN] raft: no known peers, aborting election"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] memberlist: Stream connection from=172.17.0.1:35392"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [INFO] serf: EventMemberJoin: server-3 192.168.1.20"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:20+03:00" level=info msg="2021/06/21 21:54:20 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:20+03:00" level=info msg="agent: Received event" event=member-update node=server-2
time="2021-06-21T21:54:20+03:00" level=debug msg="agent: Member event" event=member-update member=server-2 node=server-2
time="2021-06-21T21:54:20+03:00" level=info msg="agent: Received event" event=member-join node=server-2
time="2021-06-21T21:54:20+03:00" level=debug msg="agent: Member event" event=member-join member=server-3 node=server-2
time="2021-06-21T21:54:20+03:00" level=warning msg="non-server in gossip pool" member=server-3 node=server-2
time="2021-06-21T21:54:20+03:00" level=info msg="2021/06/21 21:54:20 [INFO] serf: EventMemberUpdate: server-3"
time="2021-06-21T21:54:21+03:00" level=info msg="agent: Received event" event=member-update node=server-2
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-update member=server-2 node=server-2
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-update member=server-3 node=server-2
time="2021-06-21T21:54:42+03:00" level=info msg="2021/06/21 21:54:42 [DEBUG] memberlist: Stream connection from=172.17.0.1:35416"
time="2021-06-21T21:55:00+03:00" level=info msg="2021/06/21 21:55:00 [DEBUG] memberlist: Initiating push/pull sync with: server-1 192.168.1.20:8901"
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.508+0300 [WARN] raft: failed to get previous log: previous-index=5 last-index=0 error=\"log not found\""
time="2021-06-21T21:55:09+03:00" level=info msg="2021/06/21 21:55:09 [DEBUG] memberlist: Stream connection from=172.17.0.1:35450"
time="2021-06-21T21:55:30+03:00" level=info msg="2021/06/21 21:55:30 [DEBUG] memberlist: Initiating push/pull sync with: server-1 192.168.1.20:8901"
time="2021-06-21T21:55:42+03:00" level=info msg="2021/06/21 21:55:42 [DEBUG] memberlist: Stream connection from=172.17.0.1:35472"
time="2021-06-21T21:56:00+03:00" level=info msg="2021/06/21 21:56:00 [DEBUG] memberlist: Initiating push/pull sync with: server-1 192.168.1.20:8901"
time="2021-06-21T21:56:03+03:00" level=info msg="2021-06-21T21:56:03.709+0300 [WARN] raft: heartbeat timeout reached, starting election: last-leader=172.17.0.4:8902"
time="2021-06-21T21:56:03+03:00" level=info msg="2021-06-21T21:56:03.710+0300 [INFO] raft: entering candidate state: node=\"Node at 172.17.0.5:8905 [Candidate]\" term=3"
time="2021-06-21T21:56:03+03:00" level=info msg="2021-06-21T21:56:03.765+0300 [DEBUG] raft: votes: needed=2"
time="2021-06-21T21:56:03+03:00" level=info msg="2021-06-21T21:56:03.765+0300 [DEBUG] raft: vote granted: from=server-2 term=3 tally=1"
time="2021-06-21T21:56:04+03:00" level=info msg="2021-06-21T21:56:04.775+0300 [INFO] raft: duplicate requestVote for same term: term=3"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.002+0300 [WARN] raft: Election timeout reached, restarting election"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.002+0300 [INFO] raft: entering candidate state: node=\"Node at 172.17.0.5:8905 [Candidate]\" term=4"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.073+0300 [DEBUG] raft: votes: needed=2"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.073+0300 [DEBUG] raft: vote granted: from=server-2 term=4 tally=1"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.166+0300 [DEBUG] raft: vote granted: from=server-3 term=4 tally=2"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.166+0300 [INFO] raft: election won: tally=2"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.166+0300 [INFO] raft: entering leader state: leader=\"Node at 172.17.0.5:8905 [Leader]\""
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.166+0300 [INFO] raft: added peer, starting replication: peer=server-3"
time="2021-06-21T21:56:05+03:00" level=info msg="dkron: cluster leadership acquired" node=server-2
time="2021-06-21T21:56:05+03:00" level=info msg="dkron: monitoring leadership" node=server-2
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.168+0300 [INFO] raft: pipelining replication: peer=\"{Voter server-3 192.168.1.20:8908}\""
time="2021-06-21T21:56:05+03:00" level=info msg="agent: Starting scheduler" node=server-2
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.375+0300 [INFO] raft: updating configuration: command=AddStaging server-id=server-1 server-addr=192.168.1.20:8902 servers=\"[{Suffrage:Voter ID:server-2 Address:192.168.1.20:8905} {Suffrage:Voter ID:server-3 Address:192.168.1.20:8908} {Suffrage:Voter ID:server-1 Address:192.168.1.20:8902}]\""
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.398+0300 [INFO] raft: added peer, starting replication: peer=server-1"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.899+0300 [WARN] raft: failed to contact: server-id=server-1 time=500.523252ms"
time="2021-06-21T21:56:06+03:00" level=info msg="2021-06-21T21:56:06.388+0300 [WARN] raft: failed to contact: server-id=server-1 time=989.367814ms"
time="2021-06-21T21:56:06+03:00" level=info msg="2021-06-21T21:56:06.843+0300 [WARN] raft: failed to contact: server-id=server-1 time=1.444322169s"
time="2021-06-21T21:56:07+03:00" level=info msg="2021-06-21T21:56:07.324+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=1.925178228s"
time="2021-06-21T21:56:07+03:00" level=info msg="2021-06-21T21:56:07.772+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=2.373390365s"
time="2021-06-21T21:56:08+03:00" level=info msg="2021-06-21T21:56:08.233+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=2.834782463s"
time="2021-06-21T21:56:08+03:00" level=info msg="2021-06-21T21:56:08.721+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=3.322728031s"
time="2021-06-21T21:56:09+03:00" level=info msg="2021-06-21T21:56:09.161+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=3.763070811s"
time="2021-06-21T21:56:09+03:00" level=info msg="2021-06-21T21:56:09.640+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=4.241981019s"
time="2021-06-21T21:56:10+03:00" level=info msg="2021-06-21T21:56:10.059+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=4.660430933s"
time="2021-06-21T21:56:10+03:00" level=info msg="2021-06-21T21:56:10.525+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=5.126260702s"
time="2021-06-21T21:56:10+03:00" level=info msg="2021-06-21T21:56:10.992+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=5.593855959s"
time="2021-06-21T21:56:11+03:00" level=info msg="2021-06-21T21:56:11.482+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=6.083859843s"
time="2021-06-21T21:56:11+03:00" level=info msg="2021-06-21T21:56:11.953+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=6.554133231s"
time="2021-06-21T21:56:12+03:00" level=info msg="2021-06-21T21:56:12.441+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=7.042791739s"
time="2021-06-21T21:56:12+03:00" level=info msg="2021-06-21T21:56:12.919+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=7.520645571s"
time="2021-06-21T21:56:13+03:00" level=info msg="2021-06-21T21:56:13.369+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=7.970339885s"
time="2021-06-21T21:56:13+03:00" level=info msg="2021-06-21T21:56:13.852+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=8.45335128s"
time="2021-06-21T21:56:14+03:00" level=info msg="2021-06-21T21:56:14.342+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=8.943714888s"
time="2021-06-21T21:56:14+03:00" level=info msg="2021-06-21T21:56:14.775+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=9.376557655s"
time="2021-06-21T21:56:15+03:00" level=info msg="2021-06-21T21:56:15.227+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=9.828982045s"
time="2021-06-21T21:56:15+03:00" level=info msg="2021-06-21T21:56:15.657+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=10.25905281s"
time="2021-06-21T21:56:16+03:00" level=info msg="2021-06-21T21:56:16.105+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=10.70610359s"
time="2021-06-21T21:56:16+03:00" level=info msg="2021-06-21T21:56:16.538+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=11.139443632s"
time="2021-06-21T21:56:17+03:00" level=info msg="2021-06-21T21:56:17.006+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=11.607479774s"
time="2021-06-21T21:56:17+03:00" level=info msg="2021-06-21T21:56:17.505+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=12.106496346s"
time="2021-06-21T21:56:17+03:00" level=info msg="2021-06-21T21:56:17.975+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=12.576854575s"
time="2021-06-21T21:56:18+03:00" level=info msg="2021-06-21T21:56:18.397+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=12.998896552s"
time="2021-06-21T21:56:18+03:00" level=info msg="2021-06-21T21:56:18.874+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=13.475334705s"
time="2021-06-21T21:56:19+03:00" level=info msg="2021-06-21T21:56:19.330+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=13.931473472s"
time="2021-06-21T21:56:19+03:00" level=info msg="2021-06-21T21:56:19.762+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=14.364102464s"
time="2021-06-21T21:56:20+03:00" level=info msg="2021-06-21T21:56:20.252+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=14.853538604s"
time="2021-06-21T21:56:20+03:00" level=info msg="2021-06-21T21:56:20.747+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=15.348427874s"
time="2021-06-21T21:56:21+03:00" level=info msg="2021-06-21T21:56:21.227+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=15.828460853s"
time="2021-06-21T21:56:21+03:00" level=info msg="2021-06-21T21:56:21.709+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=16.311030627s"
time="2021-06-21T21:56:22+03:00" level=info msg="2021-06-21T21:56:22.181+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=16.782339046s"
time="2021-06-21T21:56:22+03:00" level=info msg="2021-06-21T21:56:22.641+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=17.242139034s"
time="2021-06-21T21:56:23+03:00" level=info msg="2021-06-21T21:56:23.096+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=17.697258938s"
time="2021-06-21T21:56:23+03:00" level=info msg="2021-06-21T21:56:23.551+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=18.152098677s"
time="2021-06-21T21:56:24+03:00" level=info msg="2021-06-21T21:56:24.007+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=18.608273252s"
time="2021-06-21T21:56:24+03:00" level=info msg="2021-06-21T21:56:24.450+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=19.051413506s"
time="2021-06-21T21:56:24+03:00" level=info msg="2021-06-21T21:56:24.894+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=19.495422107s"
time="2021-06-21T21:56:25+03:00" level=info msg="2021-06-21T21:56:25.354+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=19.95582861s"
time="2021-06-21T21:56:25+03:00" level=info msg="2021-06-21T21:56:25.850+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=20.451723961s"
time="2021-06-21T21:56:26+03:00" level=info msg="2021-06-21T21:56:26.342+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=20.943843759s"
time="2021-06-21T21:56:26+03:00" level=info msg="2021-06-21T21:56:26.835+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=21.436943031s"
time="2021-06-21T21:56:27+03:00" level=info msg="2021-06-21T21:56:27.320+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=21.921907093s"
time="2021-06-21T21:56:27+03:00" level=info msg="2021-06-21T21:56:27.815+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=22.416407762s"
time="2021-06-21T21:56:28+03:00" level=info msg="2021-06-21T21:56:28.314+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=22.915595999s"
time="2021-06-21T21:56:28+03:00" level=info msg="2021-06-21T21:56:28.815+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=23.416741016s"
time="2021-06-21T21:56:29+03:00" level=info msg="2021-06-21T21:56:29.279+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=23.880262129s"
time="2021-06-21T21:56:29+03:00" level=info msg="2021-06-21T21:56:29.775+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=24.377050917s"
time="2021-06-21T21:56:30+03:00" level=info msg="2021-06-21T21:56:30.244+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=24.8454258s"
time="2021-06-21T21:56:30+03:00" level=info msg="2021-06-21T21:56:30.688+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=25.289983219s"
time="2021-06-21T21:56:30+03:00" level=info msg="2021/06/21 21:56:30 [DEBUG] memberlist: Initiating push/pull sync with: server-3 192.168.1.20:8907"
time="2021-06-21T21:56:31+03:00" level=info msg="2021-06-21T21:56:31.181+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=25.782364955s"
time="2021-06-21T21:56:31+03:00" level=info msg="2021-06-21T21:56:31.663+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=26.264395101s"
time="2021-06-21T21:56:32+03:00" level=info msg="2021-06-21T21:56:32.074+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=26.675823695s"
time="2021-06-21T21:56:32+03:00" level=info msg="2021-06-21T21:56:32.556+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=27.15807081s"
time="2021-06-21T21:56:32+03:00" level=info msg="2021-06-21T21:56:32.996+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=27.597300662s"
time="2021-06-21T21:56:33+03:00" level=info msg="2021-06-21T21:56:33.442+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=28.043196741s"
time="2021-06-21T21:56:33+03:00" level=info msg="2021-06-21T21:56:33.889+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=28.490329586s"
time="2021-06-21T21:56:34+03:00" level=info msg="2021-06-21T21:56:34.361+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=28.962116303s"
time="2021-06-21T21:56:34+03:00" level=info msg="2021-06-21T21:56:34.852+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=29.453169001s"
time="2021-06-21T21:56:35+03:00" level=info msg="2021-06-21T21:56:35.326+0300 [DEBUG] raft: failed to contact: server-id=server-1 time=29.927847644s"
time="2021-06-21T21:56:35+03:00" level=info msg="2021-06-21T21:56:35.400+0300 [ERROR] raft: failed to appendEntries to: peer=\"{Voter server-1 192.168.1.20:8902}\" error=\"read tcp 172.17.0.5:43738->192.168.1.20:8902: i/o timeout\""
time="2021-06-21T21:56:35+03:00" level=info msg="2021-06-21T21:56:35.559+0300 [ERROR] raft: failed to heartbeat to: peer=192.168.1.20:8902 error=\"read tcp 172.17.0.5:43742->192.168.1.20:8902: i/o timeout\""
Logs from server-3:
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-files]" node=server-3 path=/dkron/dkron-processor-files
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-processor-files pid=12
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-processor-files
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin789662903 network=unix node=server-3 subsystem_name=plugins.dkron-processor-files timestamp="2021-06-21T21:54:19.359+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-fluent]" node=server-3 path=/dkron/dkron-processor-fluent
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-processor-fluent pid=19
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-processor-fluent
time="2021-06-21T21:54:19+03:00" level=debug msg="time=\"2021-06-21T21:54:19+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_HOST\"" node=server-3 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:54:19+03:00" level=debug msg="time=\"2021-06-21T21:54:19+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_PORT\"" node=server-3 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:54:19+03:00" level=debug msg="time=\"2021-06-21T21:54:19+03:00\" level=warning msg=\"empty value for environment variable FLUENTBIT_TAG\"" node=server-3 subsystem_name=plugins.dkron-processor-fluent
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin415757600 network=unix node=server-3 subsystem_name=plugins.dkron-processor-fluent timestamp="2021-06-21T21:54:19.374+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-log]" node=server-3 path=/dkron/dkron-processor-log
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-processor-log pid=26
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-processor-log
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin819100987 network=unix node=server-3 subsystem_name=plugins.dkron-processor-log timestamp="2021-06-21T21:54:19.383+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-processor-syslog]" node=server-3 path=/dkron/dkron-processor-syslog
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-processor-syslog pid=33
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-processor-syslog
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin522715826 network=unix node=server-3 subsystem_name=plugins.dkron-processor-syslog timestamp="2021-06-21T21:54:19.395+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-gcppubsub]" node=server-3 path=/dkron/dkron-executor-gcppubsub
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-executor-gcppubsub pid=40
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-executor-gcppubsub
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin302201081 network=unix node=server-3 subsystem_name=plugins.dkron-executor-gcppubsub timestamp="2021-06-21T21:54:19.412+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-http]" node=server-3 path=/dkron/dkron-executor-http
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-executor-http pid=47
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-executor-http
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin105551594 network=unix node=server-3 subsystem_name=plugins.dkron-executor-http timestamp="2021-06-21T21:54:19.421+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-kafka]" node=server-3 path=/dkron/dkron-executor-kafka
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-executor-kafka pid=54
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-executor-kafka
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin531749734 network=unix node=server-3 subsystem_name=plugins.dkron-executor-kafka timestamp="2021-06-21T21:54:19.429+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-nats]" node=server-3 path=/dkron/dkron-executor-nats
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-executor-nats pid=61
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-executor-nats
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin018042023 network=unix node=server-3 subsystem_name=plugins.dkron-executor-nats timestamp="2021-06-21T21:54:19.437+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-rabbitmq]" node=server-3 path=/dkron/dkron-executor-rabbitmq
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-executor-rabbitmq pid=68
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-executor-rabbitmq
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin297845923 network=unix node=server-3 subsystem_name=plugins.dkron-executor-rabbitmq timestamp="2021-06-21T21:54:19.446+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=debug msg="starting plugin" args="[/dkron/dkron-executor-shell]" node=server-3 path=/dkron/dkron-executor-shell
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin started" node=server-3 path=/dkron/dkron-executor-shell pid=75
time="2021-06-21T21:54:19+03:00" level=debug msg="waiting for RPC address" node=server-3 path=/dkron/dkron-executor-shell
time="2021-06-21T21:54:19+03:00" level=debug msg="plugin address" address=/tmp/plugin611496565 network=unix node=server-3 subsystem_name=plugins.dkron-executor-shell timestamp="2021-06-21T21:54:19.454+0300"
time="2021-06-21T21:54:19+03:00" level=debug msg="using plugin" node=server-3 version=1
time="2021-06-21T21:54:19+03:00" level=info msg="agent: Dkron agent starting" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [INFO] serf: EventMemberJoin: server-3 192.168.1.20"
time="2021-06-21T21:54:19+03:00" level=info msg="agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="agent: Joining cluster..." cluster=LAN node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.20:8901"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [INFO] serf: EventMemberJoin: server-2 192.168.1.20"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [INFO] serf: EventMemberJoin: server-1 192.168.1.20"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.20:8904"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.6:8907"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] memberlist: Stream connection from=172.17.0.6:36614"
time="2021-06-21T21:54:19+03:00" level=info msg="agent: Join LAN completed. Synced with 3 initial agents" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached." node=server-3
time="2021-06-21T21:54:19+03:00" level=info node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] [WARNING] Running in \"debug\" mode. Switch to \"release\" mode in production." node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg=" - using env:\texport GIN_MODE=release" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg=" - using code:\tgin.SetMode(gin.ReleaseMode)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /debug/vars --> github.com/gin-contrib/expvar.Handler.func1 (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /health --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).APIRoutes.func1 (4 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1 --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/ --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/members --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).membersHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/leader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaderHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/isleader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).isLeaderHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] POST /v1/leave --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaveHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] POST /v1/restore --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).restoreHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/busy --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).busyHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] POST /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] PATCH /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobsHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] DELETE /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobDeleteHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job/run --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] POST /v1/jobs/:job/toggle --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobToggleHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] PUT /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobGetHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /v1/jobs/:job/executions --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).executionsHandler-fm (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET / --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func1 (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] [WARNING] Since SetHTMLTemplate() is NOT thread-safe. It should only be called" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="at initialization. ie. before any route is registered or the router is listening in a socket:" node=server-3
time="2021-06-21T21:54:19+03:00" level=info node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="\trouter := gin.Default()" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="\trouter.SetHTMLTemplate(template) // << good place" node=server-3
time="2021-06-21T21:54:19+03:00" level=info node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="api: Running HTTP server" address=":8080" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] GET /ui/*filepath --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func2 (6 handlers)" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="[GIN-debug] Listening and serving HTTP on :8080" node=server-3
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:19+03:00" level=info msg="2021/06/21 21:54:19 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:20+03:00" level=info msg="2021/06/21 21:54:20 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:20+03:00" level=info msg="2021/06/21 21:54:20 [DEBUG] serf: messageJoinType: server-3"
time="2021-06-21T21:54:20+03:00" level=info msg="2021-06-21T21:54:20.668+0300 [INFO] raft: initial configuration: index=0 servers=[]"
time="2021-06-21T21:54:20+03:00" level=info msg="2021-06-21T21:54:20.669+0300 [INFO] raft: entering follower state: follower=\"Node at 172.17.0.6:8908 [Follower]\" leader="
time="2021-06-21T21:54:20+03:00" level=info msg="dkron: monitoring leadership" node=server-3
time="2021-06-21T21:54:20+03:00" level=info msg="agent: registering usage stats for cluster ID 'YfybLU5paUS6k13BMHNGxFjxpDAvnMzdm9WsXeIhAlk='" node=server-3
time="2021-06-21T21:54:20+03:00" level=info msg="2021/06/21 21:54:20 [INFO] serf: EventMemberUpdate: server-3"
time="2021-06-21T21:54:21+03:00" level=info msg="agent: Listen for events" node=server-3
time="2021-06-21T21:54:21+03:00" level=info msg="agent: Received event" event=member-join node=server-3
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-join member=server-1 node=server-3
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-join member=server-3 node=server-3
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-join member=server-2 node=server-3
time="2021-06-21T21:54:21+03:00" level=info msg="adding server" node=server-3 server=server-1
time="2021-06-21T21:54:21+03:00" level=info msg="adding server" node=server-3 server=server-3
time="2021-06-21T21:54:21+03:00" level=info msg="adding server" node=server-3 server=server-2
time="2021-06-21T21:54:21+03:00" level=info msg="agent: Received event" event=member-update node=server-3
time="2021-06-21T21:54:21+03:00" level=debug msg="agent: Member event" event=member-update member=server-3 node=server-3
time="2021-06-21T21:54:22+03:00" level=info msg="2021-06-21T21:54:22.457+0300 [WARN] raft: no known peers, aborting election"
time="2021-06-21T21:54:22+03:00" level=info msg="2021/06/21 21:54:22 [DEBUG] memberlist: Stream connection from=172.17.0.11:49794"
time="2021-06-21T21:55:02+03:00" level=info msg="2021-06-21T21:55:02.534+0300 [WARN] raft: failed to get previous log: previous-index=6 last-index=0 error=\"log not found\""
time="2021-06-21T21:55:09+03:00" level=info msg="2021/06/21 21:55:09 [DEBUG] memberlist: Initiating push/pull sync with: server-2 192.168.1.20:8904"
time="2021-06-21T21:55:12+03:00" level=info msg="2021/06/21 21:55:12 [DEBUG] memberlist: Stream connection from=172.17.0.1:34966"
time="2021-06-21T21:55:39+03:00" level=info msg="2021/06/21 21:55:39 [DEBUG] memberlist: Initiating push/pull sync with: server-1 192.168.1.20:8901"
time="2021-06-21T21:56:03+03:00" level=info msg="2021-06-21T21:56:03.766+0300 [WARN] raft: rejecting vote request since we have a leader: from=172.17.0.5:8905 leader=172.17.0.4:8902"
time="2021-06-21T21:56:04+03:00" level=info msg="2021-06-21T21:56:04.748+0300 [WARN] raft: heartbeat timeout reached, starting election: last-leader=172.17.0.4:8902"
time="2021-06-21T21:56:04+03:00" level=info msg="2021-06-21T21:56:04.748+0300 [INFO] raft: entering candidate state: node=\"Node at 172.17.0.6:8908 [Candidate]\" term=3"
time="2021-06-21T21:56:04+03:00" level=info msg="2021-06-21T21:56:04.832+0300 [DEBUG] raft: votes: needed=2"
time="2021-06-21T21:56:04+03:00" level=info msg="2021-06-21T21:56:04.832+0300 [DEBUG] raft: vote granted: from=server-3 term=3 tally=1"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.074+0300 [DEBUG] raft: lost leadership because received a requestVote with a newer term"
time="2021-06-21T21:56:05+03:00" level=info msg="2021-06-21T21:56:05.165+0300 [INFO] raft: entering follower state: follower=\"Node at 172.17.0.6:8908 [Follower]\" leader="
time="2021-06-21T21:56:09+03:00" level=info msg="2021/06/21 21:56:09 [DEBUG] memberlist: Initiating push/pull sync with: server-1 192.168.1.20:8901"
time="2021-06-21T21:56:12+03:00" level=info msg="2021/06/21 21:56:12 [DEBUG] memberlist: Stream connection from=172.17.0.1:35026"
time="2021-06-21T21:56:30+03:00" level=info msg="2021/06/21 21:56:30 [DEBUG] memberlist: Stream connection from=172.17.0.1:35030"
time="2021-06-21T21:56:39+03:00" level=info msg="2021/06/21 21:56:39 [DEBUG] memberlist: Initiating push/pull sync with: server-1 192.168.1.20:8901"
time="2021-06-21T21:57:09+03:00" level=info msg="2021/06/21 21:57:09 [DEBUG] memberlist: Initiating push/pull sync with: server-2 192.168.1.20:8904"
time="2021-06-21T21:57:39+03:00" level=info msg="2021/06/21 21:57:39 [DEBUG] memberlist: Initiating push/pull sync with: server-2 192.168.1.20:8904"
/dkron/dkron raft list-peers --rpc-addr dkron-server-1.backpack.test:8902 executed inside server-1 container prints:
Node ID Address State Voter
(unknown) server-1 172.17.0.4:8902 leader true
server-2 server-2 192.168.1.20:8905 follower true
server-3 server-3 192.168.1.20:8908 follower true
Some time later:
Node ID Address State Voter
server-2 server-2 192.168.1.20:8905 follower true
server-3 server-3 192.168.1.20:8908 follower true
I am having no leader elected on a single server instance. Not sure what's wrong. I copied over the configs I used from my single server test box and its not working on the new vm. It keeps saying "no member leader found in member list". When I use the debug mode it appears to be looking for more than one vote.
Debug log below. Open configs in the .yml file are
# Dkron example configuration file
# This node is running in server mode
server: true
# Provides the number of expected servers in the datacenter.
# Either this value should not be provided or the value must agree with other servers in the cluster.
# When provided, Dkron waits until the specified number of servers are available and then bootstraps the cluster.
# This allows an initial leader to be elected automatically. This flag requires server mode.
bootstrap-expect: 1
zubair.patel@pim-cpt-apps-t1:~$ sudo dkron agent --log-level=debug
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-processor-files]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-files
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-files pid=5786
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-files
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin088594320 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-files timestamp="2021-06-22T13:27:48.500Z"
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-processor-fluent]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-fluent
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-fluent pid=5795
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-fluent
DEBU[2021-06-22T13:27:48Z] time="2021-06-22T13:27:48Z" level=warning msg="empty value for environment variable FLUENTBIT_HOST" node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-fluent
DEBU[2021-06-22T13:27:48Z] time="2021-06-22T13:27:48Z" level=warning msg="empty value for environment variable FLUENTBIT_PORT" node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-fluent
DEBU[2021-06-22T13:27:48Z] time="2021-06-22T13:27:48Z" level=warning msg="empty value for environment variable FLUENTBIT_TAG" node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-fluent
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin179512185 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-fluent timestamp="2021-06-22T13:27:48.507Z"
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-processor-log]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-log
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-log pid=5804
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-log
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin736850028 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-log timestamp="2021-06-22T13:27:48.516Z"
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-processor-syslog]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-syslog
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-syslog pid=5813
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-syslog
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin332703956 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-syslog timestamp="2021-06-22T13:27:48.523Z"
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-executor-gcppubsub]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-gcppubsub
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-gcppubsub pid=5822
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-gcppubsub
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin388284725 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-executor-gcppubsub timestamp="2021-06-22T13:27:48.536Z"
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-executor-http]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-http
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-http pid=5830
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-http
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin045858031 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-executor-http timestamp="2021-06-22T13:27:48.544Z"
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-executor-kafka]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-kafka
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-kafka pid=5838
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-kafka
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin263738071 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-executor-kafka timestamp="2021-06-22T13:27:48.552Z"
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-executor-nats]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-nats
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-nats pid=5847
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-nats
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin179153499 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-executor-nats timestamp="2021-06-22T13:27:48.558Z"
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-executor-rabbitmq]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-rabbitmq
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-rabbitmq pid=5854
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-rabbitmq
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin955293796 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-executor-rabbitmq timestamp="2021-06-22T13:27:48.565Z"
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
DEBU[2021-06-22T13:27:48Z] starting plugin args="[/usr/bin/dkron-executor-shell]" node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-shell
DEBU[2021-06-22T13:27:48Z] plugin started node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-shell pid=5863
DEBU[2021-06-22T13:27:48Z] waiting for RPC address node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-shell
DEBU[2021-06-22T13:27:48Z] plugin address address=/tmp/plugin140161328 network=unix node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-executor-shell timestamp="2021-06-22T13:27:48.573Z"
DEBU[2021-06-22T13:27:48Z] using plugin node=pim-cpt-apps-t1 version=1
INFO[2021-06-22T13:27:48Z] agent: Dkron agent starting node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] 2021/06/22 13:27:48 [INFO] serf: EventMemberJoin: pim-cpt-apps-t1 192.168.66.64
INFO[2021-06-22T13:27:48Z] agent: joining: [] replay: true node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] - using env: export GIN_MODE=release node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] - using code: gin.SetMode(gin.ReleaseMode) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /debug/vars --> github.com/gin-contrib/expvar.Handler.func1 (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /health --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).APIRoutes.func1 (4 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1 --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/ --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).indexHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/members --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).membersHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/leader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaderHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/isleader --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).isLeaderHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] POST /v1/leave --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).leaveHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] POST /v1/restore --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).restoreHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/busy --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).busyHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] POST /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] PATCH /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/jobs --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobsHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] DELETE /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobDeleteHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] POST /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] POST /v1/jobs/:job/run --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobRunHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] POST /v1/jobs/:job/toggle --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobToggleHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] PUT /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobCreateOrUpdateHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/jobs/:job --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).jobGetHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /v1/jobs/:job/executions --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).executionsHandler-fm (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET / --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func1 (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] [WARNING] Since SetHTMLTemplate() is NOT thread-safe. It should only be called node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] at initialization. ie. before any route is registered or the router is listening in a socket: node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] router := gin.Default() node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] router.SetHTMLTemplate(template) // << good place node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] GET /ui/*filepath --> github.com/distribworks/dkron/v3/dkron.(*HTTPTransport).UI.func2 (6 handlers) node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] api: Running HTTP server address=":8080" node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] [GIN-debug] Listening and serving HTTP on :8080 node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] 2021-06-22T13:27:48.590Z [INFO] raft: initial configuration: index=1436 servers="[{Suffrage:Voter ID:pim-cpt-apps-t1 Address:192.168.66.64:6868} {Suffrage:Voter ID:pim-cpt-dev-t2 Address:192.168.66.62:6868}]"
INFO[2021-06-22T13:27:48Z] agent: usage report client disabled node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] 2021-06-22T13:27:48.590Z [INFO] raft: entering follower state: follower="Node at 192.168.66.64:6868 [Follower]" leader=
INFO[2021-06-22T13:27:48Z] 2021/06/22 13:27:48 [INFO] serf: EventMemberUpdate: pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] dkron: monitoring leadership node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:48Z] agent: Listen for events node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:49Z] agent: Received event event=member-update node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:49Z] agent: Member event event=member-update member=pim-cpt-apps-t1 node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:50Z] 2021-06-22T13:27:50.041Z [WARN] raft: heartbeat timeout reached, starting election: last-leader=
INFO[2021-06-22T13:27:50Z] 2021-06-22T13:27:50.042Z [INFO] raft: entering candidate state: node="Node at 192.168.66.64:6868 [Candidate]" term=2367
INFO[2021-06-22T13:27:50Z] 2021-06-22T13:27:50.044Z [DEBUG] raft: votes: needed=2
INFO[2021-06-22T13:27:50Z] 2021-06-22T13:27:50.044Z [DEBUG] raft: vote granted: from=pim-cpt-apps-t1 term=2367 tally=1
INFO[2021-06-22T13:27:51Z] 2021-06-22T13:27:51.881Z [WARN] raft: Election timeout reached, restarting election
INFO[2021-06-22T13:27:51Z] 2021-06-22T13:27:51.881Z [INFO] raft: entering candidate state: node="Node at 192.168.66.64:6868 [Candidate]" term=2368
INFO[2021-06-22T13:27:51Z] 2021-06-22T13:27:51.883Z [DEBUG] raft: votes: needed=2
INFO[2021-06-22T13:27:51Z] 2021-06-22T13:27:51.883Z [DEBUG] raft: vote granted: from=pim-cpt-apps-t1 term=2368 tally=1
INFO[2021-06-22T13:27:53Z] 2021-06-22T13:27:53.853Z [WARN] raft: Election timeout reached, restarting election
INFO[2021-06-22T13:27:53Z] 2021-06-22T13:27:53.853Z [INFO] raft: entering candidate state: node="Node at 192.168.66.64:6868 [Candidate]" term=2369
INFO[2021-06-22T13:27:53Z] 2021-06-22T13:27:53.855Z [DEBUG] raft: votes: needed=2
INFO[2021-06-22T13:27:53Z] 2021-06-22T13:27:53.856Z [DEBUG] raft: vote granted: from=pim-cpt-apps-t1 term=2369 tally=1
^CCaught signal: interruptINFO[0005] agent: Gracefully shutting down agent...
INFO[0005] Waiting for jobs to finish...
INFO[0005] No jobs left. Exiting.
INFO[2021-06-22T13:27:54Z] agent: Called member stop, now stopping node=pim-cpt-apps-t1
INFO[2021-06-22T13:27:54Z] 2021/06/22 13:27:54 [INFO] serf: EventMemberLeave: pim-cpt-apps-t1 192.168.66.64
DEBU[2021-06-22T13:27:54Z] 2021/06/22 13:27:54 [ERR] plugin: plugin server: accept unix /tmp/plugin736850028: use of closed network connection node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-log
2021/06/22 13:27:54 [ERR] yamux: Failed to write header: write unix @->/tmp/plugin736850028: write: broken pipe
DEBU[2021-06-22T13:27:54Z] 2021/06/22 13:27:54 [ERR] plugin: plugin server: accept unix /tmp/plugin179512185: use of closed network connection node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-fluent
DEBU[2021-06-22T13:27:54Z] received EOF, stopping recv loop err="rpc error: code = Unavailable desc = transport is closing" node=pim-cpt-apps-t1 subsystem_name=plugins.stdio
DEBU[2021-06-22T13:27:54Z] received EOF, stopping recv loop err="rpc error: code = Unavailable desc = transport is closing" node=pim-cpt-apps-t1 subsystem_name=plugins.stdio
DEBU[2021-06-22T13:27:54Z] received EOF, stopping recv loop err="rpc error: code = Unavailable desc = transport is closing" node=pim-cpt-apps-t1 subsystem_name=plugins.stdio
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-gcppubsub pid=5822
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-rabbitmq pid=5854
DEBU[2021-06-22T13:27:54Z] received EOF, stopping recv loop err="rpc error: code = Unavailable desc = transport is closing" node=pim-cpt-apps-t1 subsystem_name=plugins.stdio
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] received EOF, stopping recv loop err="rpc error: code = Unavailable desc = transport is closing" node=pim-cpt-apps-t1 subsystem_name=plugins.stdio
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-log pid=5804
DEBU[2021-06-22T13:27:54Z] 2021/06/22 13:27:54 [ERR] plugin: plugin server: accept unix /tmp/plugin332703956: use of closed network connection node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-syslog
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-syslog pid=5813
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-fluent pid=5795
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-http pid=5830
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] 2021/06/22 13:27:54 [ERR] plugin: plugin server: accept unix /tmp/plugin088594320: use of closed network connection node=pim-cpt-apps-t1 subsystem_name=plugins.dkron-processor-files
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-kafka pid=5838
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] received EOF, stopping recv loop err="rpc error: code = Unavailable desc = transport is closing" node=pim-cpt-apps-t1 subsystem_name=plugins.stdio
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-processor-files pid=5786
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-nats pid=5847
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
DEBU[2021-06-22T13:27:54Z] plugin process exited node=pim-cpt-apps-t1 path=/usr/bin/dkron-executor-shell pid=5863
DEBU[2021-06-22T13:27:54Z] plugin exited node=pim-cpt-apps-t1
zubair.patel@pim-cpt-apps-t1:~$
Same Question:
- docker-compose
version: "2.3"
services:
worker:
image: dkron/dkron:3.2.3
container_name: kettle-worker-1
hostname: kettle-worker-1
networks:
- net
ports:
- 8080:8080
- 8946:8946
- 8945:8945
volumes:
- /etc/localtime:/etc/localtime:ro
- dkron-resource:/opt/dkron-resource
- ./volume/dkron/data:/var/lib/dkron/data
command: |
agent
--log-level warn
--tag kettle=worker
--tag docker-cli=true
--tag python=true
--bind-addr 0.0.0.0:8946
--advertise-addr ${HOST_IP}:8946
--enable-prometheus
--disable-usage-stats
--server
--bootstrap-expect=1
--http-addr :8080
--rpc-port 8945
--data-dir /var/lib/dkron/data
networks:
net:
external:
name: infra
volumes:
dkron-resource:
- docker logs
$ docker logs -f kettle-worker-1
time="2023-04-23T10:09:13+08:00" level=info msg="No valid config found: Applying default values." error="Config File \"dkron\" Not Found in \"[/etc/dkron /root/.dkron /opt/dkron/config]\""
time="2023-04-23T10:09:42+08:00" level=error msg="no member leader found in member list" node=kettle-worker-1
time="2023-04-23T10:09:46+08:00" level=error msg="no member leader found in member list" node=kettle-worker-1
- dkron raft list-peers
root@kettle-worker-1:/dkron# ./dkron raft list-peers --rpc-addr localhost:8945
INFO[0000] No valid config found: Applying default values. error="Config File \"dkron\" Not Found in \"[/etc/dkron /root/.dkron /opt/dkron/config]\""
Node ID Address State Voter
(unknown) kettle-worker-1 [::]:8945 leader true
@cobolbaby dkron work with docker compose with default params, it looks like you are trying to work with a specific environment config --advertise-addr 192.168.16.116:8946 can you try removing any special networking params and check if it is able to elect a leader?
@vcastellm We have several VMs, and Dkron service will be deployed on each node. To build a cluster, it depends on the network of the host machine, so I configured the --advertise-addr option, which is essential.
I wonder if the no leader is caused by not exposing the udp protocol port? The Serf document mentions that the --advertise-addr port will receive packets of two protocols, TCP and UDP. But even if I open the udp port, the problem remains.
ports:
- 6230:6230
- 8946:8946
- 8946:8946/udp
- 8945:8945
@vcastellm The root cause was found after code debugging.
Since Dkron is deployed in a container, --bind-addr is either configured as 0.0.0.0:8946 or as ${CONTAINER_IP}:8946. But I don't want to rely on the cross-host container network, but only on the host network, so I configured the following parameters:
--bind-addr 0.0.0.0:8946
--advertise-addr ${HOST_IP}:8946
But it seems that leaderMember() function in the new version will cause the issue, while the old version can support such a configuration. member.Tags["rpc_addr"] is ${HOST_IP}:8945 and string(l) is [::]:8945.
https://github.com/distribworks/dkron/blob/9982ceb9af37543d5133e506899d245e8fa53b6e/dkron/agent.go#L611-L617