tile38 icon indicating copy to clipboard operation
tile38 copied to clipboard

Follower offset

Open damirda opened this issue 4 years ago • 2 comments

Is your feature request related to a problem? Please describe.

I have some Tile38 servers with very high frequency updates, so I want to make sure that replicas are up to date, before I offload some of the queries from the master.

I tried to find any metrics about the offset of follower, but I couldn't find any? info replication in redis returns offset for every follower and master.

role:master
connected_slaves:2
slave0:ip=172.23.0.3,port=6379,state=online,offset=56,lag=0
slave1:ip=172.23.0.4,port=6379,state=online,offset=56,lag=0
...
master_repl_offset:56

In Tile38, some data is missing:

  "info": {
    "connected_slaves": 2,
    "role": "master",
    "slave0": "ip=172.21.0.3,port=9851,state=online",
    "slave1": "ip=172.21.0.4,port=9851,state=online"
  }

I checked recently added Prometheus metrics as well, nothing about replica lag.

Describe the solution you'd like Either offset as a part of info replication or some other metric for follower lag.

Describe alternatives you've considered No ideas, honestly. :-)

damirda avatar Jun 14 '21 03:06 damirda

Currently, a Tile38 leader does not track the offsets or lag of the followers.

But Tile38 does have the SERVER command which returns the "aof_size" field.

You may be able to use the "aof_size" of the leader and each follower to calculate the offset.

tidwall avatar Jun 15 '21 12:06 tidwall

We execute AOFSHRINK from time to time, so aof_size is not really precise indicator. I'll check how hard would it be to add some offset/lag. Thank you for the clarification.

damirda avatar Jun 15 '21 16:06 damirda