pd icon indicating copy to clipboard operation
pd copied to clipboard

Random connect PD instance for the MemberList grpc call

Open wxiaomou opened this issue 11 months ago • 0 comments

Feature Request

Random connect PD instance for the MemberList call

Describe your feature request related problem

When PD looses leadership, all TiKV/TiDB pods start to call MemberList on PD/ETCD to find the leader. During incident we found that all these calls always go to a particular PD instance. It seems we always sort the PD members, thus when TiKV/TiDB trying to get the PD leaders they always reach to a specific PD pod which caused disproportional GRPC calls increase on the PD instance.

func (c *RaftCluster) Members() []*Member {
	c.Lock()
	defer c.Unlock()
	var ms MembersByID
	for _, m := range c.members {
		ms = append(ms, m.Clone())
	}
	sort.Sort(ms)
	return []*Member(ms)
}

Describe the feature you'd like

Describe alternatives you've considered

Teachability, Documentation, Adoption, Migration Strategy

wxiaomou avatar Feb 28 '25 00:02 wxiaomou