apple
Clients fail to send read requests to all replicas
We recently encountered a situation where a two-storage-server team was hot with reads. At some point, one of these storage servers started experiencing noticeably more reads than the other. We decided to exclude the worse one, and after data movement completed we ended up in a state where only a single storage server was hot. A subsequent exclude of the new process had the same outcome.
Eventually we bounced the clients, and load started to distribute evenly between the two storage servers again.
Is there a way to reproduce this behavior in our simulation test? I'm thinking loud: if we can record in a client which SS sends the data to it and check the number of replies from each SS in a team is "roughly" same, we might be able to reproduce it?
