membershipChanged -> ringChanged

[kriskowal]

Evidently, ringChanged is a subset of membershipChanged (membership only grows), and affinity would only change in response to a ringChanged event. We should carefully consider using ringChanged instead to avoid unnecessary affinity updates in service proxy.

[jwolski]

I support this issue

[Raynos]

:+1:

As mentioned it would be nice to get stats into how often a ring change happens and what "damage" or side effect it causes on the hyperbahn network.

[jwolski]

@Raynos Ringpop has got ring.server-added and ring.server-removed metrics (counts) which are emitted every time the ring rebalances (in either direction more/less capacity). By "damage" I'm guessing you mean total percentage of ring rebalancing? If so, we can do that too. Though it's not currently there.

[Raynos]

@jwolski By damage I meant measuring impact.

• Amount of socket thrashing in hyperbahn exit workers
• Amount of Declined Error Frames caused to applications.
• Increased latency due to retries

etc. it would be good for us to understand how much work the hyperbahn ring has to do and how it impacts traffic each time a ring change occurs.

These are really hyperbahn specific metrics and are not a ringpop concern or issue.

[jwolski]

@Raynos :+1:

[jcorbin]

@jwolski , since I've taken over this can of worms, I'd like to understand the difference between the "membershipChanged" and "ringChanged" events:

• it's unclear to me from the above: which one fires more often? (i.e. it sounds like one of them is abstracting over details, what are those details)
• we're currently lacking a good test for our current "membershipChanged" semantics, does switching to "ringChanged" make it easier, harder, or no different to produce in a test?

[jcorbin]

@rf, I started trying to write a membershipChanged test here: #116