Scale tests todo:

[Laremere]

In no particular order, probably missing some:

• [x] Get Metrics working.
• [x] Add metric for tickets created / deleted / assigned.
• [x] Add metric for matches received / assigned.
• [ ] Add metric for proportion of matches pass evaluation.
• [x] Add metric for mmlogic query time?
• [x] Add metric for iteration cycle time.
• [ ] Add extension to tickets with start time, add metric for ticket lifetime on matched.
• [ ] If possible, add metric for current ticket pool age lifetimes.
• [ ] Add mode for waiting for assignments on the frontend.
• [ ] Make frontend ticket creation a Poisson distribution.
• [x] Add scenario for running the whole pool through one mmf (ala firstmatch).
• [x] Add scenario for skill based matchmaking with regions and gamemodes.
• [x] (maybe) Add battle royal scenario.
• [ ] Report the scenario's running stats for various improvement proposals.
• [ ] Reduce interval between metric collection, move from 5m to 1m intervals (or less if possible) in grafana for scale only. (Increased data is fine for short term running scale tests...)

[sawagh]

Can we move this to v0.10 or do we expect anything more here today?

[Laremere]

• [ ] Add an optional delay between ticket being in a match, and getting assigned. In a real world scenario, a significant number of tickets might be in a pending state because getting servers takes some time.

[yfei1]

Moving this to v0.10

[Laremere]

Rundown of scenarios I want to run:

Breaking down the scale test into two halves: logical scenario: The MMF, Evaluator, tickets, and profiles. 3 flavors:

• First Match: one profile, mmf pairs tickets, drains pool with no special logic.
• Battle Royal: 20 regional profiles, tickets choose one region to search in. Matches of 100 players are made.
• Team Shooter: profiles are split on region, mode, and skill ranges. Tickets have a random skill, one or few regions, and one mode. Match function groups tickets with similar skill, evaluator ranks by lowest skill difference. Configurable number of regions, modes, skill splits.

infra scenario: How Open Match is being called. Flavors:

• (infra1) Frontend client creates tickets, backend client directly deletes tickets.
• (infra2) Frontend client creates tickets then polls for assignment before deleting the ticket. Backend assigns tickets.
• (infra3) Frontend client creates tickets then uses wait for assignment before deleting the ticket. Backend assigns tickets.

List of scenarios I want to run:

• The "control" scenario. Team shooter with a ~10 profiles, using infra1.
• control, but with first match instead
• control, but with battle royal instead
• control, but with team shooter having ~100 profiles.
• control, but with team shooter having ~1000 profiles.
• control, but with infra 2
• control, but with infra 3

That's 7 scenarios.

[Laremere]

Per discussion: Logical Scenario will become GameScenario. Infra Scenario will become UsagePattern.