cockroach roachtest: add impact tests

Adds a new latency sensitive test and a framework to run them in. Specifically it does a few things:

Gathers all the latency results back at the "roachtest client" rather than at the workload nodes by utilizing UDP to aggregate the results.
Steps the load back to 30% of the original load by measuring during the fill and then using that as a baseline to calculate from.
Collects selective profiles and adds them to the artifacts as part of the test run.

As of today, this test doesn't fail in the case of high latency as I'm working out the details of where to draw the threshold, but it has all the data so we can make that decision once we have more of these tests written.

May 23 '24 23:05 andrewbaptist

This change is

May 23 '24 23:05 cockroach-teamcity

@renatolabs can you take a look at one part of this commit:

https://github.com/cockroachdb/cockroach/pull/124636/commits/71b247190fc6de9741e94c0e523150a4fad36a54#diff-f24d39122407bf6046b58adbb2bd7ad996ccd3cc775c699d63e47eab82dac085R378

Should I consider moving these functions into the Cluster package. I could see something like this being really useful when we are trying to debug slow operations in some tests.

May 24 '24 12:05 andrewbaptist

Should I consider moving these functions into the Cluster package

It looks like the functions should already be ready to be consumed as-is (every test lives in the tests package). If you want/prefer, you could also create a specific package in roachtestutil for a separate namespace. I'd avoid adding even more functions to the cluster or test "interfaces" as they are already very large and we don't need functions at that level to enable sharing of functionality.

May 26 '24 11:05 renatolabs

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

Jun 06 '24 12:06 blathers-crl[bot]

@srosenberg This is the set of tests that measure the latency impacts from different perturbations.

Jul 09 '24 14:07 andrewbaptist

TFTR!

bors r=kvoli

Jul 09 '24 15:07 andrewbaptist

Build succeeded:

Jul 09 '24 16:07 craig[bot]