cloud-bulldozer
Node density heavy updates
Description
Let's try to make node-density-heavy great again. The main problem of this benchmark is that it can be "too heavy" depending on how the pods are scheduled. In the previous implementation we've seen several posgres databases being scheduled to a certain node in a short period of time. That generated a very high CPU load in the node, leading to huge increases in the time taken by the workload.
The problem with the previous is that results were not very deterministic, since the test duration was very volatile.
This PR reduces the number of postgres databses deployed and increases the number of client applications. Now every 2 pair of clients will connect to the same database randomly using one of the two created services in each job iteration.
After executing this test several times in a baremetal based environment with 26 worker nodes, I've verified that results are more stable and reliable:
(UUIDs from today 14th of Sept)
- 8c5f3552-b0be-4b21-af09-bb242f79c9b6
- fae1097f-ee69-4f03-bf38-02d33e0a1480
- 3d181ece-f13a-4889-87b9-8c531e65c7c0
All the tests above were executed in ~600secs and podLatency was quite stable as well, < 3mins in all cases
I think it would be useful to have more results from other platforms, if possible, to ensure the test is stable enough
This change introduces more nuances that we would need to account for. In a relatively small test, I saw 9 perfapp pods restart because of concurrency failures creating the euler table in postgres (one of the following errors)
time="2022-09-28 16:27:09" level=fatal msg="pq: duplicate key value violates unique constraint \"pg_type_typname_nsp_index\""
time="2022-09-28 16:28:20" level=fatal msg="pq: type \"euler\" already exists"
I know that I originally asked to make it "less heavy", but I vote to keep it as it is with 1:1 ratio, since it allows us to tightly control the load generated on the nodes.
We can instead invest the effort in developing other workloads that stress other parts of the system, simulate more diverse real-world applications, etc.
