Add integration test for CPU performance guest vs baremetal host

[alxiord]

Add a test case that exercises the same CPU performance test suite both on a Firecracker guest and on the host it's running on. The guest and host need to have the same shape in order for the test results to be conclusive. The performance numbers obtained on the guest must be at least 95% of their corresponding values from the host.

[goandrei]

Hi,

I've done a quick research on what tools we could use on measuring the CPU performance. I found the following ones the most interesting:

• stress-ng : has multiple CPU stress tests(around 60) testing pi generation, linear algebra computations, sqrt etc. It also supports jobfiles which can make the addition of a new test easier. It also includes tests for Cache thrashing, Context switching, Virtual Memory, Sockets, I/0 etc. It may be used for testing other features in the future.
• p7zip : a compression/decompression based test.
• sysbench : generates all the prime number up to a limit N. There is a similar test in stress-ng.
• superPI : generates PI using Gauss–Legendre algorithm. There is a similar test in stress-ng.
• mmtests : more like a workload test suite, but still, it may be useful in the future for performance testing.
• Dhrystone : it tests the CPU's performance on integer computations. It was made in 1984 and some said that it might show misleading results. Also, there are a lot of computation tests on a lot of data types(including integers) in stress-ng.

My final choice would be stress-ng since it covers the use case of most of the other tools and it is also quite flexible. It offers a big range of tests using different approaches of stressing the CPU which may result in more accurate numbers since we use a great variety of cases. I was thinking about creating a config file(the so-called jobfile in stress-ng docs) since it will simplify an eventual addition of a new test case in the future.

[serban300]

Can we also try NAS-PB ?

[pb8o]

Resolving for now as we don't have a good example use case. For what's worth I run a simple FFT test with stress-ng and I don't see much difference:

Host:

% stress-ng --cpu 2 --cpu-method fft --cpu-ops 100000 --metrics-brief
stress-ng: info:  [1434384] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [1434384] dispatching hogs: 2 cpu
stress-ng: info:  [1434384] successful run completed in 25.26s
stress-ng: info:  [1434384] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: info:  [1434384]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: info:  [1434384] cpu              100000     25.20     50.31      0.02      3968.45        1986.89

Guest:

% stress-ng --cpu 2 --cpu-method fft --cpu-ops 100000 --metrics-brief
stress-ng: info:  [1469] defaulting to a 1 day, 0 secs run per stressor
stress-ng: info:  [1469] dispatching hogs: 2 cpu
stress-ng: metrc: [1469] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [1469]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [1469] cpu              100000     22.26     44.49      0.00      4493.14        2247.62
stress-ng: info:  [1469] skipped: 0
stress-ng: info:  [1469] passed: 2: cpu (2)
stress-ng: info:  [1469] failed: 0
stress-ng: info:  [1469] metrics untrustworthy: 0
stress-ng: info:  [1469] successful run completed in 22.35 secs