glass-dev
How to benchmark functions that return generators
Is Your Feature Request Related to a Problem? Please Describe
Naively wrapping a function in the benchmark fixture does not provide a true benchmark of the function as each timed execution will only run until the first yield.
Describe the Solution You'd Like
A clear way to benchmark functions which return generators. Possibly wrapping the full test in the benchmark decorator?
Describe Alternatives You've Considered
We could instead not benchmark functions that return generators or stop returning generators from functions.
Additional Context
No response
Well, the random sampling of the simulated fields is a generator, so not timing that would be problematic :)
Why can we not consume the entire generator in the test?
def benchmark(...):
for _ in returns_generator():
pass
@ntessore Some of the existing tests result in an error if we consume the entire generator. For example, consuming everything returned from iternorm usually results in the error "covariance matrix is not positive definite"
I suppose we could do something like this? Does anyone see an issue with this?
def test_iternorm_no_size(xp: ModuleType, benchmark: BenchmarkFixture) -> None:
"""Benchmarks for glass.iternorm with default value for size."""
# Call jax version of iternorm once jax version is written
if xp.__name__ == "jax.numpy":
pytest.skip("Arrays in iternorm are not immutable, so do not support jax")
# check output shapes and types
k = 2
array_in = [xp.asarray(x) for x in xp.arange(10_000)]
def function_to_benchmark(array_in):
output = []
generator = glass.fields.iternorm(
k,
(x for x in array_in),
)
try:
for result in generator:
output.append(result)
except:
pass
return output
results = benchmark(function_to_benchmark, array_in)
j, a, s = results[0]
assert isinstance(j, int)
assert a.shape == (k,)
assert s.shape == ()
assert s.dtype == xp.float64
assert s.shape == ()
What are the results if you were to run that?
The tests passes and on my local machine it takes about 55 us
The table for just the iternorm tests where I have implemented something like the above in this branch is shown below.
------------------------------------- benchmark: 8 tests -------------------------------------
Name (time in ms) Mean StdDev Rounds
----------------------------------------------------------------------------------------------
test_iternorm_k_0[array_api_strict] 36.7713 (699.90) 0.3660 (131.53) 25
test_iternorm_k_0[numpy] 3.7731 (71.82) 0.1363 (48.97) 241
test_iternorm_no_size[array_api_strict] 0.8354 (15.90) 0.1118 (40.17) 753
test_iternorm_no_size[numpy] 0.0525 (1.0) 0.0028 (1.0) 2342
test_iternorm_specify_size[array_api_strict-1] 2.9070 (55.33) 0.0470 (16.89) 306
test_iternorm_specify_size[array_api_strict-2] 2.9038 (55.27) 0.0966 (34.71) 293
test_iternorm_specify_size[numpy-1] 0.2197 (4.18) 0.0178 (6.40) 3268
test_iternorm_specify_size[numpy-2] 0.1980 (3.77) 0.0108 (3.88) 3732
----------------------------------------------------------------------------------------------