Allow multiple perf_patterns on one line in handlers_perflog

[stevenvdb]

When specifying multiple entries in perf_patterns (for instance as in cscs-checks/microbenchmarks/mpi/halo_exchange/halo_cell_exchange.py), every performance variable is put on a separate line by the logging handler. All attributes that are not related to the performance variable are repeated on all those lines.

For the HaloCellExchangeTest you could argue that every performance represents a separate aspect of the test, so it makes sense to put them on separate lines. However, when you are for instance looking at walltime, CPU time, communication time, memory, ... in one test, it would make more sense to keep those performance variables tied together. At the moment, it is necessary to join lines on some common attribute (such as jobid) to find out which results belong together.

I think it would be a useful feature to allow the %(check_perf_*)s attributes to report multiple values, similar to the %(check_tags)s attribute. Before attempting to implement this, I would like the opinion of a developer on this. Or maybe there is a hack to work around this issue?

[vkarak]

Hi @stevenvdb, what you propose here makes sense and I don't see for the moment a quick hack around it, because currently ReFrame does a separate log call for each performance variable. I could see that as a separate logging format specifier, perhaps %(check_perf_data)s, that would format all the performance data at once. If you plan to work on that, let me know.

[jgphpc]

For example, this config:

            'handlers_perflog': [
                {
                    'type': 'filelog',
                   ...
                   'format': '%(check_job_completion_time)s|reframe %(version)s|%(check_info)s|jobid=%(check_jobid)s|num_tasks=%(check_num_tasks)s|%(check_perf_patterns)s|%(check_perf_unit)s',

and this check:

 self.perf_patterns = {
            't0': sn.extractsingle(regex0, self.stdout, 1, int),
            't1': sn.extractsingle(regex1, self.stdout, 1, int)
        }

        self.reference = {
            '*': {
                't0': (0, None, None, 's'),
                't1': (0, None, None, 's'),
            }
        }

will report:

2021-03-09T06:58:43+01:00|reframe 3.6.0-dev.0+a3d0b0cd|XTest on pilatus:login using PrgEnv-gnu|jobid=197658|num_tasks=1|{"t0": 1, "t1": 2}|s

[vkarak]

@jgphpc Yes, and if you want the references, you can use %(check_reference)s in your format specification.