How to process multiple benchmark files?

[Levi-Armstrong]

I have tried the GitHub action layout below but get the following error.

2021-06-09T15:56:24.8321073Z [command]/usr/bin/git -c user.name=github-action-benchmark -c [email protected] -c http.https://github.com/.extraheader= fetch origin gh-pages:gh-pages
2021-06-09T15:56:25.0151197Z From https://github.com/Levi-Armstrong/tesseract-1
2021-06-09T15:56:25.0152452Z  ! [rejected]          gh-pages   -> gh-pages  (non-fast-forward)
2021-06-09T15:56:25.0201890Z ##[error]Command 'git' failed with args '-c user.name=github-action-benchmark -c [email protected] -c http.https://github.com/.extraheader= fetch origin gh-pages:gh-pages': From https://github.com/Levi-Armstrong/tesseract-1
 ! [rejected]          gh-pages   -> gh-pages  (non-fast-forward)

     - name: Store Bullet Discrete Simple benchmark result
        uses: rhysd/github-action-benchmark@v1
        with:
          name: Bullet Discrete Simple C++ Benchmark
          tool: 'googlecpp'
          output-file-path: /home/runner/work/tesseract/tesseract/benchmarks/tesseract_collision_bullet_discrete_simple_benchmarks_results.json
          # Use personal access token instead of GITHUB_TOKEN due to https://github.community/t5/GitHub-Actions/Github-action-not-triggering-gh-pages-upon-push/td-p/26869/highlight/false
          github-token: ${{ secrets.GITHUB_TOKEN }} # GitHub API token to make a commit comment
          auto-push: false
          # Show alert with commit comment on detecting possible performance regression
          alert-threshold: '200%'
          comment-on-alert: true
          fail-on-alert: false
          alert-comment-cc-users: '@mpowelson'
          max-items-in-chart: 20

      - name: Store Bullet Discrete BVH benchmark result
        uses: rhysd/github-action-benchmark@v1
        with:
          name: Bullet Discrete BVH C++ Benchmark
          tool: 'googlecpp'
          output-file-path: /home/runner/work/tesseract/tesseract/benchmarks/tesseract_collision_bullet_discrete_bvh_benchmarks_results.json
          # Use personal access token instead of GITHUB_TOKEN due to https://github.community/t5/GitHub-Actions/Github-action-not-triggering-gh-pages-upon-push/td-p/26869/highlight/false
          github-token: ${{ secrets.GITHUB_TOKEN }} # GitHub API token to make a commit comment
          auto-push: false
          # Show alert with commit comment on detecting possible performance regression
          alert-threshold: '200%'
          comment-on-alert: true
          fail-on-alert: false
          alert-comment-cc-users: '@mpowelson'
          max-items-in-chart: 20

      - name: Store FCL Discrete BVH benchmark result
        uses: rhysd/github-action-benchmark@v1
        with:
          name: FCL Discrete BVH C++ Benchmark
          tool: 'googlecpp'
          output-file-path: /home/runner/work/tesseract/tesseract/benchmarks/tesseract_collision_fcl_discrete_bvh_benchmarks_results.json
          # Use personal access token instead of GITHUB_TOKEN due to https://github.community/t5/GitHub-Actions/Github-action-not-triggering-gh-pages-upon-push/td-p/26869/highlight/false
          github-token: ${{ secrets.GITHUB_TOKEN }} # GitHub API token to make a commit comment
          auto-push: false
          # Show alert with commit comment on detecting possible performance regression
          alert-threshold: '200%'
          comment-on-alert: true
          fail-on-alert: false
          alert-comment-cc-users: '@mpowelson'
          max-items-in-chart: 20

[esigo]

Here is an example: https://github.com/open-telemetry/opentelemetry-cpp/blob/0b1a74c7abd3375ef459b28f6b3536a42cf9e71a/ci/do_ci.sh#L40-L49

[Levi-Armstrong]

@esigo Does this create a single benchmark file which is to be uploaded?

[Levi-Armstrong]

I ended up writing a python script to join the files to avoid docker in docker.

#!/usr/bin/python3
import json
import os

build_dir = os.getcwd() + "/../../build"
result_files = []

search_path = build_dir + "/tesseract_collision/test/benchmarks"
for file in os.listdir(search_path):
    if file.endswith(".json"):
        result_files.append(os.path.join(search_path, file))

search_path = build_dir + "/tesseract_environment/test/benchmarks"
for file in os.listdir(search_path):
    if file.endswith(".json"):
        result_files.append(os.path.join(search_path, file))

cnt = 0
all_data = {}
for file in result_files:
    # Opening JSON file
    f = open(file, "r")

    # returns JSON object as a dictionary
    data = json.loads(f.read())

    if cnt == 0:
        all_data = data
    else:
        for benchmark in data["benchmarks"]:
            all_data["benchmarks"].append(benchmark)
        # endfor
    # endif

    # Closing file
    f.close()

    # Increment counter
    cnt+=1
# endfor

# the base path to where the output must be stored
if not os.path.exists("/root/benchmarks"):
    os.makedirs("/root/benchmarks")
# endif

# the json file where the output must be stored
out_file = open("/root/benchmarks/tesseract-benchmark_result.json", "w+")
json.dump(all_data, out_file, indent = 4)
out_file.close()

[edulix]

Thanks for the script @Levi-Armstrong , although it would be nice if the github action supported this natively.

[Levi-Armstrong]

Is there a blocker that is preventing this action from taking a list of files versus just a single file?

[si618]

Whilst it'd be nice to have this baked in, here's some C# code to combine benchmark results.

Not as pretty as @Levi-Armstrong's python solution, but seems to do the job if you work in dotnet 6+.

public static void CombineBenchmarkResults(
    string resultsDir = "./BenchmarkDotNet.Artifacts/results",
    string resultsFile = "Combined.Benchmarks",
    string searchPattern = "Foo.*.json")
{
    var resultsPath = Path.Combine(resultsDir, resultsFile + ".json");

    if (!Directory.Exists(resultsDir))
    {
        throw new DirectoryNotFoundException($"Directory not found '{resultsDir}'");
    }

    if (File.Exists(resultsPath))
    {
        File.Delete(resultsPath);
    }

    var reports = Directory
        .GetFiles(resultsDir, searchPattern, SearchOption.TopDirectoryOnly)
        .ToArray();
    if (!reports.Any())
    {
        throw new FileNotFoundException($"Reports not found '{searchPattern}'");
    }

    var combinedReport = JsonNode.Parse(File.ReadAllText(reports.First()))!;
    var title = combinedReport["Title"]!;
    var benchmarks = combinedReport["Benchmarks"]!.AsArray();
    // Rename title whilst keeping original timestamp
    combinedReport["Title"] = $"{resultsFile}{title.GetValue<string>()[^16..]}";

    foreach (var report in reports.Skip(1))
    {
        var array = JsonNode.Parse(File.ReadAllText(report))!["Benchmarks"]!.AsArray();
        foreach (var benchmark in array)
        {
            // Double parse avoids "The node already has a parent" exception
            benchmarks.Add(JsonNode.Parse(benchmark!.ToJsonString())!);
        }
    }

    File.WriteAllText(resultsPath, combinedReport.ToString());
}