Unable to compare summary results against specific ref: instead of previous ref

[darrenge]

I was trying to implement this one:https://github.com/benchmark-action/github-action-benchmark/pull/163 where I want to compare and display the results of the current commit to the results of commit ID: 9f43c15bbe42c350f93bd7494d2918974f1963b1 instead of the Previous commit (607222ae0d077886ea469a720865ec9cc56a77fe) in the summary part of the perf workflow.

I just added "ref:" at the bottom of it.

Maybe PR 163 is not actually implementing this and it isn't possible to do what I want it to do.

This is on the summary page:

Benchmark suite	Current: 612f42b21f8b6347229a05032e451dddfd3823c8	Previous: 607222ae0d077886ea469a720865ec9cc56a77fe	Ratio
BDN.benchmark.Operations.BasicOperations.InlinePing(Params: ACL)	1748.2839510599772 ns (± 15.153228689520862)	1817.795346069336 ns (± 11.748397682141338)	0.96

My YML code:

      # Run `github-action-benchmark` action for the Continuous Benchmarking Charts (https://microsoft.github.io/garnet/charts/)
      - name: Store benchmark result for charts
        uses: benchmark-action/github-action-benchmark@v1
        with:
          name: ${{matrix.test}} (${{matrix.os}}  ${{matrix.framework}} ${{matrix.configuration}})
          tool: 'benchmarkdotnet'
          output-file-path: ./test/BDNPerfTests/BenchmarkDotNet.Artifacts/results/BDN.benchmark.${{ matrix.test }}-report-full-compressed.json
          github-token: ${{ secrets.GITHUB_TOKEN }}
          summary-always: true
          gh-pages-branch: 'continuousbenchmark'
          benchmark-data-dir-path: 'website/static/charts'
          auto-push: true
          max-items-in-chart: 50
          ref: 9f43c15bbe42c350f93bd7494d2918974f1963b1

[darrenge]

Looking at my chart and the Summary, it looks like when I use "ref:" it actually just makes the "current" commit ID to that reference commit ID. I was hoping that the "ref:" (or some other method) would make "ref:" be the "Previous" so we can compare our current commit to a known commit ID. Is this the way it was designed or maybe I am doing something wrong.

So if I set ref to an older commit (43ccb9179e3c0605133527bc74352c2e783d96a9)

I get:

Operations.BasicOperations (ubuntu-latest net8.0 Release)

Benchmark suite	Current: 43ccb9179e3c0605133527bc74352c2e783d96a9	Previous: 2e2d5be893688a45e1e474bfe3fc91b59503fa81	Ratio
BDN.benchmark.Operations.BasicOperations.InlinePing(Params: ACL)	1779.5574655532837 ns (± 16.755712929111606)	1747.3529995509557 ns (± 15.165183223620078)	1.02
BDN.benchmark.Operations.BasicOperations.InlinePing(Params: AOF)	1761.639777501424 ns (± 16.90147161207196)	1734.16211160024 ns (± 4.226903598447717)	1.02
BDN.benchmark.Operations.BasicOperations.InlinePing(Params: None)	1761.3406321207683 ns (± 14.816458751802491)	1747.3257058461506 ns (± 13.906530528394674)	1.01