Using BenchmarkCI in Buildkite pipelines

[ven-k]

I tried using following script for running benchmarks on buildkite pipline

using Pkg

# Activate `benchmark`
Pkg.activate("benchmark")
Pkg.develop(path=".") # Add parent Project.toml to development env 
Pkg.instantiate()
Pkg.build()
Pkg.precompile()

# Start benchmarking
using BenchmarkCI
BenchmarkCI.judge()
BenchmarkCI.postjudge()

But it errors with:

Activating environment at `/<buildkite-path>benchmark/Project.toml`
:
Installed Parsers ──────────────────── v2.1.2
:
✓ Parsers
:
 15 dependencies successfully precompiled in 12 seconds (58 already precompiled)
:
[ Info: Using existing manifest file.
--
  | PkgBenchmark: Running benchmarks...
  | Activating environment at `<buildkite-path>/benchmark/Project.toml`
  | PkgBenchmark: creating benchmark tuning file /<buildkite-path>/benchmark/tune.json..
(1/3) tuning "new"...
:
:
PkgBenchmark: benchmark results written to .benchmarkci/result-target.json

  | PkgBenchmark: Running benchmarks...
  | ERROR: LoadError: LoadError: ArgumentError: Package Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] is required but does not seem to be installed:
  | - Run `Pkg.instantiate()` to install all recorded dependencies.

( : to be replaced with entire package list)

The issue isn't specific to Parsers, but its just something it is trying to find first.

---

• BenchmarkCI does an impressive job with Github Actions.
• If this could be reused with Buildkite runners (which is widely being adopted), it would be great.
• While here, postjudge() is something to inspect if it can post even from a buildkite pipeline

[tkf]

Hmm... It's a bit puzzling that the error occurs at such a late phase. Is this from a public repository? Can I see a CI log?

[ven-k]

Sadly its a private repo.