Shailen Sobhee

Impossible to test perfzero without docker

2

Execute: ` python3 perfzero/lib/benchmark.py --git_repos="https://github.com/tensorflow/models.git;benchmark" --python_path=models --data_downloads="gs://tf-perf-imagenet-uswest1/tensorflow/cifar10_data/cifar-10-batches-bin" --benchmark_methods=official.r1.resnet.estimator_benchmark.Resnet50EstimatorBenchmarkReal.benchmark_graph_1_gpu` Error: ``` [...] benchmarks/perfzero/lib/perfzero/utils.py", line 288, in run_commands cmd, exit_code, stdout)) Exception: "['gsutil', '-m', 'cp', '-r', '-n', 'gs://tf-performance/auth_tokens/benchmark_upload_gce.json', '/benchmarks/perfzero/workspace']" failed with code:1...

URL link in Readme is broken

3

From this README, at the end, there is a table: https://github.com/IntelAI/models/blob/master/docs/general/tensorflow/LaunchBenchmark.md Under the column "Run from the Model Zoo repository", most FP32 urls are broken. You may wish to check...

Please provide a way to get the plugin without a 3rd party package manager

2

Downloading the Chocolatey package management system, just for one plugin, is a massive overkill. It's also tricky to use it, behind proxies and all. Thank you for considering that.

Added more detailed metrics to logger that help characterize an LLM's performance.