Liwei Peng

@timoschick The PET results are very impressive for fewshot learning. For our scenario, we have big collection of data in classes that are related to our target fewshot classes, so...

Hi, Thanks for publishing PET's excellent results on RAFT benchmark in your recent ["True Few-Shot Learning with Prompts – A Real-World Perspective"](https://arxiv.org/abs/2111.13440) paper. The best practices in the paper are...

superbench logging: 'color' output should be optional

3

In current [superbench code](https://github.com/microsoft/superbenchmark/blob/main/superbench/common/utils/logging.py#L74), the logger has a hard-coded 'color' mode. While this design is good for interactive mode, it is bad when processing the logs programmatically in system like...

superbench mpi job should use the proper Ethernet interface

1

Several superbench tests (e.g. nccltests, ib-traffic ) use openmpi to launch the tests in multiple nodes. Some node type is designed to have multiple ethernet interfaces (e.g. azure2, eth0, docker0,...

ib validation benchmark should support mixed IB device naming schema

1

This is for superbench latest code. Current superbench ib validation benchmark is designed to have consistent IB device names across the nodes. A user must specify this name or a...

When superbench (version 0.6.0-rc1) runs a test, the test output is saved in some memory. It didn't flush the log to output file. This is hard for users to track...

**What's the issue, what's expected?**: This is superbench release/0.6. The 'sb result diagnosis' command didn't exit with error code properly when input data file has wrong format. e.g. for an...

why '/dev/nvidia-uvm' is a required file check for nvidia GPU?

2

**What's the issue, what's expected?**: This is v0.6.0. In (superbench code gpu.py) [(https://github.com/microsoft/superbenchmark/blob/main/superbench/common/devices/gpu.py#L24), it checks whether a GPU is nvidia by checking both '/dev/nvidiactl' and '/dev/nvidia-uvm'. The question is: why...

sb should return non-zero exit code when executor.py failed

1

**What's the issue, what's expected?**: This is using v0.6.0 release. The benchmark gemm-flops is run on a platform where the GPU is probably not supported (Tesla K80). The superbench has...

## Summary This issue was found using v0.6.0 release. In the system, the ansible was not setup properly because of a test environment issue. When I ran 'sb deploy -f...