Add perf benchmark tests for v2

[goodsong81]

Summary

[Updates]

• Adapt v2 regression tests to v1 perf benchmark convention

$ pytest tests/perf -h
...
Custom options:
  --benchmark-type={accuracy,efficiency,all}
    Choose accuracy|efficiency|all. Defaults to accuracy.
  --model-category={default,all}
    Choose default|all. Defaults to all.
  --data-size={small,medium,large,all}
    Choose small|medium|large|all. Defaults to all.
  --num-repeat=NUM_REPEAT
    Overrides default per-data-size number of repeat setting. Random seeds are set to 0 ~ num_repeat-1 for the trials. Defaults to 0 (small=3, medium=3, large=1).
  --num-epoch=NUM_EPOCH
    Overrides default per-model number of epoch setting. Defaults to 0 (per-model epoch & early-stopping).
  --eval-upto={train,export,optimize}
    Choose train|export|optimize. Defaults to train.
  --data-root=DATA_ROOT
    Dataset root directory.                                                                                                                                               
  --output-root=OUTPUT_ROOT
    Output root directory. Defaults to temp directory.                                                                                                                    
  --summary-csv=SUMMARY_CSV
    Path to output summary cvs file. Defaults to {output-root}/benchmark-summary.csv                                                                                      
  --dry-run
    Print OTX commands without execution.
  --deterministic
    Turn on deterministic training.
  --user-name=USER_NAME
    Sign-off the user name who launched the regression tests this time, e.g., `--user-name "John Doe"`.
  --mlflow-tracking-uri=MLFLOW_TRACKING_URI
    URI for MLFlow Tracking server to store the regression test results.

[Sample]

pytest -v -s tests/perf --benchmark-type efficiency --data-root /mnt/sdc/otx_v2_dataset/ --data-size all --model-category all --dry-run

[Future work]

• Export & optimize evaluation
• Reference criteria checking
• MLFlow logging
• Anomaly tests (to be added once the task is enabled)

How to test

[!] Only local CLI test is supported for now (WIP for CI)

# full train accuracy test
pytest -v -s tests/perf --benchmark-type accuracy --data-root /path/to/data/root

# default models only
pytest -v -s tests/perf --benchmark-type accuracy --data-root /path/to/data/root --model-category default

# small datasets only
pytest -v -s tests/perf --benchmark-type accuracy --data-root /path/to/data/root --data-size small

# efficiency test on large datasets only
pytest -v -s tests/perf --benchmark-type efficiency --data-root /path/to/data/root --data-size large

# dry run
pytest -v -s tests/perf --data-root /path/to/data/root --dry-run

Checklist

• [ ] I have added unit tests to cover my changes.​
• [ ] I have added integration tests to cover my changes.​
• [ ] I have added e2e tests for validation.
• [ ] I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• [ ] I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• [ ] I have linked related issues.

License

• [x] I submit my code changes under the same Apache License that covers the project. Feel free to contact the maintainers if that's a concern.
• [x] I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0