Add a feature to adapt max value of HPO batch size search space

[eunwoosh]

Summary

This PR adds new HPO feature. New feature can adapt max value of HPO batch size search space automatically. It provides user an easy way to avoid OOM due to setting wrong batch size search space. It exploits auto-batch size feature, so it has two ways to use.

• Safe : Just check max value of batch size search space is ok and decrease it if now.
• Full : Increase max value of batch size as possible as memory can handle.

How to test

  otx train --config ... --run_hpo true --hpo_config.adapt_bs_search_space_max_val Safe
  otx train --config ... --run_hpo true --hpo_config.adapt_bs_search_space_max_val Full

Checklist

• [x] I have added unit tests to cover my changes.​
• [ ] I have added integration tests to cover my changes.​
• [ ] I have ran e2e tests and there is no issues.
• [ ] I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• [ ] I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• [ ] I have linked related issues.

License

• [ ] I submit my code changes under the same Apache License that covers the project. Feel free to contact the maintainers if that's a concern.
• [ ] I have updated the license header for each file (see an example below).

# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 82.34%. Comparing base (45aea55) to head (ace2d59). Report is 1 commits behind head on releases/2.0.0.

Additional details and impacted files

@@                Coverage Diff                 @@
##           releases/2.0.0    #3532      +/-   ##
==================================================
- Coverage           83.03%   82.34%   -0.69%     
==================================================
  Files                 254      254              
  Lines               25266    25280      +14     
==================================================
- Hits                20979    20818     -161     
- Misses               4287     4462     +175     

Flag	Coverage Δ
py310	82.34% <100.00%> (-0.68%)	:arrow_down:
py311	?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.