Parametrize sleep intervals

[al-rigazzi]

Description

Currently, SmartSim's code contains a few sleep commands, which are used to wait for jobs to spin, files to be written, and so on. For the sake of compatibility with slower WLMs or file systems, the sleep intervals are slightly larger than what some users would like to see for HPC applications. A simple parameter (similar to the one available for JobManagers logging) would help demanding users reducing spin-up time, when this is possible. SmartSim could also benefit from it in general, as it would be possible to reduce the default wait time and let users know they can increase it (this has to be investigated). It could also speed up testing, as we could lower the wait time for CI runs.

Justification

Users who have a setup allowing for shorter wait times will benefit from a quicker and more reactive job launch.

Implementation Strategy

All hard-coded ints passed to sleep commands in the code base should be identified, categorized (e.g. file_system_sleep, wlm_sleep, ...), and replaced with configuration parameters. A subset of the sleep interval parameters should be exposed to the user, while parameters which are thought to be hard-coded will not be exposed, but unified and documented internally.

Acceptance criteria

• [ ] All sleep intervals are parametrized according to their goal
• [ ] All parameters are clearly documented
• [ ] Some of the parameters are exposed to the user