SmartSim icon indicating copy to clipboard operation
SmartSim copied to clipboard

More granular installation of SmartSim

Open ashao opened this issue 3 years ago • 1 comments

Description

SmartSim's two-step install process (a 'pip install' followed by the smart build step) can inadvertently change versions of packges that exist in already established conda or Python virtual environments. This is particularly a problem in the smart build step where we rely on fixed versions of the ML backends for compatibility with RedisAI. While we currently allow users the option to specify a path to tensorflow and torch installations, smart build will currently attempt to install the torch package.

During the pip install phase, these ML packages are only installed if the [ml] extras is specified. The default SmartSim install thus has a minimal number of dependencies, but the documentation needs to be updated to explain the SmartSim capabilities that are enabled by the different variants of both the pip install and smart build.

Justification

Give the users guidance and control over what python packages are installed during the SmartSim install. Users will thus more likely be able to incorporate SmartSim into existing python environments and reduce the potential for dependency conflicts. This is particularly pertinent for users who may have compiled custom versions of python or ML libraries optimized for their systems and hardware.

Implementation Strategy

  • [x] Include smart build option to never install Python packages, but issue a warning that this may lead to undesirable behavior
  • [x] Add documentation to describe what each of the 'extra' options installs during the pip install phase
  • [ ] Describe the steps needed to enable a specific functionality of SmartSim (i.e. pip install smartsim will allow users to launch experiments and the orchestrator, but not allow for ML models to be loaded or run from the database)

ashao avatar Aug 04 '22 21:08 ashao

With the merge of #336, SmartSim will no longer fetch ML deps within smart build, meaning there is no risk that a user may accidentally alter their existing python environment via smart.

If a python ML dep is missing or an unexpected version is detected during smart build a warning will be issued, but it will not stop the user from building the ML backends for SmartSim.

Further documentation regarding the smart build (particularly a demonstration showing the --torch_dir and --libtensorflow_dir flags) could be useful in showing users how they may build SmartSim against newer versions of Torch/Tensorflow.

MattToast avatar Aug 26 '23 00:08 MattToast