pytorch
CPU Launcher fails available check with venv
🐛 Describe the bug
cpu launcher doesn't work
Error logs
2024-03-28T08:00:31,792 [DEBUG] W-9000-embeddings_1.1 org.pytorch.serve.wlm.WorkerLifeCycle - launcherAvailable cmdline: [python, -m, torch.backends.xeon.run_cpu, --use_logical_core, --enable_jemalloc, --no_python, hostname]
2024-03-28T08:00:31,821 [WARN ] W-9000-embeddings_1.1 org.pytorch.serve.wlm.WorkerLifeCycle - torch.backends.xeon.run_cpu is not available. Proceeding without worker core pinning. For better performance, please make sure torch.backends.xeon.run_cpu is available.
launcherAvailable cmdline is running outside the virtual env of torchserve like the below command which fails
python -m torch.backends.xeon.run_cpu --use_logical_core --no_python hostname
/usr/bin/python: Error while finding module specification for 'torch.backends.xeon.run_cpu' (ModuleNotFoundError: No module named 'torch')
Instead using the virtual env where the torchserve is running works
[rocky@hostname ~]$ source /opt/venv-torchserve/bin/activate
(venv-torchserve) [rocky@hostname ~]$ python -m torch.backends.xeon.run_cpu --use_logical_core --no_python hostname
2024-03-28 08:04:22,935 - __main__ - INFO - Use TCMalloc memory allocator
2024-03-28 08:04:22,936 - __main__ - INFO - OMP_NUM_THREADS=4
2024-03-28 08:04:22,936 - __main__ - INFO - Using Intel OpenMP
2024-03-28 08:04:22,936 - __main__ - INFO - KMP_BLOCKTIME=1
2024-03-28 08:04:22,936 - __main__ - INFO - LD_PRELOAD=/opt/venv-torchserve/lib/libiomp5.so:/usr/lib64/libtcmalloc.so
2024-03-28 08:04:22,936 - __main__ - INFO - numactl -C 0-3 -m 0 hostname
Installation instructions
Torchserve 0.10.0 installed in a virtual env with pip
Model Packaging
not applicable
config.properties
ipex_enable=true
cpu_launcher_enable=true
cpu_launcher_args=--use_logical_core
Versions
TorchServe 0.10.0 Python 3.9
Repro instructions
Start a torchserve from a virtual env with cpu_launcher_enable true
Possible Solution
i think https://github.com/pytorch/serve/blob/v0.10.0/frontend/server/src/main/java/org/pytorch/serve/wlm/WorkerLifeCycle.java#L69 should be updated to use python environment used by the running torchserve
@ElfoLiNk ipex_enable=true should be set. Please check https://github.com/pytorch/serve/blob/master/examples/intel_extension_for_pytorch/README.md#torchserve-with-launcher
Hi @lxning, I already have the ipex_enable=true. I updated the bug adding more info
