Hudayday

examples/Inference failed

2

File "/home/meisme/PiPPy/examples/inference/hf_generate.py", line 12, in import pippy.fx ModuleNotFoundError: No module named 'pippy.fx' The pippy.fx seems appear nowhere.

Llama 2 Execution Bug

4

### System Info CPU: x86_64, memory: 1024GB, GPU: 8*A6000 48GB each, Tensorrt-LLM version 0.9.0.DEV20240226. NVIDIA-Driver Version: 535.171.04 CUDA Version: 12.2; OS - Ubuntu 22.04 ### Who can help? _No response_...

How does auto_parallel work?

1

Hi, I just noticed that auto_parallel support is available. However, after reviewing the code, I still have no idea how it determines the best configuration. It seems to solve the...

Unsupported auto parallel + int4 quantization on models

1

### System Info Tensorrt-LLM rel 0.9.0 ### Who can help? @Tracin ### Information - [X] The official example scripts - [ ] My own modified scripts ### Tasks - [X]...

time python ../vidur/profiling/mlp/main.py --models meta-llama/Llama-2-70b-hf --num_gpus 8 > mlp.txt && time python ../vidur/profiling/attention/main.py --models meta-llama/Llama-2-70b-hf --num_gpus 8 > att.txt 0%| | 0/1044 [00:00