vnkc1

Mixtral 8x7B smoothquant failed

2

### System Info p4de (4 80GB A100 GPUs) ### Who can help? @Tracin @byshiue ### Information - [X] The official example scripts - [ ] My own modified scripts ###...

Mixtral engine build gives CUDA OOM on 8 40GB GPUs (0.8.0 release)

3

### System Info p4d with 8 GPUs : NVIDIA A100 40GB x 8 package version tensorrt-9.2.0.post12.dev5-cp310-none-linux_x86_64.whl [TensorRT-LLM] TensorRT-LLM version: 0.8.00.8.0 ### Who can help? @byshiue ### Information - [X] The...

How do I specify `max_tokens_in_paged_kv_cache` property during trtllm generation?

2

### System Info n/a ### Who can help? @byshiue ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks - [ ]...

### System Info A100 GPUs (40GB) ### Who can help? @byshiue ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks -...

Llama-2 13B SmoothQuant W8A8 Per-Tensor TP-4 performance is poor in v0.9.0 release

7

### System Info GPUs: A100, 4 GPUs (40 GB memory) Release: tensorrt-llm 0.9.0 ### Who can help? @Tracin ### Information - [X] The official example scripts - [ ] My...

Example `gpu_device_ids` for multi-model usage?

1

### System Info P4D (A100 40 GB x 8) ### Who can help? @juney-nvidia @byshiue ### Information - [X] The official example scripts - [ ] My own modified scripts...