plt12138

v0.8.0 KeyError: 'builder_config' when benchmarking with new versions config.json

12

### System Info - CPU：4090 * 4 - TensorRT-LLm : v0.8.0 - CUDA Version: 12.3 - NVIDIA-SMI 545.29.06 ### Who can help? _No response_ ### Information - [X] The official...

v0.9.0 tensorrt_llm_bls model return error: Model '${tensorrt_llm_model_name}' is not ready.

1

### System Info TensorRT-LLM：v0.9.0 tensorrtllm_backend：v0.9.0 ### Who can help? @kaiyux ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks - [...

How to build the Mistral using BF16

7

I want to build the Mistral model using a AWQ and BF16. ``` python3 ../quantization/quantize.py --model_dir dolphin-2.6-mistral-7b-sft-yhy \ --dtype bfloat16 --qformat int4_awq --awq_block_size 128 \ --output_dir ./quantized_int4-awq-bf16 --calib_size 32 ```...