dbrx issues

How inference efficiency is measured

11

The tech report described the methodology of the inference efficiency measurement but not in detail. It compared the Llama2-70B and DBRX. We have great interests in the comparison. So we...

FC-Li

Missing tokenizer when use vllm

4

``` File "/home/paas/vllm/vllm/engine/llm_engine.py", line 222, in _init_tokenizer self.tokenizer: BaseTokenizerGroup = get_tokenizer_group( File "/home/paas/vllm/vllm/transformers_utils/tokenizer_group/__init__.py", line 20, in get_tokenizer_group return TokenizerGroup(**init_kwargs) File "/home/paas/vllm/vllm/transformers_utils/tokenizer_group/tokenizer_group.py", line 23, in __init__ self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config) File...

matrixssy

What's the optimal parallel strategy using TensorRT-LLM?

2

Thanks for your great efforts first. I read the PR you opened in the TensorRT-LLM repo and noticed that EP +TP, PP + TP, and TP are supported during inference....

iteratorlee

Loading over multiple gpus in 8bit and 4bit with transformers loader

4

I can load the instruct model using the transformers loader and 8bit bits and bytes, I can get it to load evenly among multiple gpus. However, I cannot seem to...

RandomInternetPreson