并发请求stream报错

[liHai001]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• [X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• [X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

多个请求同时调用stream:

model.chat( image=images, msgs=prompt, tokenizer=tokenizer, sampling=True, stream=True):

其中，model定义： model = AutoModel.from_pretrained(MODEL_PATH, quantization_config=q_config, low_cpu_mem_usage=True, torch_dtype=torch.float16, trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True, use_fast=True)

报错： Exception in thread Thread-7: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/usr/local/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/app/.venv/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/app/.venv/lib/python3.8/site-packages/transformers/generation/utils.py", line 1914, in generate result = self._sample( File "/app/.venv/lib/python3.8/site-packages/transformers/generation/utils.py", line 2693, in _sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0

期望行为 | Expected Behavior

No response

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):

备注 | Anything else?

No response