transformers-stream-generator

transformers-stream-generator copied to clipboard

spacing problem.

4

Thanks to your nice works! but I met some problem as spacing each token. for example... ... 'on' 'st' 'amps' 'and' 'sh' 'ipping' '.' ... stamps is one word and...

circuluspibo

ziyue246

使用tar.gz文件安装时会出现以下问题： Building wheel for transformers-stream-generator (setup.py) ... error error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [7 lines of output] /opt/conda/lib/python3.10/site-packages/setuptools/dist.py:723: UserWarning: Usage...

Lizhecheng02

main.py line 987 should be origin code will multiply unfinished_sequences with a array which elements not only the 0 or 1 for example eos_token_id = [1,2] next_tokens = torch.LongTensor([1,2,3,4,5]) sum(next_tokens...

llltttppp

do_sample=False

2

在`do_sample=True`时可以流式输出，设置为False时就一次输出了。

vicwer

generate 函数中 is_greedy_gen_mode 和 is_sample_gen_stream_mode 同时为 True

1

pip install transformers_stream_generator==0.0.4 后调试 llama 时，发现若使用如下命令 ````py tokens = None for token in torch_model.generate( input_ids=input_ids, max_length=1024, num_beams=1, num_return_sequences=1, no_repeat_ngram_size=15, repetition_penalty=1, temperature=0.65, do_stream=True): if tokens is None: tokens = token else:...

AIxyz

Token Yielding Problem

1

The Code: ```py from transformers import AutoTokenizer, TextGenerationPipeline, TextStreamer, GenerationConfig from auto_gptq import AutoGPTQForCausalLM import torch from transformers_stream_generator import init_stream_support init_stream_support() repo = "TheBloke/tulu-7B-GPTQ" model_basename = "gptq_model-4bit-128g" test_tokenizer = AutoTokenizer.from_pretrained(...

xdevfaheem

支持bloom吗

1

使用bloom报错 print(tokenizer.decode(result, skip_special_tokens=True)) File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3477, in decode return self._decode( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 549, in _decode text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens) TypeError: argument 'ids': 'list' object cannot be interpreted as...

kevinuserdd

transformers_stream_generator/main.py:139: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation) warnings.warn(

1

transformers_stream_generator/main.py:139: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please...

lucasjinreal

The generator doesn't work if set num_beams not equals 1

3

if I set num_beams=3 or other number that not equals 1, this tools will not work, and the generator can not output as stream, how to solve this?

zoubaihan