DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Does deepspeed-mii support prefix_allowed_tokens_fn?

Open zcakzhuu opened this issue 1 year ago • 0 comments

I use transformers pipeline to generate json dictionaries and I need to specify a prefix_allowed_tokens_fn such that the tokens that can be generated at some steps are fixed. By looking into the source code, it doesn't seem like the deepspeed pipeline support this. Could someone verify whether it is supported please?

zcakzhuu avatar May 14 '24 08:05 zcakzhuu