DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard
verified publisher icon microsoft

Server turns into broken state if queried with very long prompt

Open ttim opened this issue 2 years ago • 1 comments

I've queried server with ~7K character prompt, server turned into some broken state afterwards. The only fix I found is to restart MII server

ttim avatar Nov 10 '23 23:11 ttim

Hi @ttim thank you for reporting this issue. Could you please share some additional information so that I could reproduce this and provide a fix? Which model are you using? Are you increasing the max_length parameter when running generate()?

mrwyattii avatar Nov 13 '23 16:11 mrwyattii