Patrick Devine
Patrick Devine
I'm trying to duplicate this on macos and am not seeing it, although I am getting it to print the replacement character for some of its output. I've tried both...
Hi @taozhiyuai , I think you could probably ask HuggingFace to change that? There isn't anything we can change. I'll go ahead and close the issue.
@Mushoz I think this is still an HF issue though, no?
@jaypeche Thanks for the update! Is there anything we need to do on our end?
Hey @jaypeche , is it OK to close the issue as completed? Not sure if there is anything for us to do on our end.
There aren't any plans for this currently, but if enough people ask for it, we'd definitely consider it!
@mputzi it's coming back through vulkan (as @rick-github mentioned)... unfortunately the ROCm support for it is broken.
Here's a sample of how the Markdown could look: | Model | Step | Count | Duration | nsPerToken | tokensPerSec | |-------|------|-------|----------|------------|--------------| | gpt-oss:20b | prefill | 124 |...
@yilei-ding the template for `mixtral:8x7b-instruct-v0.1-fp16` was _slightly_ off (there was an additional space at the beginning of the template) which may have been causing poor results. I've just pushed an...
OK, I have re-converted the fp16 version and I get comparable performance for both. On the new version I get: ``` total duration: 1m28.047026667s load duration: 2.070959ms prompt eval count:...