fedem96 comments

Results 5 comments of


                                            fedem96

When will FP8 be available for Mixtral?

> @hawkeoni @Marlinski Sorry guys, I think Mixtral-FP8 is delayed a little bit and will be released very soon. Hi @Tracin, I suggest to keep somewhere a compatibility table between...

When will FP8 be available for Mixtral?

> > > @hawkeoni @Marlinski Sorry guys, I think Mixtral-FP8 is delayed a little bit and will be released very soon. > > > > > > Hi @Tracin, I...

When will FP8 be available for Mixtral?

> @fedem96 I think the table is not up to date. I was able to export and run Mixtral in FP16 as well as in W4A16 but it does not...

[Feature Request] Support for Constrained Decoding (such as generating Json formatted output)

Adding constrained decoding to this library (like SGLang JSON decoding) would be great, as it would allow a more reliable, faster generation. Is there any news about which release might...

Build via Docker is much big than the image in NGC

I'm also having the same issue