gregszumel

Hey @michaelbearne -- thanks for this PR! I tried running `mix test` on my machine and had some lib-not-found issues, so maybe ort is changing the way it handles downloading...

Hi Nathan, thanks for the detailed description. I think the issue is coming from `lib/ortex/util.ex`, specifically here which assumes Linux: ``` case "libonnxruntime.so.1.17.0" in onnx_runtime_filenames do true -> nil false...

Sorry for the late reply! It definitely may have something to do with the ort 2.0 changes. I'll dig into this today.

I could confirm that there's an output ordering discrepancy in vanilla ort 2.0, so I think it's safe to assume that ortex is just propagating this issue. Specifically, ort moved...

Hey @EricLBuehler - Not sure about the roadmap and you may already have this solved, but I think I can help explain the above code. If I'm understanding the above...

@EricLBuehler Looks like it got merged - thanks at @LaurentMazare! I hope to find some time this week to implement top-k; I'm thinking it's home is probably in either in...

nice, this is cool! I'll work on deleting from the prefix-cache if some byte-threshold was exceeded. Long term we could bring the trie structure back and do some vLLM style...