Michael Mi

Sounds like we should set currObj to be the closest one from last layer. set to top_candidates.top() seems set to a farest one since top_candidates is a max_heap. correct me...

Hello, at the moment, function calling isn't supported, but it's on our [roadmap](https://github.com/vectorch-ai/ScaleLLM/issues/84). It's worth noting that there are few open-source models with this capability currently available. As far as...

thank you for your interest in ScaleLLM. Yeah, it is indeed in our [roadmap](https://github.com/vectorch-ai/ScaleLLM/issues/84). we do have some internal numbers but not ready to share yet. As part of our...

thanks for your advice. yeah, it is our plan. i am working on setting up the whl build for each release. for now, i am trying to reduce whl size...

Good question. For now, Scalellm only supports running in a docker. We will be trying to add python and js wrapper once finishing ongoing performance optimizations. Stay tuned!

A quick update for you: python is supported in [latest release](https://github.com/vectorch-ai/ScaleLLM/releases/tag/v0.1.0). you can install scalellm with pip: `pip install scalellm` and start rest api server with `python3 -m scalellm.serve.api_server` Please...

Hi @zhyncs A quick update for you: python is supported in [latest release](https://github.com/vectorch-ai/ScaleLLM/releases/tag/v0.1.0). you can install scalellm with pip: `pip install scalellm` and start rest api server with `python3 -m...

Thanks for letting me know. Let me try to downgrade GCC to 10 and republish new packages using manylinux2014 (CentOS 7 based) Toolchain: GCC 10

Yeah, we can use it directly. Just sharing our plans on this: One continuous benchmark will be setup in coming weeks, comparing offline and serving metrics between scalellm, vllm and...

Thanks, never tried that benchmark script. Will try that after wrapping up current feature parity works for logprobs and best_of. Thanks