Junrong Lin comments

Results 13 comments of


                                            Junrong Lin

FastChat as a backend service to support more models

Hi @bluechanel , thx for PR. Since the current open-source solution is entirely based on fastchat. How about also refactor the `opensource model`?

feat: Add serve module and cli tools

Hi! @dandansamax @Wendong-Fan Do you have time to take a look at this PR and give me some feedback?

[Feature Request] message_to_prompt -> .apply_chat_template

Emmm ok I see some inflexibilities of directly applying this method like it cannot apply to a single message but should be a chat history. During inference it is fine...

[Feature Request] Integrate Together AI

I'm ok with both solutions

add SGLang as rollout engine to verl

@vermouth1992 Hi, the test `dataset.yml` encounter ``` ../../../../.local/lib/python3.10/site-packages/torch/__init__.py:290: in from torch._C import * # noqa: F403 E ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory ```...

[BREAKING] [refactor] Unify async rollout under SGLangRollout, and support sglang==0.4.6.post5

I will push new docker image later today

veRL-SGLang slower than expected (GH200)

> > Update: flashinfer-python 0.2.3 has the same issue and 0.2.5 has OOM every time > > Log flashinfer_python 0.2.5 > > Update: flashinfer-python 0.2.5 works with newest commit and...

[Feature] Enable return routed experts

@zhaochenyang20 @zhuzilin @yizhang2077 @fzyzcjy

[Feature] Enable return routed experts

> Great job. But lint it please 😂 Done

[Feature] Enable return routed experts

> File "/sglang-checkout/python/sglang/srt/managers/scheduler_output_processor_mixin.py", line 373, in process_batch_result_decode .buffer[ ^^^^^^ AttributeError: 'NoneType' object has no attribute 'buffer' > > CI failed @ocss884 Done