swiftLLM icon indicating copy to clipboard operation
swiftLLM copied to clipboard
verified publisher icon interestingLSY

Batch size capped at 100 requests

Open ASHISHAVHAD opened this issue 2 months ago • 0 comments

Even though --max-batch-size default value is 512, I could not get it to exceed 100. I ran it on gpu with much more vram also (From 6gb to 48gb), changed values of flags in the engine_config.py file. Is it some bug in the code? or am I missing something?

ASHISHAVHAD avatar Nov 30 '25 05:11 ASHISHAVHAD