Ashish Avhad

icon indicating a website link https://www.cse.iitb.ac.in/~ashishavhad/

icon company IIT Bombay icon location Mumbai icon location MS in CSE at IIT Bombay '28. Current research areas: LLMs, LLM inferencing, GPUs, Computing Systems.

Results 1 issues of Ashish Avhad

Batch size capped at 100 requests

Even though --max-batch-size default value is 512, I could not get it to exceed 100. I ran it on gpu with much more vram also (From 6gb to 48gb), changed...