Luca Gibelli comments

Results 7 comments of


                                            Luca Gibelli

Fix vLLM queue overflow with serialized semaphore release

I'm testing on a 4090 (24GB of VRAM) on a system with 20 GB of DDR4. The reason why lowering --pages_per_groups doesn't help is that each worker submits all pages...

Fix vLLM queue overflow with serialized semaphore release

The only real advantage of my approach is smoother queue depth. With your approach the queue would jump between 50→550→50→550. I will try your approach tomorrow and report back if...

Fix vLLM queue overflow with serialized semaphore release

Tested locally on my 4090.

vLLM queue overflow from burst page submissions

when using my PR #316, I run: ` python -m olmocr.pipeline ./localworkspace --markdown --workers 2 --gpu-memory-utilization 0.85 --max_model_len 8192 --pages_per_batch 5 --min_queue_threshold 30 --semaphore_release_interval 3 --pdfs /path/to/*.pdf ` without my...

Luca Gibelli

Fix vLLM queue overflow with serialized semaphore release

Fix vLLM queue overflow with serialized semaphore release

Fix vLLM queue overflow with serialized semaphore release

vLLM queue overflow from burst page submissions

vLLM queue overflow from burst page submissions

Support for File Inputs In Azure OpenAI

Support for File Inputs In Azure OpenAI