Michael Mi issues

Repositories
Issues
Comments

Results 53 issues of


                                            Michael Mi

[wip][feat] added python AsyncLLMEngine

Adding more bechmarks and unittests for kernels and dependencies

ScaleLLM Roadmap

We're excited to present the features we're currently working on and planning to support in this roadmap document. Your feedback is highly valued, so please don't hesitate to comment or...

roadmap

cuda graph capture may occasionally become stuck with multiple gpus.

It is a known issue that CUDA graph capture may occasionally become stuck when multiple workers are in use. further investigation is needed. Disabled cuda graph for multiple gpus by...

bug

enhancement

LoRA: QLoRA/S-LoRA: Serving thousands of LoRA adapters

Introducing the Mamba model

Introducing a ring attention mechanism for handling long contexts

Quantization: Supporting FP8 for both models and KV caches

Enhancing documentation for improved usability

Developing Python wrapper for easier integration

enhancement