IST Austria Distributed Algorithms and Systems Lab

icon indicating a website link https://ista.ac.at/en/research/alistarh-group/

Results 1 repositories owned by IST Austria Distributed Algorithms and Systems Lab

marlin

607
Stars
46
Forks
Watchers

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

verified publisher icon IST-DASLab