FasterTransformer icon indicating copy to clipboard operation
FasterTransformer copied to clipboard
verified publisher icon NVIDIA

Int4 Support

Open atyshka opened this issue 2 years ago • 3 comments

Are there plans to add Int4 support to FasterTransformer? This would be very useful in terms of speed and memory usage.

atyshka avatar Apr 21 '23 15:04 atyshka

Thank you for the suggestion. We will consider it.

byshiue avatar Apr 23 '23 02:04 byshiue

upvote this feature request.

donglinz avatar Apr 26 '23 08:04 donglinz

FasterTransformer development has transitioned to TensorRT-LLM.

Int4 (AWQ) is supported in TensorRT-LLM, please take a try.

byshiue avatar Oct 20 '23 10:10 byshiue