Feedback about Real Time Inference on Raspberry Pi 4 and 5 (40 fps!)

[shimon-c]

There is the following issue on this page: https://docs.pytorch.org/tutorials/intermediate/realtime_rpi.html

I have a YOLO like model which I used QAT and the quantized model is very slow. It is slower 4 time than the floating point model. Although I used fused layers.

Any help?

[shimon-c]

Aslo try to jit YOLO model it failed for troch

[patrocinio]

Can you share:

• What size YOLO model you're using (YOLOv5s, YOLOv8n, etc.)?
• Are you running on Raspberry Pi or another ARM device?
• Did you verify the qnnpack backend is set?
• What does your quantization code look like?

[svekars]

cc: @d4l3k

[d4l3k]

@shimon-c can you share the yolo model you're using?

QAT (Quantized Aware Training) is a training technique and it's expected that enabling QAT (which doesn't actually quantize the model) will reduce the performance as it runs the full model + calculates the quantization losses.

To actually quantize the model it's best to hand tune the model to fuse and lower to int8 precision. The quantizated models in that tutorial in torchvision are optimized for performance.