nncf icon indicating copy to clipboard operation
nncf copied to clipboard

[Test] PTQ for QAT initialization for Torch samples

Open daniil-lyakhov opened this issue 2 years ago • 4 comments

Changes

Test for QAT after PTQ for torch classifications

Reason for changes

To check PTQ quantized model is trainable in torch

Related tickets

129584

Results

Testcase Task num epoch for fine-tuning now num epoch for fine-tuning before
resnet50_imagenet_int8 Classification 0 1
squeezenet1_1_imagenet_int8 Classification 0 5
resnet50_imagenet_int8_per_tensor Classification 0 1
inception_v3_imagenet_int8 Classification 0 1
mobilenet_v3_small_imagenet_int8 Classification 5 5
mobilenet_v2_imagenet_int8 Classification 1 5
squeezenet1_1_imagenet_int8_per_tensor classification 0 5
mobilenet_v2_imagenet_int8_per_tensor Classification 1 10
ssd512_vgg_voc_int8 Object Detection 0 78
ssd300_vgg_voc_int8 Object Detection 0 24
unet_mapillary_int8 Semantic Segmentation 0 300
unet_camvid_int8 Semantic Segmentation 0 10
icnet_camvid_int8 Semantic Segmentation 0 300

daniil-lyakhov avatar Jan 17 '24 18:01 daniil-lyakhov

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.57%. Comparing base (12f4720) to head (e4bd12f). Report is 12 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2404      +/-   ##
===========================================
- Coverage    91.18%   83.57%   -7.62%     
===========================================
  Files          492      494       +2     
  Lines        45096    45377     +281     
===========================================
- Hits         41123    37923    -3200     
- Misses        3973     7454    +3481     

see 110 files with indirect coverage changes

Flag Coverage Δ
COMMON ?
ONNX 34.66% <ø> (+<0.01%) :arrow_up:
OPENVINO ∅ <ø> (∅)
TENSORFLOW 30.10% <ø> (+0.23%) :arrow_up:
TORCH 65.92% <ø> (+0.16%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
common 88.89% <ø> (-4.88%) :arrow_down:
torch 93.48% <ø> (-0.12%) :arrow_down:
tensorflow 93.74% <ø> (ø)
onnx 93.09% <ø> (+0.03%) :arrow_up:
openvino 25.76% <ø> (-68.31%) :arrow_down:
ptq 64.84% <ø> (-25.38%) :arrow_down:

codecov[bot] avatar Jan 22 '24 15:01 codecov[bot]

@daniil-lyakhov do you have a build in CI with these tests?

ljaljushkin avatar Feb 26 '24 12:02 ljaljushkin

torch_weekly/31 - failed

AlexanderDokuchaev avatar Mar 04 '24 10:03 AlexanderDokuchaev

torch_weekly/31 - failed

torch_weekly/38 - in progress

daniil-lyakhov avatar Mar 14 '24 12:03 daniil-lyakhov

Build torch_weekly/42/ is successfull CC: @AlexanderDokuchaev

daniil-lyakhov avatar Mar 21 '24 18:03 daniil-lyakhov