dvclive DvcliveLoggerHook `by

I added the following configuration to my mmcv, mmdet training config.

log_config = dict(
    hooks=[
        dict(
            type="DvcliveLoggerHook",
            path="{{ fileDirname }}/../live",
            interval=1,
            by_epoch=True,
        ),
    ],
)

I would expect this to log the metrics per epoch. However they are logged per iteration. I tried multiple (three) mmdet models and the behavior is the same for all.

Possibly relevant part of the log file

 mmdet - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH   ) StepLrUpdaterHook                  
(NORMAL      ) CheckpointHook                     
(LOW         ) EvalHook                           
(VERY_LOW    ) DvcliveLoggerHook                  
 -------------------- 
before_train_epoch:
(VERY_HIGH   ) StepLrUpdaterHook                  
(NORMAL      ) NumClassCheckHook                  
(LOW         ) IterTimerHook                      
(LOW         ) EvalHook                           
(VERY_LOW    ) DvcliveLoggerHook                  
 -------------------- 
before_train_iter:
(VERY_HIGH   ) StepLrUpdaterHook                  
(LOW         ) IterTimerHook                      
(LOW         ) EvalHook                           
 -------------------- 
after_train_iter:
(ABOVE_NORMAL) OptimizerHook                      
(NORMAL      ) CheckpointHook                     
(LOW         ) IterTimerHook                      
(LOW         ) EvalHook                           
(VERY_LOW    ) DvcliveLoggerHook                  
 -------------------- 
after_train_epoch:
(NORMAL      ) CheckpointHook                     
(LOW         ) EvalHook                           
(VERY_LOW    ) DvcliveLoggerHook                  
 -------------------- 
before_val_epoch:
(NORMAL      ) NumClassCheckHook                  
(LOW         ) IterTimerHook                      
(VERY_LOW    ) DvcliveLoggerHook                  
 -------------------- 
before_val_iter:
(LOW         ) IterTimerHook                      
 -------------------- 
after_val_iter:
(LOW         ) IterTimerHook                      
 -------------------- 
after_val_epoch:
(VERY_LOW    ) DvcliveLoggerHook                  
 --------------------

Jul 07 '22 13:07 rick-van-veen

I would expect this to log the metrics per epoch.

Hi @rick-van-veen ! I took a look into this and appears to not be related to DVCLive.

The problem is that the by_epoch flag doesn't mean what you expect (introduced in https://github.com/open-mmlab/mmcv/pull/346), it doesn't affect when log is called in MMCV but how the iter number is retrieved.

It would be better to open an issue in https://github.com/open-mmlab/mmcv asking how to only log at the end of the epoch and/or ask to update the behavior of the by_epoch flag to match your expectations.

We could update the behavior of the DVCLive logger only but it would be better to have a consistent behavior inside the MMCV repo which is where the DVCLive integration resides

Jul 12 '22 10:07 daavoo

I created an issue in the mmcv repository and I am not entirely sure, but it seems the by_epoch argument does do what I would expect it to do in the TextLogger and that we can infer from that it is up to the logger what to do with the by_epoch bahaviour (also because they do not intent to implement it in mmcv (https://github.com/open-mmlab/mmcv/issues/2116#issuecomment-1186215776)?

Jul 18 '22 05:07 rick-van-veen

I created an issue in the mmcv repository and I am not entirely sure, but it seems the by_epoch argument does do what I would expect it to do in the TextLogger and that we can infer from that it is up to the logger what to do with the by_epoch bahaviour (also because they do not intent to implement it in mmcv (open-mmlab/mmcv#2116 (comment))?

I see. I will open a P.R. in mmcv handling by_epoch

Jul 21 '22 15:07 daavoo

@daavoo What's the status of this issue?

Mar 06 '23 22:03 dberenbaum

DvcliveLoggerHook `by_epoch` not working