Forgetting measure does not work in online scenarios

[AlbinSou]

🐛 Describe the bug Forgetting measure fails to update correctly in online scenarios

🐜 To Reproduce Add ExperienceForgetting measure to the evaluation metrics in split_cifar10/online_replay.py, the metric does not fail with an error but is not reported by the loggers.

🐝 Expected behavior The metric should be reported by the loggers once the second task has finished

🦋 Additional context I started investigating and I think the following lines are causing the metric to fail:

In avalanche/evaluation/metrics/forgetting_bw.py GenericExperienceForgetting line 213

if self.train_exp_id == self.eval_exp_id:
            self.update(
                self.eval_exp_id, self.metric_result(strategy), initial=True
            )
        else:
            # update other experiences
            # if experience has not been encountered in training
            # its value will not be considered in forgetting
            self.update(self.eval_exp_id, self.metric_result(strategy))

The problem is that since this is online learning, self.train_exp_id at the end of training task 1 is very high (like 950 in the case of split cifar10 with bs 10), and does not correspond to self.eval_exp_id (0 for task 0 etc..). So the forgetting is never computed in initial state and thus never reported.

I think it's not necessary that this metric works when task boundaries are not provided, but it would be good that it works when they are provided. You can assign me to this issue I will try to fix it since it also causes CumulativeAccuracyForgetting (the one I'm trying to implement) to fail.

[AntonioCarta]

Thanks @AlbinSou

The current plan is to remove the stream-level metrics (i.e. anything that is computed on more than one experience). We are going to provide a method to extract the accuracy matrix (for each model, for each exp) and then standalone functions to compute forgetting fwt/bwt.

@JuliousHurtado is working on this.

[AlbinSou]

Ok, should I still try to change GenericForgettingExperience so that it works with Online Scenarios when task boundaries are provided or do you think the structure of all forgetting metrics will change a lot and it's better to wait for the changes ?

[AntonioCarta]

Ok, should I still try to change GenericForgettingExperience so that it works with Online Scenarios when task boundaries are provided or do you think the structure of all forgetting metrics will change a lot and it's better to wait for the changes ?

I would not touch them for now. You can implement your own metrics but I would leave the current ones as they are until we have the new ones.

[AntonioCarta]

@JuliousHurtado @AlbinSou does the new metric work with OCL?