Error in evaluation of a strategy

[Ai360n]

Hello,

I am writing to inform you that unfortunately there is a problem when a scenario will be evaluated after training. In other words, when the below function will be called:

strategy.eval(scenario.test_stream)

This error will be shown:

AttributeError: 'RLScenario' object has no attribute 'test_stream'

Is there any update or solution for this issue?

Thanks in advance

avalanche-lib==0.3.1

[EliaPiccoli]

As you can see in the examples folder to make it work correctly you should write:

strategy.eval(scenario.eval_stream)

@AntonioCarta Should we stick with this name for the test/evaluation stream or should we change it?

[AntonioCarta]

I think test_stream is a better default. Most benchmarks have a train/test split, not a train/eval split.

[Ai360n]

As you can see in the examples folder to make it work correctly you should write:

strategy.eval(scenario.eval_stream)

@AntonioCarta Should we stick with this name for the test/evaluation stream or should we change it?

Yes, thank you. Pardon me, now, as I save the checkpoints, I get this error during the evaluation when I use eval_stream:

RuntimeError: Checkpoint file checkpoints/99/checkpoint.pth already exists.

What is the problem? However, I save the checkpoint at end!

[AntonioCarta]

We changed the checkpointing mechanism in the avalanche master because it had some issues. It is also untested with avalanche-RL still, so you may have to wait for the next avalanche release for a stable checkpointing system.

[Ai360n]

Thank you so much for your reply. Excuse me, when the next version of Avalanche will be release?

Thanks

[AntonioCarta]

It's planned for the end of June. Avalanche-rl will support the new version soon after the release.