IndexError: The shape of the mask [0] at index 0 does not match the shape of the indexed tensor [1, 4096, 320] at index 0

[Be-a-Warior]

Hello, during the stage_1 of training, there was a dimension mismatch issue during the first validation set validation after training 200 steps.

[charliezon]

I also met this problem.

[somaonishi]

As a temporary solution, apply deepcopy to net in log_validation worked.

[somaonishi]

I think this problem is probably caused by reference_control_reader and reference_control_writer.

After log_validation, I applied the following and it worked in some iter.

reference_control_reader.clear()
reference_control_writer.clear()