GPU memory continue increase while testing

[KevinQian97]

Hi, I met a strange problem when testing trained model. I found the GPU memory consumption continue increase and finally raise the error of memory out. Presently, I didn't find someone else raise the similar problem.

I am using CUDA 10 with RTX2080Ti.

[kayser-soze]

See what I wrote in https://github.com/msracver/Flow-Guided-Feature-Aggregation/issues/68#issue-570612160