Missing Head and Tail Frame when inference a whole video.

[LokiXun]

Hi, When i Test the model on REDS, setting N=1 make on inference 2N+1 =3 frames(default), I found the inference results would miss the initial frame and the end frame of the video.

What should I do to get those frame back

[LokiXun]

Thanks for relpy. 😃

Seems like only use one frame to inference for the initial frame & end frame respectively? How could model use the temporal information when only input with one frame?

Does every frame used when you calculate metrics for the test dataset in paper? Like getting rid of the initial & end frame to get the average metrfics for each frame?

Hello, we tried to solve the issue.

This is what we did:

Update the test function in train.py to process the initial and end frames during inference. This will ensure that all frames in the video are included in the output, addressing the issue of missing frames when N=1.

You can review changes in this commit: jack-samo10000@2524ae2.

Caution

Disclaimer: The concept of solution was created by AI and you should never copy paste this code before you check the correctness of generated code. Solution might not be complete, you should use this code as an inspiration only.

Latta AI seeks to solve problems in open source projects as part of its mission to support developers around the world. Learn more about our mission at https://latta.ai/ourmission . If you no longer want Latta AI to attempt solving issues on your repository, you can block this account.

[LokiXun]

Hello, I test the commited code. It does work and return such error when inference the initial frame.

  File "xxx/FMA-Net-main/model.py", line 618, in forward
    F, KD, f_Y, f, anchor_D = self.degradation_learning_network(x)
  File "/root/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "xxx/FMA-Net-main/model.py", line 488, in forward
    F, Fw, f = blk(F, Fw, f, F0_c)
  File "/root/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "xxx/FMA-Net-main/model.py", line 414, in forward
    warped_F = self.bwarp(F, f)


einops.EinopsError:  Error while processing rearrange-reduction pattern "(b n t) c h w -> b (n c) t h w".
 Input tensor shape: torch.Size([9, 10, 180, 320]). Additional info: {'t': 3, 'n': 9}.
 Shape mismatch, can't divide axis of length 9 in chunks of 27

[LokiXun]

hello~is there any solution for this issue?

[GeunhyukYouk]

Hello, I apologize for the delayed response.

FMA-Net is a kind of sliding-window-based method that takes an odd number (2N+1) of consecutive frames as input to restore the center frame. As you mentioned, when N=1, the first and last frames are indeed excluded.

To address this issue, we extended FMA-Net into a recurrent-based model, and we plan to release the results soon. Thank you for your patience!