[Bug]: AMF hangs on trying to decode video with momentary switch between progressive and interlaced fragment mid-stream

[DimkaTsv]

Describe the bug Original topic: https://github.com/rigaya/VCEEnc/issues/101

I have file [actually there were 4 sources like this from "pack of 6"] and i attached fragment which was cut from it. This fragment has momentary (about 0.5 seconds long) switch from progressive to interlaced mode and back to progressive in the beginning. When i try to play it with PlaybackHW, it will hang video stream output after first 1.5 seconds (or so) of fragment. But for some reason it actually does not hang HW decoder itself. Windows will report 90% utilisation for quite some time afterwards. Other players play this file... fine (except small processing lag spike at that point). Even with HW decode.

When i do transcoding of this fragment, result depends on fact if i used cut fragment or actual video from which i came. Both results are only applicable if HW decoder was used, and not SW.

1. For raw fragment, result will contain only first 1.5-2 seconds of video, but full audio stream.
2. For actual source, result will have black frame for duration of affected fragment. Timeline won't be affected, but seeking through corrupted part will be impossible. Audio stream will disappear after some time as well, but will also return before image comes back.

In addition i can note, that if i transcode fragments with VCEEnc, it will give proper beginning as result (aka no corruption is visible) [Still will hang and not produce proper output though]. But if i use TranscodeHW, it will mangle that first second that is actually encoded in very visible corruption.

If i force SW decoder, issue with transcoding of this source doesn't appear.

Source (fragmented): https://drive.google.com/file/d/17SqY4eXRZ8-QrJQR50ABo8z2lwdxfY02/view?usp=drive_link VCEEnc result (abruptly ended video stream): https://drive.google.com/file/d/1vrd68dxFwPHXvvmCoXo2nxHgfJS-IBsc/view?usp=drive_link TranscodeHW result (corrupted beginning AND abruptly ended video stream): https://drive.google.com/file/d/1uglDd6vi8dsTi1qy1MaMI1UFTJBv0IC-/view?usp=drive_link

VCEEnc without sound - in the end of 2 second transcoded sample there is small fragment from the very end of the almost 26 minute source fragment. https://drive.google.com/file/d/1J8yjSvNrgOLGLweaLUKw9kLVL9Ma00tQ/view?usp=sharing

To Reproduce Steps to reproduce the behavior:

1. Use attached source fragment
2. Use either VCEEnc, or TranscodeHW
3. Used commands (as drag and drop .bat file) Command VCEEnc: %~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed%~x1" --codec hevc Command VCEEnc (with sound): %~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed%~x1" --codec hevc --audio-copy Command TranscodeHW: %~dp0\TranscodeHW.exe -input %1 -output %~n1_processed%~x1 -codec HEVC

Behaviour: VCEEnc won't report progress until it finishes (which is usually not a thing), and amount of encoded frames will be really low, despite very actively using HW codec on the GPU. TranscodeHW will do same thing (aka will be "stuck" for longer, eventually HW codec usage drops to 0. But it won't end yet, as load on CPU continues. After more time it will produce "result". [And it takes quite significant amount of time at that]

These 12390 ms are a lie. It took TranscodeHW several minutes (around 5 or so... Maybe even 10?) to do the job on source.

Frames processed: 43 Frame process time: 12390.9561ms FPS: 0.1
 Average FPS: 69.7 Combined FPS: 69.7

Setup (please complete the following information):

• Windows 11, 23H2, build 22631.3155
• Driver: AMD Adrenalin 24.1.1 and 24.2.1 (reproduced on)
• GPU: AMD RX 7800XT 16GB
• Which component has the issue : Decoder mainly (encoder for secondary problem with TranscodeHW corrupting starting frames?)

Debug Log (please upload or paste):

Transcode HW reports this:

2024-02-28 04:32:34.643     2174 [matroska]   Error: C:\Users\DimkaTsv\Desktop\Utilites\VCE_Encoder\AMF-1.4.33\amf\public\src\components\ComponentsFFMPEG\UtilsFFMPEG.cpp(293):Application provided invalid, non monotonically increasing dts to muxer in stream 0: 292>= 0
SubmitInput() returned error: AMF_FAIL
2024-02-28 04:32:34.645     2174 [matroska]   Error: C:\Users\DimkaTsv\Desktop\Utilites\VCE_Encoder\AMF-1.4.33\amf\public\src\components\ComponentsFFMPEG\UtilsFFMPEG.cpp(293):Application provided invalid, non monotonically increasing dts to muxer in stream 0: 292>= 41
2024-02-28 04:32:34.645     2174 [matroska]   Error: C:\Users\DimkaTsv\Desktop\Utilites\VCE_Encoder\AMF-1.4.33\amf\public\src\components\ComponentsFFMPEG\UtilsFFMPEG.cpp(293):Application provided invalid, non monotonically increasing dts to muxer in stream 0: 292>= 41
SubmitInput() returned error: AMF_FAIL
2024-02-28 04:32:34.647     2174 [matroska]   Error: C:\Users\DimkaTsv\Desktop\Utilites\VCE_Encoder\AMF-1.4.33\amf\public\src\components\ComponentsFFMPEG\UtilsFFMPEG.cpp(293):Application provided invalid, non monotonically increasing dts to muxer in stream 0: 292>= 83
2024-02-28 04:32:34.647     2174 [matroska]   Error: C:\Users\DimkaTsv\Desktop\Utilites\VCE_Encoder\AMF-1.4.33\amf\public\src\components\ComponentsFFMPEG\UtilsFFMPEG.cpp(293):Application provided invalid, non monotonically increasing dts to muxer in stream 0: 292>= 83
SubmitInput() returned error: AMF_FAIL

But i saw it report similar report from samples which were applied by me to this thread: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/issues/447 Even though they are from completely different source and even completely different person. I suppose it is related to corrupted frames at beginning of the video?

Expected behavior Video should transcode without dropping huge part of video stream.

Screenshots No screenshots are applicable here.

Additional context