Frames lost during transcode.

[huiyiqun]

I have a small application which decodes one stream, alters one of its planes, and then encodes and remuxes it into output container.

However, the first few frames of the output video seems has wrong time invervals(the video seems slowed down). And the application has following output:

Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Timestamps are unset in a packet for stream 1. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
Packet with invalid duration -9223372036854775808 in stream 1
Encoder did not produce proper pts, making some up.

I happens to find that the large negative number is extactly the start_time of the stream, while I'm not sure whether it's related.

In [1]: import av

In [2]: f = av.open('test.mp4', 'w')

In [3]: f.add_stream('libx264')
Out[3]: <av.VideoStream #0 libx264, yuv420p 640x480 at 0x7fada6cc2df0>

In [4]: f.add_stream('libx264').start_time
Out[4]: -9223372036854775808

versions:

~/Develop/patchwork> ./venv/bin/python -V
Python 3.6.0
~/Develop/patchwork> ./venv/bin/python -m av --version
PyAV v0.3.3
git origin: [email protected]:mikeboers/PyAV
git commit: unknown-commit
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/huiyiqun/Develop/patchwork/venv/lib/python3.6/site-packages/av/__main__.py", line 44, in <module>
    main()
  File "/home/huiyiqun/Develop/patchwork/venv/lib/python3.6/site-packages/av/__main__.py", line 24, in main
    for libname, config in sorted(av._core.versions.iteritems()):
AttributeError: 'dict' object has no attribute 'iteritems'

In [2]: av._core.versions
Out[2]:
{'libavcodec': {'configuration': '--prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth--enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab',
  'license': 'GPL version 3 or later',
  'version': (57, 64, 101)},
 'libavdevice': {'configuration': '--prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab',
  'license': 'GPL version 3 or later',
  'version': (57, 1, 100)},
 'libavfilter': {'configuration': '--prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab',
  'license': 'GPL version 3 or later',
  'version': (6, 65, 100)},
 'libavformat': {'configuration': '--prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab',
  'license': 'GPL version 3 or later',
  'version': (57, 56, 101)},
 'libavresample': {'configuration': '',
  'license': '',
  'version': (-1, -1, -1)},
 'libavutil': {'configuration': '--prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid--enable-netcdf --enable-shared --enable-version3 --enable-x11grab',
  'license': 'GPL version 3 or later',
  'version': (55, 34, 101)},
 'libswresample': {'configuration': '--prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2--enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab',
  'license': 'GPL version 3 or later',
  'version': (2, 3, 100)},
 'libswscale': {'configuration': '--prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth--enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab',
  'license': 'GPL version 3 or later',
  'version': (4, 2, 100)}}

In case you need this:

~> ffmpeg
ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.1 (GCC) 20170109
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

[huiyiqun]

I found that the real problem is that the first few frames(about 40 frames) of the original stream were lost during the transcode procedure.

[huiyiqun]

Minimum program to reproduce the problem:

import av

i = av.open('./input.mov')
o = av.open('./output.mov', 'w')

input_stream = next(stream for stream in i.streams if isinstance(stream, av.video.stream.VideoStream))

output_stream = o.add_stream('libx264', 1/input_stream.rate)
output_stream.bit_rate = 4000000
output_stream.height = input_stream.height
output_stream.width = input_stream.width

for frame in i.decode(streams=(input_stream, )):
    frame.pts = None
    pkt = output_stream.encode(frame)
    if pkt is not None:
        o.mux(pkt)

while True:
    pkt = output_stream.encode()
    if pkt is None:
        break
    o.mux(pkt)

o.close()

[huiyiqun]

It's very strange for me that I couldn't explain why, so I have to illustrate everything I have found:

1. with ffprobe, I could see that the nb_frame is still correct, but if I try to list all the frames, the number of frames listed is less than nb_frame.
2. with ffplay, I will see that the video starts from the second key frame(the first key frame is the first frame of the stream).
3. with mplayer, I will see that the video starts from the real starting but before the second key frame, the content is unclear as if the first key frame is lost.

If I set the size of gop of the output_stream to 1, then the output video seems normal, but that's not acceptable.

[mikeboers]

Do you find this happens with every video you run through your script, or just specific ones? In either case, can you please upload (small) samples to https://www.dropbox.com/request/CV725W7HRqaSv4N3t3HK ?

Thanks,

Mike

[huiyiqun]

Actually, every video.

For example a random video downloaded from youtube:

~> ffprobe -show_streams -select_streams v input.mp4 | grep nb_frames
nb_frames=34776
~> ffprobe -show_frames -select_streams v input.mp4 | grep \/FRAME | wc -l
34776

However, the video through the transcoding of my script:

~> ffprobe -show_streams -select_streams v output.mp4 | grep nb_frames
nb_frames=34776
~> ffprobe -show_frames -select_streams v output.mp4 | grep \/FRAME | wc -l
34716

I guess you could easily reproduce this phenomenon with an arbitary video. If needed, A sample will sitill be uploaded after I find a small one.

[kanehekili]

The current version (via pip install) does not encode at all:

pkt = videoOutStream.encode(frame)

File "av/video/stream.pyx", line 86, in av.video.stream.VideoStream.encode (src/av/video/stream.c:2974)

File "av/video/stream.pyx", line 147, in av.video.stream.VideoStream.encode (src/av/video/stream.c:2664)

File "av/utils.pyx", line 78, in av.utils.err_check (src/av/utils.c:1732) av.AVError: [Errno 38] Function not implemented

[huiyiqun]

@kanehekili Have you checked that the encoder you chooses is included in your ffmpeg build?

It seems there is no new version recently.

[ramoncaldeira]

@huiyiqun I stumbled upon the same problem and managed to solve it by changing the encoder. I replaced libx264 with libopenh264.

You can list the encoders available on your system using the ffmpeg -codecs command. On my machine it returns the following: DEV.LS h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 libopenh264) (encoders: libx264 libx264rgb libopenh264).

[github-riese]

Got the same problem here. pyav 8.0.2 on macOS 10.15.7 (brew) Feeding frames into libx264 for the first 42 frames never returned a packet. When on finishing write I called encode() without a frame, it returned a bunch of packets that, funny enough, matched my first 42 frames in terms of pts perfectly. Circumvented the issue by doing so:

• if encode(frame) returns nothing, proceed
• if encode(frame)returns a list of packets I check their PTS.
  • if the PTS in the first packet equals the expected PTS for the first frame, it gets muxed
  • otherwise I queue the packets internally, queue the packets from the last encode() w/o params as well, sort by PTS and mux the bunch in turn.

Thus I loose no more frames. Feels like a dirty hack but solved my problem all right.

[kanehekili]

Your description has nothing to do with PyAV - this is how h264 works.

Feeding frames into libx264 for the first 42 frames never returned a packet.

Yes. The muxer needs a bunch of frames before it can start to make packages - that's the intrinsic behaviour of any muxer that compresses. You'll experience that behaviour also if you using libavcodec/lib264 with C/C++.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[CLIsVeryOK]

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university

[QoT]

The closest I could get is that related lost frames issue has something to do with variable frame rate or frame rate rounding.

The thing is that my source video has variable frame rate of 239.548, while re-encoded (to 224x224) the same video with ffmpeg has fixed frame rate 240.

Both ffprobe shows and AV library show that source video has 341118 frames, but 341761 in re-encoded video.

341118*(240/239.548)-341118=643.65 which is very similar to (341761-341118) == 643 frame diff.

So, that must have something to do with variable frame rate or frame rate rounding.

The strangest thing is that ffmpeg exports correctly all frames but ffprobe displays less than real frame number: ffprobe -show_streams -select_streams v A704C95E-AAFA-42B5-8163-2FA8274F0279.mov | grep nb_frames

nb_frames=341118

ffprobe -show_streams -select_streams v A704C95E-AAFA-42B5-8163-2FA8274F0279_224.mov | grep nb_frames

nb_frames= 341761

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[CLIsVeryOK]

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university