datarhei
Corrupt packages and visual artefacts in SRT Server
Describe the bug
Corrupt packages in the SRT Servers output stream that correlate with visual artifacts in the video
Process log entries show up like:
2024-11-25T10:30:28.000Z [in#0/mpegts @ 0x7e349dfb6a80] [warning] corrupt input packet in stream 0
or sometimes also as:
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 204
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 192
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 188
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 204
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 192
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 204
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 192
2024-11-27T11:26:32.000Z [mpegts ] [warning] changing packet size to 188
2024-11-27T11:26:32.000Z [mpegts ] [warning] Packet corrupt (stream = 0, dts = NOPTS).
this is also a re-filing - or sum up - of existing issues that seem to be related or show the same problem such as https://github.com/datarhei/restreamer/issues/839 and https://github.com/datarhei/restreamer/issues/755
After a series of tests, I can confirm that this happens regardless of the input video, whether it is a still image or not, regardless of the encoder (OBS on Mac and ffmpeg on Linux), whether the upstream is delivered from a remote machine via network or with ffmpeg from localhost or with a latency setting of latency=1000000 or more. I have tried running Restreamer on 2 different machines (different versions of Ubuntu and Debian) and with network_mode=host. Sometimes, it shows up more frequently than at other times or is more or less obvious (it might certainly be enhanced by network issues!), but it shows nonetheless, in best cases, only every 4 or 5 minutes.
Crucially this problem does not appear when using the RTMP server or when bypassing restreamer sending the upstream directly to a different SRT server. So far I can only conclude that this must be a bug in the SRT server implementation. If I can do anything reasonable to track this down let me know.
To Reproduce
- Start a docker instance of restreamer latest (v2.12.0)
- Create a channel with an SRT server and send an upstream, for instance with
ffmpeg -re \
-stream_loop -1 \
-i BigBuckBunny.mp4 \
-vcodec libx264 \
-g 60 \
-b:v 5M \
-ar 44100 -acodec aac \
-b:a 128K \
-f mpegts \
"srt://localhost:6000?mode=caller&transtype=live&streamid=[....]&latency=1000000"
- Watch video/logs for up to 4 or 5 minutes.
Expected behavior
Clear, uncorrupted video stream
Screenshots
Desktop (please complete the following information):
- OS: Several (Ubuntu, Debian,, Mac OS)
- Browser: Several (Firefox and others, as well as none/or headless in another srt or rtmp publication)
Additional context
- Server OS: Ubuntu 24.04.1 LTS, Debian GNU/Linux 12, Mac OS
- Docker version 27.3.1, Docker version 23.0.4
- restreamer-v2.12.0 / datarhei-core v16.16.0 (linux/amd64)
Thanks for this detailed bug report. Unfortunately, I couldn't reproduce yet the described bug (I tested on MacOS with ffmpeg and Restreamer on the same machine), I have a suspicion where the problem might be.
You send the stream with ffmpeg and with a certain latency to the Restreamer SRT server. Restreamer is then pulling the stream via SRT from it's own SRT server with a certain latency in order to convert it into HLS. This internal SRT latency is by default 20ms and cannot be changed currently. This might cause the issues when there are some packet drops internally.
The solution would be to make the internal latency configurable.
To exclude that the SRT server itself is the problem, try to pull the stream from the SRT server (e.g. with ffplay) with a different latency. Use the same URL as for sending, but remove the ,mode:publish part from the streamid parameter.
ffplay -f mpegts -i "srt://localhost:6000?mode=caller&transtype=live&streamid=[...]&latency=100000"
yes, I am also seeing these artifacts and corrupt packages if directly pulled by ffplay and even higher latencies. Most surprisingly I am seeing even more of these artifacts (and corrupt packages logs) in the stream received with ffplay - despite the latency settings - than in the restreamer preview when watched side by side!
I tried to reproduce this. My setup is a MacBook Pro that is sending the stream to the Restreamer SRT server running on a Raspberry Pi 4. And also pulling from the SRT server back to the MacBook. All via wifi.
Sending the stream:
ffmpeg -re -stream_loop -1 -i big_buck_bunny.mp4 -codec copy -f mpegts "srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream,mode:publish&latency=120000"
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
built with Apple clang version 16.0.0 (clang-1600.0.26.3)
configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
libavutil 59. 39.100 / 59. 39.100
libavcodec 61. 19.100 / 61. 19.100
libavformat 61. 7.100 / 61. 7.100
libavdevice 61. 3.100 / 61. 3.100
libavfilter 10. 4.100 / 10. 4.100
libswscale 8. 3.100 / 8. 3.100
libswresample 5. 3.100 / 5. 3.100
libpostproc 58. 3.100 / 58. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'big_buck_bunny.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Duration: 00:09:56.46, start: 0.000000, bitrate: 5589 kb/s
Stream #0:0[0x1](eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1280x720, 5146 kb/s, 24 fps, 24 tbr, 19200 tbn (default)
Metadata:
handler_name : Apple Video Media Handler
vendor_id : [0][0][0][0]
encoder : H.264
timecode : 00:00:00:00
Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 437 kb/s (default)
Metadata:
handler_name : Apple Sound Media Handler
vendor_id : [0][0][0][0]
Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74)
Metadata:
handler_name : Apple Video Media Handler
timecode : 00:00:00:00
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Output #0, mpegts, to 'srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream,mode:publish&latency=120000':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1280x720, q=2-31, 5146 kb/s, 24 fps, 24 tbr, 90k tbn (default)
Metadata:
handler_name : Apple Video Media Handler
vendor_id : [0][0][0][0]
encoder : H.264
timecode : 00:00:00:00
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 437 kb/s (default)
Metadata:
handler_name : Apple Sound Media Handler
vendor_id : [0][0][0][0]
Press [q] to stop, [?] for help
frame=206779 fps= 24 q=-1.0 size= 6056052KiB time=02:23:35.76 bitrate=5758.2kbits/s speed= 1x
[q] command received. Exiting.
[out#0/mpegts @ 0x600001f9c3c0] video:5392518KiB audio:460323KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 3.475855%
frame=206791 fps= 24 q=-1.0 Lsize= 6056277KiB time=02:23:36.27 bitrate=5758.1kbits/s speed= 1x
Pulling the data
ffmpeg -f mpegts -i "srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream&latency=120000" -codec copy -f null -
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
built with Apple clang version 16.0.0 (clang-1600.0.26.3)
configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
libavutil 59. 39.100 / 59. 39.100
libavcodec 61. 19.100 / 61. 19.100
libavformat 61. 7.100 / 61. 7.100
libavdevice 61. 3.100 / 61. 3.100
libavfilter 10. 4.100 / 10. 4.100
libswscale 8. 3.100 / 8. 3.100
libswresample 5. 3.100 / 5. 3.100
libpostproc 58. 3.100 / 58. 3.100
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
...
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] non-existing PPS 0 referenced
[h264 @ 0x10af045b0] decode_slice_header error
[h264 @ 0x10af045b0] no frame!
Input #0, mpegts, from 'srt://192.168.1.154:6000?mode=caller&transtype=live&streamid=e0016116-a18d-42fc-b542-3570a371cd41.stream&latency=120000':
Duration: N/A, start: 8.737667, bitrate: N/A
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1280x720, 24 fps, 24 tbr, 90k tbn
Stream #0:1[0x101](eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, fltp, 445 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf61.7.100
Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1280x720, q=2-31, 24 fps, 24 tbr, 90k tbn
Stream #0:1(eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, fltp, 445 kb/s
Press [q] to stop, [?] for help
14:22:40.936679/!W:SRT.br: @410272501: RCV-DROPPED 6 packet(s). Packet seqno %1907416130 delayed for 60.178 ms
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 4543500).
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
Last message repeated 1 times
15:28:30.965826/!W:SRT.br: @410272501: RCV-DROPPED 123 packet(s). Packet seqno %1909626488 delayed for 1726.802 ms
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 359839050).
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
Last message repeated 1 times
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 361174050).ed= 1x
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 361211550).
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 762737700).eed= 1x
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
[mpegts @ 0x1240050e0] Packet corrupt (stream = 0, dts = 762917700).eed= 1x
[in#0/mpegts @ 0x6000032a8300] corrupt input packet in stream 0
frame=206499 fps= 24 q=-1.0 size=N/A time=02:23:25.45 bitrate=N/A speed= 1x
[q] command received. Exiting.
[out#0/null @ 0x600003bac000] video:5386027KiB audio:462487KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=206511 fps= 24 q=-1.0 Lsize=N/A time=02:23:25.95 bitrate=N/A speed= 1x
There were some errors, but only occasionally, not in the extreme extend you are describing.
hmm. I compared it with other SRT implementations, and interestingly I run into similar issues if I try a self-compiled version of https://github.com/Edward-Wu/srt-live-server (with Ubuntu's libsrt) over a local network and even localhost (on my most powerful machine). But I can distribute a clear, uncorrupted stream even over long-distance networks with ovenmediaengine - so there must be some key difference in srt server implementations...
srt-live-server and OME are using the Haivision reference SRT implementation. Restreamer is using an implementation of the SRT protocol in Go. The SRT specification leaves some room for variation in implementation.
SRT is based on UDP, which might not be treated with highest priority by network devices on the way from you to the server and back and packets will get dropped. SRT tries to recover from packet loss, but it depends on the configured latency how good it manages to.
I updated the Restreamer's SRT server to the latest version. Please try out the datarhei/restreamer:dev image.
Just gave it a quick run: running datarhei/restreamer:dev image with default settings does not solve that issue. Sorry but we have now moved to using OME.
This is a repost from the Discord server as it pertains to this bug report.
I am attempting to play with restreamer and OBS in an attempt to find a fix/work-around for the "Video Glitches" while using SRT #839 and this thread.
I am wondering how to accurately go about setting/adjusting the SRT buffer/latency settings, would passing rcvBufBytes, rcvlatency in OBS perhaps make a difference, does Restreamer have a set default?
I am attempting to keep the stream latency down to a minimum...currently with passing h265, encoding to h264, I tested on the built-in player at about 10 secs. I am on Starlink and they throttle my TCP connection sessions, so UDP allows to me hold a higher "consistent'ish" bitrate. My only issue is with the "screen glitches" as mentioned by others in the bug report.
I am wondering if there is a way to adjust the buffer size or latency to avoid said glitches, which really are only noticeable on still images, or the transition from still to moving and not constantly present in moving video.
This only occurs on SRT, RTMP works fine, except for the fact that I cannot hold a steady bitrate on TCP.
I see the following errors in the Process Log when the artifacting occurs (Multiple, but all basically the same)...
[mpegts ] [warning] Packet corrupt (stream = 0, dts = 61914000).
[in#0/mpegts @ 0x14d5d1025e40] [warning] corrupt input packet in stream 0
I am running Restreamer in Docker on Unraid... *On a side note, does Restreamer have the ability to use iGPU if passed to the container using "/dev/dri/" as a container device? I use this method for my Plex Transcoding.
Update... it appears that the ffmpeg ignores "peerlatency/latency" variables sent in the OBS stream key and defaults a value of 20000. My thoughts were to increase the latency and/or buffer size to attempt to offset the video glitching, but there doesn't appear (at least to me) to be a way to set these variables.
