server Add support for closed captions

Adds support for ASTC A/53 Part 4 Closed Captions (essentially EIA-608 and CTA-708).

This adds a extensible side-data mixer (side-data naming is based on FFmpeg's AVFrameSideDataType), which for now just handles closed captions. For closed captions, for each layer, it chooses which input layer to take the closed captions from based on a priority associated with each layer. That priority can be any positive floating-point number. Zero, NaN, or negative numbers mean that there is no priority, so that layer won't be used for closed captions, allowing you to turn off closed captions.

The side-data mixer currently doesn't properly support switching the closed captions source in the middle of an input video, currently it'll just stop the previous video's side-data and start the next one's side data, rather than e.g. building a proper EIA-608 transition sequence. It does warn when you change the source like that though.

The side-data mixer also has functionality to use side-data from dropped frames in the input -- the code allows different side-data types to handle that differently since some types of side-data may only want to use the side-data from the frames that are actually sent to the output (e.g. AV_FRAME_DATA_DYNAMIC_HDR_PLUS I guess), and some may not want to miss any frames (closed captions, where missing a frame of data can cause the captions to be garbled).

So far, I've implemented closed captions support in:

FFmpeg producer: it supports getting closed captions from either the video stream's side data, or from a separate subtitles stream (e.g. in .mxf files). It also has partial untested support for reading a subtitles-only file (e.g. .scc).
FFmpeg consumer: it supports putting closed captions into the video stream's side data. It doesn't yet support generating the VANC stream for .mxf files, or other file types that need the closed captions to be a separate subtitles stream.

I added a ACMP command (MIXER CLOSED_CAPTIONS_PRIORITY) for changing a layer's closed-captions priority.

Since information on which inputs to use for closed captions or not is stored in the frame transforms, and those are all cleared by the MIXER CLEAR command, I decided to add a per-producer closed-captions priority setting that is associated with the producer rather than only with frame transforms. to set that, you have to pass the new CLOSED_CAPTIONS_PRIORITY option to whatever LOAD/LOADBG/PLAY command you use to create that producer. That has the benefit of also following the transition between different producers in a layer, rather than you having to be quick and sending MIXER CLOSED_CAPTIONS_PRIORITY at the right time. An additional benefit of having the new CLOSED_CAPTIONS_PRIORITY argument for PLAY is that you can enable closed captions directly from the config file if you didn't want to start up an ACMP client or type any commands into stdin.

I'm planning on adding closed captions support for the decklink producer, but I'm waiting to rebase this PR on top of @niklaspandersson's work https://github.com/CasparCG/server/issues/92#issuecomment-2956980473

Jun 10 '25 03:06 programmerjake

I'm not sure this PR will fix everything wanted in #92, so I'm not marking this as fixing that issue.

Jun 10 '25 03:06 programmerjake

I'll work on adding the proper #ifdefs for older versions of ffmpeg tomorrow, I was testing on Debian 12 and it turns out Ubuntu 22.04 uses an even older version of ffmpeg than I was expecting.

Jun 10 '25 04:06 programmerjake

rebased on #1642

Jun 13 '25 02:06 programmerjake

rebased on #1642 again, I also added EIA-708 VANC generation code for the decklink consumer based on libklvanc and the FFmpeg decklink code, afaict it generates VANC packets that are sent to @niklaspandersson's code successfully, but the SDI monitors I have hooked up don't show any closed captions -- either I'm not using them correctly or something is broken or both. (edit: it now works in progressive video modes, interlaced modes are still buggy)

stuff from the log:

[2025-06-13 19:10:13.669] [trace]   decklink consumer: got A53_CC side data: fc 97 23 fd 80 80 fa 00 00 fa 00 00 fa 00 00 fa and 44 bytes more
...
[2025-06-13 19:10:13.669] [trace]   decklink consumer: generated VANC packet from A53_CC side data: 96 69 49 4f 43 00 70 72 f4 fc 97 23 fd 80 80 fa 00 00 fa 00 00 fa 00 00 fa 00 00 fa 00 00 fa 00 and 41 bytes more

Jun 14 '25 02:06 programmerjake

turns out the bugs in interlaced modes were actually partially ffmpeg bugs, fixed in https://github.com/FFmpeg/FFmpeg/commit/61a9f3c0ce7f74ba869f44a49e16f3cd1b79e18d

so if we want this to work we'll need a newer version of ffmpeg than is available from apt on the Ubuntu versions we use.

Jun 18 '25 05:06 programmerjake

turns out the bugs in interlaced modes were actually partially ffmpeg bugs, fixed in FFmpeg/FFmpeg@61a9f3c

so if we want this to work we'll need a newer version of ffmpeg than is available from apt on the Ubuntu versions we use.

@programmerjake that patch was merged into ffmpeg 6.1 and Ubuntu 22.04 (Jammy) is the only distro that doesn't have it. Which Ubuntu versions are you referring to?

Jun 18 '25 14:06 dimitry-ishenko

@programmerjake that patch was merged into ffmpeg 6.1 and Ubuntu 22.04 (Jammy) is the only distro that doesn't have it. Which Ubuntu versions are you referring to?

ah, I thought it wasn't available on noble last I checked. regardless we'll need to change it to build ffmpeg rather than use the system ffmpeg when the system ffmpeg is too old.

Jun 18 '25 17:06 programmerjake

regardless we'll need to change it to build ffmpeg rather than use the system ffmpeg when the system ffmpeg is too old.

we used to do this, but it was a pain to maintain, and there was push from users to use the system version. I am open to having a cmake config field so that it can be built against a non-system version, but I don't think we will ship builds which do that

Jun 18 '25 17:06 Julusian

we used to do this, but it was a pain to maintain, and there was push from users to use the system version.

then we can't use ffmpeg for framerate conversion (also used whenever the channel is set to interlaced mode) when there are closed captions, since it messes up closed captions by duplicating and/or deleting frames with their closed captions side data, which was fixed in that commit.

Jun 18 '25 17:06 programmerjake

then we can't use ffmpeg for framerate conversion (also used whenever the channel is set to interlaced mode) when there are closed captions, since it messes up closed captions by duplicating and/or deleting frames with their closed captions side data, which was fixed in that commit.

As long as it works with the version in ubuntu 24.04, then I don't mind it not working 100% in 22.04.
Especially as that will leave 2 solutions for anyone who needs it, either update your ubuntu or rebuild against a custom ffmpeg.

Jun 18 '25 18:06 Julusian

regardless we'll need to change it to build ffmpeg rather than use the system ffmpeg when the system ffmpeg is too old.

@programmerjake sorry I don't follow. Like I said, Ubuntu Jammy is the only version that has FFmpeg older than 6.1. Are you on Debian Stable maybe? I believe they still use FFmpeg 5.1

Jun 18 '25 18:06 dimitry-ishenko

I thought the point of supporting Ubuntu Jammy is that it's supposed to just work, requiring you to compile your own ffmpeg and not providing one with the released binary packages doesn't sound like being supported to me.

I'm using Debian 12 for development, but that's not what I'm complaining about since you never claimed to support it, though if it can use the same mechanism I want to have for supporting newer ffmpeg on jammy, that'd be nice.

Jun 18 '25 18:06 programmerjake

I thought the point of supporting Ubuntu Jammy is that it's supposed to just work, requiring you to compile your own ffmpeg and not providing one with the released binary packages doesn't sound like being supported to me.

But, it does just work... Heck, I even have a version for Focal in my repo:

https://launchpad.net/~ppa-verse/+archive/ubuntu/casparcg

(which I will probably get rid of now that Focal is no longer officially supported.)

Now, if we want support for the Closed Captions for Jammy, then we have two options:

Try to back-port FFmpeg 6.1.1 from noble to jammy.
Try to back-port the patch you've linked to above into ffmpeg 4.4.2 which is used in jammy.

Both of these options will need to be hosted somewhere. I don't mind throwing it in my repo, since I already need a patched version of FFmpeg with DeckLink support (for another project).

I'm using Debian 12 for development, but that's not what I'm complaining about since you never claimed to support it, though if it can use the same mechanism I want to have for supporting newer ffmpeg on jammy, that'd be nice.

I just confirmed that noble's version of FFmpeg 6.1.1 package does build on Debian stable. If you are interested, I can set up auto-build on github and you can grab .deb packages from there.

Let me see if 6.1.1 compiles for jammy.

Jun 18 '25 19:06 dimitry-ishenko

Now, if we want support for the Closed Captions for Jammy, then we have two options:

Try to back-port FFmpeg 6.1.1 from noble to jammy.

Try to back-port the patch you've linked to above into ffmpeg 4.4.2 which is used in jammy.

The commit I linked to isn't the whole fix, it's just the latest in a sequence of commits that fix the bug, since they apparently had a partial fix before.

I was thinking that since we're using custom ffmpeg, we'd compile the same version of ffmpeg used on windows (iirc 7.0.2) and just statically link it to casparcg.

Both of these options will need to be hosted somewhere. I don't mind throwing it in my repo, since I already need a patched version of FFmpeg with DeckLink support (for another project).

Oh, FFmpeg with DeckLink can't be statically linked to CasparCG since they both include the DeckLink loader.

I'm using Debian 12 for development, but that's not what I'm complaining about since you never claimed to support it, though if it can use the same mechanism I want to have for supporting newer ffmpeg on jammy, that'd be nice.

I just confirmed that noble's version of FFmpeg 6.1.1 package does build on Debian stable. If you are interested, I can set up auto-build on github and you can grab .deb packages from there.

I've just been compiling them myself which isn't a problem for me, i'm compiling everything from source anyway since I want to enable asserts and debug info and stuff. I'm not using Debian 12 for running CasparCG in production.

Jun 18 '25 20:06 programmerjake

Oh, FFmpeg with DeckLink can't be statically linked to CasparCG since they both include the DeckLink loader.

Why are we using ffmpeg with decklink enabled? I thought casparCG has its own decklink implementation...

Jun 18 '25 21:06 zcybercomputinggroup

The commit I linked to isn't the whole fix, it's just the latest in a sequence of commits that fix the bug, since they apparently had a partial fix before.

OK I got it.

I was thinking that since we're using custom ffmpeg, we'd compile the same version of ffmpeg used on windows (iirc 7.0.2) and just statically link it to casparcg.

Static linking is an infernal abomination, a heretical curse spat in the face of divine coding practices. For large projects, it bloats binaries to grotesque, unholy sizes, rips open vulnerabilities like a gaping wound in the digital abyss, squanders memory with the ravenous greed of a demon, and chains developers to an endless, torturous grind of recompilation for every trivial shift in a subproject, dragging their souls through a screaming void of build cycles.

Let's not do static linking.

Jun 18 '25 21:06 dimitry-ishenko

Oh, FFmpeg with DeckLink can't be statically linked to CasparCG since they both include the DeckLink loader.

Why are we using ffmpeg with decklink enabled? I thought casparCG has its own decklink implementation...

"We" are not. "I" am. For another project. I just offered to backport 6.1.1 to jammy since I already had to patch it for another project. But, I take my words back. Backporting FFmpeg 6.1.1 to jammy is no trivial task. There are at least 10 other packages that will need to be backported as well.

So, never mind. I agree with @Julusian: if someone wants this feature, they will either have to upgrade to noble or compile ffmpeg on their own.

Jun 18 '25 21:06 dimitry-ishenko

I was thinking that since we're using custom ffmpeg, we'd compile the same version of ffmpeg used on windows (iirc 7.0.2) and just statically link it to casparcg.

Static linking is an infernal abomination...

I think that's greatly exaggerated and dynamic linking has significant downsides too, but I don't care enough to argue about it here.

Let's not do static linking.

ok. Debian has a potentially working branch backporting ffmpeg 7.1 to debian 12, which is pretty close to jammy, maybe that could be adapted into a jammy backport?

Jun 18 '25 21:06 programmerjake

I thought the point of supporting Ubuntu Jammy is that it's supposed to just work, requiring you to compile your own ffmpeg and not providing one with the released binary packages doesn't sound like being supported to me.

My thinking here is based around a few assumptions:

This will be used by a minority of users
ubuntu 22.04 is over halfway through its life (unless you factor the extended support stage), so users should be at least thinking about updating that.
Linux users are a minority of users. Based on the latest couple of release, the linux builds have 10-20% of the number of downloads compared to windows.
- prior to 2.4.0 (a year ago) there either werent linux builds in the releases, or they were semi-broken. For a while after that, there werent the beta/nightly builds for linux either
It is only a problem if using interlaced. (This might not be worth considering, I don't have any idea about distribution of usage on this)

If users start complaining about it not working, and can't upgrade ubuntu, then I will reconsider. But given my current assumptions, I don't see it being worth the effort to build and distribute ffmpeg.

So as long as it is communicated well that this is known to be broken in some scenarios, I am ok with that

I should add that I am a daily linux user, so I do want to push users towards using linux, but have not yet seen much traction on that

Jun 18 '25 21:06 Julusian

It is only a problem if using interlaced. (This might not be worth considering, I don't have any idea about distribution of usage on this)

it is broken whenever the ffmpeg fps filter is used to change frame rates, so whenever the source file's framerate doesn't match casparcg's internal progressive frame rate.

So as long as it is communicated well that this is known to be broken in some scenarios, I am ok with that

ok, in that case I'll add compile-time detection and log an error at runtime whenever it would be broken, and then just disable closed captions support in the ffmpeg producer.

Jun 18 '25 21:06 programmerjake

I think that's greatly exaggerated and dynamic linking has significant downsides too, but I don't care enough to argue about it here.

Yep. I ran it through grok twice to make sound extra gloomy.

Debian has a potentially working branch backporting ffmpeg 7.1 to debian 12, which is pretty close to jammy, maybe that could be adapted into a jammy backport?

Unfortunately that doesn't help. For example, it needs libsrist-dev, libjxl-dev and librav1e-dev none of which are in jammy. I've tried backporting libjxl-dev, but it needed 4 other other packages that are not in jammy, and librav1e-dev needed another 30 or so rust-related packages. So, I've decided this wasn't worth the effort.

Jun 18 '25 21:06 dimitry-ishenko

Unfortunately that doesn't help. For example, it needs libsrist-dev, libjxl-dev and librav1e-dev none of which are in jammy. I've tried backporting libjxl-dev, but it needed 4 other other packages that are not in jammy, and librav1e-dev needed another 30 or so rust-related packages. So, I've decided this wasn't worth the effort.

if it isn't in jammy's ffmpeg why go to a lot of effort to enable all the new codecs and stuff in a backport to jammy? imo just disable the ffmpeg configure features that need stuff not in jammy.

Jun 18 '25 21:06 programmerjake

if it isn't in jammy's ffmpeg why go to a lot of effort to enable all the new codecs and stuff in a backport to jammy? imo just disable the ffmpeg configure features that need stuff not in jammy.

I backported ffmpeg 7.1.1-1 from debian to jammy, I disabled libjxl-dev, librav1e-dev, librist-dev, and libvpl-dev: https://salsa.debian.org/programmerjake-guest/ffmpeg-casparcg-backports/-/tree/jammy-casparcg-backport?ref_type=heads

The only missing dependency is libjs-bootstrap5 which is only needed by ffmpeg-doc, the version from ubuntu noble works fine since it is only javascript and the only dependency it has hasn't changed at all.

Jun 18 '25 23:06 programmerjake

if it isn't in jammy's ffmpeg why go to a lot of effort to enable all the new codecs and stuff in a backport to jammy? imo just disable the ffmpeg configure features that need stuff not in jammy.

@programmerjake yes you can absolutely do that in your own PPA. I don't like this approach for 2 reasons:

I have a few small scripts that allow me to spin off the same package for different distros in a mostly automated manner. It is actually easier for me to spin off 3 or 4 packages that are the same across different distros/releases than to have different ones for each distro or release. Technically the rules file can be modified to dynamically enable/disable features, but it's more work and is error prone.
Having the same package version but with different features can actually be misleading. If someone tomorrow submits a patch that makes use of one of the disabled features on jammy, we've basically just kicked the can down the road.
I prefer to have minimal impact with my changes, as they don't go through the normal testing channels. I know disabling a few features doesn't seem like a big deal, but you never know. That's why I've opted to back-port noble's version of ffmpeg (6.1.1) into jammy only. That way noble and oracular are not impacted.

I know some of my arguments may seem straw-man and you don't have to agree with them, but I work on a few mission-critical projects, where failure is not an option (or a very painful option) and that's why I do things the way I do.

Jun 19 '25 03:06 dimitry-ishenko

if it isn't in jammy's ffmpeg why go to a lot of effort to enable all the new codecs and stuff in a backport to jammy? imo just disable the ffmpeg configure features that need stuff not in jammy.

That's why I've opted to back-port noble's version of ffmpeg (6.1.1) into jammy only. That way noble and oracular are not impacted.

Backporting only to jammy was exactly what I meant, I see no reason to have our own ffmpeg for noble or oracular or any other distro if their ffmpeg already has everything we need.

Jun 19 '25 03:06 programmerjake

if it isn't in jammy's ffmpeg why go to a lot of effort to enable all the new codecs and stuff in a backport to jammy? imo just disable the ffmpeg configure features that need stuff not in jammy.

That's why I've opted to back-port noble's version of ffmpeg (6.1.1) into jammy only. That way noble and oracular are not impacted.

Backporting only to jammy was exactly what I meant, I see no reason to have our own ffmpeg for noble or oracular or any other distro if their ffmpeg already has everything we need.

@programmerjake I've back-ported ffmpeg into jammy in my PPA and recompiled the server with it. Not that it matters right now I guess, but when server 2.5 is ready for release (ahem @Julusian) it should work with your patches.

https://launchpad.net/~ppa-verse/+archive/ubuntu/casparcg

Jun 20 '25 02:06 dimitry-ishenko

So as long as it is communicated well that this is known to be broken in some scenarios, I am ok with that

ok, in that case I'll add compile-time detection and log an error at runtime whenever it would be broken, and then just disable closed captions support in the ffmpeg producer.

Added in the latest commit. I decided to use runtime version detection whenever the libavfilter major version is <= 9 (the first fixed version is libavfilter 9.8.101), since the user might be using a fixed version of the dynamic library even if they compiled with a broken version.

Jun 20 '25 07:06 programmerjake