raspberrypi
[HOW-TO] Find minimal allowed bitrate for H264Encoder
The H264Encoder does not work when the bitrate parameter is too low. This results in picamera2 not outputting any more frames, probably because all buffers are full.
I discovered that a bitrate is not too low in general, it depends on the dynamics of the picture (intra- and inter-frame). Thus having a still scene in front of the camera with a given bitrate working fine can stop working just by moving something in front of the camera.
If no bitrate is specified, picamera2 calculates a bitrate based on resolution, framerate and quality. If no quality is specified, the default is Quality.MEDIUM. This calculation unfortunately does not always give a "save" bitrate, especially for low resolutions, as I discovered in #1228
- Why does the encoder just stop working if the bitrate is too low for the given picture complexity? Is this hardware-defined behavior?
- How can I determine the minimal required bitrate for any picture complexity?
So I think I've made some progress on this. There isn't a minimum bitrate as such, but what happens is that when the encoder runs out of bits it simply doesn't return a buffer at all. Nothing comes back for that frame, not even an indication that it had no bits (you might think it could return a zero-length buffer, but I think V4L2 might regard that as being the end of the stream, so it probably isn't allowed).
Anyway, the Picamera2 V4L2 encoder was assuming that every image buffer that we put in gives us an encoded buffer out, but that's not true. I think we can tweak the encoder loop so as to avoid that and hopefully, then, there'll simply be a missing frame in the output.
However, I'm still having some problems with invalid mp4 files being created. I think the problem now is that skipping the frames is fine for P (non-IDR) frames, but for IDR frames the encoder is still sending me the SPS and PPS, but then no attached frame. This is duly getting written into the output which seems to be bad. So I'm probably going to have to run through the NAL units that come out and discard the frame if there was no actual picture there.
You could have a look at this PR https://github.com/raspberrypi/picamera2/pull/1236 and maybe try that branch. I believe it will fix the lock-ups and create valid mp4 files.
Of course you'll end up with pauses when the encoder runs out of bits, and I'm afraid I don't have a way to calculate at what bitrate (for given resolution/framerate) that is likely to start become an issue. It might be a thing you would have to experiment a bit with for your own use case. But do let me know if that branch improves things for you.
Thank you, that sounds reasonable and explains what I observed. Skipping frames seems to be the only option for the encoder if exceeding the bitrate with max compression. I can accept that very well if the picamera2 v4l2 encoder will continue encoding after a fix.
Reading about the problem with SPS and PPS without frame made me remember a V4L2 codec control:
# v4l2-ctl -l --all -d 11
[...]
sequence_header_mode 0x009909d8 (menu) : min=0 max=1 default=1 value=1 (Joined With 1st Frame)
0: Separate Buffer
1: Joined With 1st Frame
Seems like this is the default setting, if this would be changed to 0: Separate Buffer, would it cause the same problem you are currently facing with SPS/PPS without frame?
You could have a look at this PR #1236 and maybe try that branch. I believe it will fix the lock-ups and create valid mp4 files.
Of course you'll end up with pauses when the encoder runs out of bits, and I'm afraid I don't have a way to calculate at what bitrate (for given resolution/framerate) that is likely to start become an issue. It might be a thing you would have to experiment a bit with for your own use case. But do let me know if that branch improves things for you.
Thank you, I will test and report.
I am absolutely fine with having to find the bitrate at which the encoder starts to skip frames by experiment if I continues to work. I was just worried about finding this limit by experiment if hitting it would cause the encoder to stop.