[BUG] High dynamic pictures cause H264Encoder to freeze with default bitrate

[nzottmann]

I discovered a case where the default bitrate causes the H264Encoder to stop encoding frames without any visible exception. Preconditions:

• CM4, Camera v3
• Resolution 640 x 360 or lower
• Image format: YUV420 (Can't reproduce with RGB888, but lores only supports YUV420)

I discovered this in a real world case with an image in front of the camera which I think is hard to compress (small structured surface with the autofocus having trouble to focus on).

To reproduce this independently from what is in front of the lens, I wrote some code to draw rectangles on the image with random size, color and position changing in each frame. This creates a lot of intra-frame changes. My example code continuously increases the complexity by increasing the number of rectangles per frame continuously until no more frames are received. For me this occurs at about 40 rectangles per frame.

The example code seems quite long, but only the middle part is required to reproduce the problem. The other parts are only for artificially increasing the image complexity and tracking the camera state.

from picamera2.encoders import H264Encoder
from picamera2.outputs import PyavOutput
from picamera2 import Picamera2, MappedArray
import time
import random
import cv2

ts_start = time.monotonic()
max_rects = 1
last_request_timestamp = None

def apply_timestamp(request):
    global last_request_timestamp
    last_request_timestamp = request.get_metadata()['SensorTimestamp']
    with MappedArray(request, 'main') as m:
        t0 = time.monotonic()
        count = 0
        while time.monotonic() - t0 < 0.01 and count < max_rects:
            count += 1
            start = (random.randint(0,m.array.shape[1]-101), random.randint(0,m.array.shape[0]-101))
            end = (start[0]+random.randint(0,100), start[1]+random.randint(0,100))
            cv2.rectangle(m.array, start, end, (random.randint(0,255), random.randint(0,255), random.randint(0,255)), -1)

# <-- Minimal necessary code starts here
picam2 = Picamera2()
picam2.post_callback = apply_timestamp
video_config = picam2.create_video_configuration(
    main={"size": (640, 360), "format": "YUV420"},
    raw={"size": (2304, 1296)},
    controls={"FrameDurationLimits": (int(1e6/24), int(1e6/24))},
    display=None,
    use_case="video"
)
picam2.align_configuration(video_config)
picam2.configure(video_config)
encoder1 = H264Encoder(framerate=24)
output1 = PyavOutput("/dev/null", format="null")
picam2.start_recording(encoder1, output1, name="main")
# Minimal necessary code ends here -->

# Wait until camera produces frames
while last_request_timestamp is None:
    time.sleep(0.1)

last_printed_request_timestamp = None
while True:
    time.sleep(1.0)
    if last_printed_request_timestamp == last_request_timestamp:
        print('no more frames')
        break

    last_printed_request_timestamp = last_request_timestamp
    print(last_request_timestamp, max_rects)
    max_rects += 1

Without knowing the technical background of the encoder, if I watch the picture, the encoder seems to lower the visible quality with increasing complexity to keep the provided bitrate. At some point the quality seems a lowest point and the encoder stops. This lead me to look at the bitrate. Without supplying any bitrate or quality, the encoder chooses Quality.MEDIUM: https://github.com/raspberrypi/picamera2/blob/fb031cb49cb2af4dab7439be7a90784c9d3db1ab/picamera2/encoders/h264_encoder.py#L93-L108 which results in a bitrate of 533333. Setting this bitrate manually, does not change anything, as expected. Testing with twice this bitrate (533333*2), the issue is gone.

I would be fine with just choosing a higher bitrate to solve this problem, but I need to understand how to choose the minimum bitrate to solve this in all cases. I discovered this by a "lucky" coincidence, triggered by a specific image in front of the camera, so there is no reliable method to test this for all possible images.

The bitrate calculation in picamera2 depends on resolution and framerate, but it does not seem safe for the encoder requirements in all cases.

Edit: I opened a separate issue for the general question "How to find the minimal allowed bitrate for a given resolution and framerate" #1230 This issue is just a proof that the built-in calculation does currently not answer this question.