[Bug]: "PALookAheadBufferDepth" does not function: stalls encoder at best, throws errors at worst; in VCEEnc 7.09 and OBS Studio v28.0.1

[cs9kc]

Describe the bug After finding out more information about the PreAnalysis Component over in ticket 350 (https://github.com/GPUOpen-LibrariesAndSDKs/AMF/issues/350), I decided to try it out.

TL;DR:

In OBS Studio v28.0.1 on an RX 5700 XT and RX 6900 XT using 22.8.2, with LookAhead set to any value >0:

• Encoder produces output with PA enabled, but TAQ and PAHMQBE (motion boost) both disabled
• Encoder stalls after first I-frame with either TAQ or PAHMQBE enabled
• Encoder throws errors and produces an empty output file with both TAQ and PAHMQBE enabled

In VCEEnc 7.09 on an RX 6900 XT and RX 5700 XT using 22.8.2, with LookAhead set to any value >0:

• Encoder stalls and/or produces no output if any variation of "--pa lookahead=2" is used
• Encoder works fine with PA enabled, if one leaves lookahead out

To Reproduce Steps to reproduce the behavior in OBS Studio v28.0.1:

1. Set the following properties for the "AMD HW H.264" encoder

• CBR, 7000Kbps, 2s, Quality, High, 0 B-frames

HighMotionQualityBoostEnable=false EnableVBAQ=false RateControlPreanalysisEnable=0 RateControlSkipFrameEnable=false EnablePreAnalysis=true PASceneChangeDetectionEnable=false PAHighMotionQualityBoostMode=1 PATemporalAQMode=1 PAFrameSadEnable=false PALookAheadBufferDepth=2

2. Set OBS Studio's canvas in Settings -> Video to 1080p at 60fps
3. Monitor Task Manager and start the encode.
4. Output

Steps to reproduce the behavior in OBS Studio:

1. Use the following properties in a command line (set your own input and output files)

VCEEncC64.exe -i 1InputForVCEEnc.mkv -c h264 -u slow --vbr 6800 --max-bitrate 7000 --vbv-bufsize 14000 --ref 4 --ltr 0 --motion-est Q-pel --gop-len 120 --level 4.2 --profile high --pa taq=on,motion-quality=auto,lookahead=2 -o Output.mkv

2. Try with "--pa lookahead=2", "--pa motion-quality=auto,lookahead=2", and "--pa taq=on,motion-quality=auto,lookahead=2"
3. Observe encoder stall in both the command line and Task manager

Setup (please complete the following information):

• OS: Windows 10 (RX 5700 XT) and Windows 11 (RX 6900 XT)
• Driver Version: 22.8.2
• GPU: RX 5700 XT and RX 6900 XT
• Which component has the issue: h264 Encoder

Debug Log (please upload or paste): RX 6900 XT logs: RX 6900 XT Lookahead OBS Studio Log.txt RX 6900 XT Lookahead VCEEnc Log.txt

RX 5700 XT logs: RX 5700 XT OBSv28_2022-09-11 12-17-25.txt RX 5700 XT VCEEnc_Log.txt

Expected behavior I expect the performance and output to behave as normal when LookAhead is enabled, but LookAhead appears to not function properly in any configuration.

Additional context Once again, thank you all for your time and efforts. If any direct contact/testing is needed, I am part of and can be contacted via AMD Vanguard.

[rhutsAMD]

Thank you for your detailed bug report. The issue has been fixed and will be publicly available in a subsequent driver release.

[sandr1x]

Hello! I write guides for OBS setup and video encoding, so I want to clarify a few points. I have an RX 570 so I can't test it

1. Is this issue fixed?
2. Do I understand correctly that PALookAheadBufferDepth is required for AQ modes such as PAPerceptualAQMode and PATemporalAQMode to work?
3. PAPerceptualAQ works for H.264 and can I use it with TemporalAQ?
4. What is the maximum value for LookAheadBufferDepth? I didn't find it anywhere either, 41 max?
5. If I use EnablePreAnalysis=true, should I disable RateControlPreanalysisEnable and VBAQ? RateControlPreanalysisEnable works on my RX570, but I can't test EnablePreAnalysis on my video card.

If known, is PALookAheadBufferDepth fixed in OBS?

Thanks in advance for your replies.

[rhutsAMD]

Thanks for helping to document AMF features 😃. I added responses below.

1. Is this issue fixed?

• Yes, the crashing issue with PALookAheadBufferDepth is fixed in the latest public Adrenalin Edition 23.1.1 driver and newer.

2. Do I understand correctly that PALookAheadBufferDepth is required for AQ modes such as PAPerceptualAQMode and PATemporalAQMode to work?

• PALookAheadBufferDepth is required for TAQ, but it is not required for PAQ.

3. PAPerceptualAQ works for H.264 and can I use it with TemporalAQ?

• PAPerceptualAQ adjusts the block QP based on the block complexity. You cannot currently use PAQ and TAQ at the same time.

4. What is the maximum value for LookAheadBufferDepth? I didn't find it anywhere either, 41 max?

• Yes, the maximum value for LookAheadBufferDepth is 41.

5. If I use EnablePreAnalysis=true, should I disable RateControlPreanalysisEnable and VBAQ? RateControlPreanalysisEnable works on my RX570, but I can't test EnablePreAnalysis on my video card.

• EnablePreAnalysis and RateControlPreanalysisEnable are different layers of video quality improvement techniques. Both can be enabled.

• EnablePreAnalysis and VBAQ are also different layers of video quality improvement that can be enabled at the same time. Usually, when enabling PreAnalysis, other properties will be set as well which depend on computation done within PreAnalysis to improve video quality. In those cases, VBAQ may be overriden internally.

[cs9kc]

@rhutsAMD Thank you SO much for the work that you all have done, and for the very detailed post with clarification on settings and their interactions. It is MUCH appreciated! <3

[sandr1x]

@rhutsAMD This was very important information for me. Thank you very much for your work and for taking the time to explain this.