[Bug]: Handbrake segfaults on an Intel 13th gen during MFXClose() with qsv acceleration

[maru-sama]

Which component impacted?

Decode, Encode

Is it regression? Good in old configuration?

Yes, it's good in old version

What happened?

Hello, I recently switched from an older 8th gen notebook to a 13th gen. Running Handbrake on ubuntu 23.10 I noticed that it segfaulted in MXFClose() while cancelling and encoding. I compiled the latest HEAD from onevpl-intel-gpu with debug today and could reproduce the issue.

Thread 185 "ghb" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe37fe6c0 (LWP 93140)]
#0  0x00007fff251da630 in ??? ()
#1  0x00007fffa34f0274 in VideoCORE::DecreaseReference (this=0x7fffc828b440, surf=...) at /home/maru/source/onevpl-gpu/_studio/shared/include/mfxvideo++int.h:248
#2  0x00007fffa3a55b89 in operator() (__closure=0x7fff1c3c1738, global=..., s_task=...)
    at /home/maru/source/onevpl-gpu/_studio/mfx_lib/encode_hw/hevc/agnostic/base/hevcehw_base_legacy.cpp:2049
#3  0x00007fffa3a8276f in std::__invoke_impl<mfxStatus, HEVCEHW::Base::Legacy::FreeTask(const HEVCEHW::FeatureBlocks&, HEVCEHW::FeatureBase::TPushFT)::<lambda(MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&)>&, MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&>(std::__invoke_other, struct {...} &) (__f=...) at /usr/include/c++/13/bits/invoke.h:61
#4  0x00007fffa3a7b52b in std::__invoke_r<mfxStatus, HEVCEHW::Base::Legacy::FreeTask(const HEVCEHW::FeatureBlocks&, HEVCEHW::FeatureBase::TPushFT)::<lambda(MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&)>&, MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&>(struct {...} &) (__fn=...) at /usr/include/c++/13/bits/invoke.h:138
#5  0x00007fffa3a7044d in std::_Function_handler<mfxStatus(MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&), HEVCEHW::Base::Legacy::FreeTask(const HEVCEHW::FeatureBlocks&, HEVCEHW::FeatureBase::TPushFT)::<lambda(MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&)> >::_M_invoke(const std::_Any_data &, MfxFeatureBlocks::StorageW &, MfxFeatureBlocks::StorageW &)
    (__functor=..., __args#0=..., __args#1=...) at /usr/include/c++/13/bits/std_function.h:290
#6  0x00007fffa37d7a77 in std::function<mfxStatus (MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&)>::operator()(MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&) const
    (this=0x7fff1c3c1738, __args#0=..., __args#1=...) at /usr/include/c++/13/bits/std_function.h:591
#7  0x00007fffa39cb7a2 in MfxFeatureBlocks::FeatureBlocksCommon<MfxFeatureBlocks::BlockTracer>::Block<std::function<mfxStatus (MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&)> >::Call<MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&>(MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&) const (this=0x7fff1c3c1720)
    at /home/maru/source/onevpl-gpu/_studio/mfx_lib/shared/include/feature_blocks/mfx_feature_blocks_base.h:134
#8  0x00007fffa39cd37a in MfxFeatureBlocks::CallAndGetMfxSts<MfxFeatureBlocks::FeatureBlocksCommon<MfxFeatureBlocks::BlockTracer>::Block<std::function<mfxStatus (MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&)> > const&, void, MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&>(MfxFeatureBlocks::FeatureBlocksCommon<MfxFeatureBlocks::BlockTracer>::Block<std::function<mfxStatus (MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&)> > const&, MfxFeatureBlocks::StorageW&, MfxFeatureBlocks::StorageW&) (blk=...)
    at /home/maru/source/onevpl-gpu/_studio/mfx_lib/shared/include/feature_blocks/mfx_feature_blocks_utils.h:285

Running the same setup on the older 8th gen does not result in a segfault.

To reproduce:

1. Start an encoding with either h264 or h265 QSV
2. Cancel the process and confirm
3. Segfaiult

Please note MFXCLose is called in various places in the code and does not segfault. Only when cancelling a running encode this happens. the relevant call is here.

https://github.com/HandBrake/HandBrake/blob/cd38834a597ab5b0f34cf086860abba7236f2fc1/libhb/qsv_libav.c#L244

What's the usage scenario when you are seeing the problem?

Transcode for media delivery

What impacted?

No response

Debug Information

1. libva latest HEAD
2. libgmm 22.3.9
3. media driver 23.2.4

Do you want to contribute a patch to fix the issue?

None