AV import leads to OpenCV imshow freeze

[Flova]

Overview

Importing av breaks OpenCV aka cv2. This was first noticed in https://github.com/pytorch/vision/issues/5940. Importing other libraries that depend on av breaks the cv2.imshow() command. This is only the case of av is imported before the first execution of the said OpenCV function, if cv2.imshow() is run before the import all subsequent calls after the import also run.

Expected behavior

OpenCV and av should not affect each other

Actual behavior

The OpenCV call gets stuck, being unkillable at 100% CPU.

Investigation

See: https://github.com/pytorch/vision/issues/5940

Reproduction

The complete reproduction by multiple individuals can be found here: https://github.com/pytorch/vision/issues/5940

Versions

Complete Version locks for all dependencies can be found also here: https://github.com/pytorch/vision/issues/5940

• OS: Ubuntu 20.04
• PyAV runtime:

PyAV v9.2.0
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-mediafoundation --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-lzma --enable-version3 --enable-zlib
library license: GPL version 3 or later
libavcodec     58.134.100
libavdevice    58. 13.100
libavfilter     7.110.100
libavformat    58. 76.100
libavutil      56. 70.100
libswresample   3.  9.100
libswscale      5.  9.100

• FFmpeg:

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil      56. 31.100 / 56. 31.100
libavcodec     58. 54.100 / 58. 54.100
libavformat    58. 29.100 / 58. 29.100
libavdevice    58.  8.100 / 58.  8.100
libavfilter     7. 57.100 /  7. 57.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  5.100 /  5.  5.100
libswresample   3.  5.100 /  3.  5.100
libpostproc    55.  5.100 / 55.  5.100

Research

I have done the following:

• [x] Checked the PyAV documentation
• [x] Searched on Google
• [x] Searched on Stack Overflow
• [x] Looked through old GitHub issues
• [ ] Asked on PyAV Gitter
• [ ] ... and waited 72 hours for a response.

[jlaine]

Can we have a minimal reproduction that doesn't involve torchvision or conda?

Is this behaviour limited to Linux? Cross-platform?

[Flova]

I only tested this using Ubuntu 20.04

I didn't use conda for the mentioned minimal example, just a venv. But I can test it using only av.

[jlaine]

Here is the script:

import av
import cv2
import numpy as np

cv2.imshow("debug", np.zeros((128, 128, 3), dtype=np.uint8))
cv2.waitKey()

AFAICT the call that's hanging is a call to X11 libraries, so I'm guessing something conflicts between the libraries bundled by av and cv2.. meaning it's as much OpenCV's problem as ours!

[Flova]

I simultaneously opened issue https://github.com/opencv/opencv/issues/21952 at OpenCV's repo. I don't want to blame any of you guys in any way, just reported it and maybe somebody from cv2 responds as well and we are able to find a solution that works fine for both projects. I am not that into the exact implementation of both projects so idk how realistic that is.

[jlaine]

I'd definitely welcome at the very least understanding what the root cause is :)

[RichardFieb]

I tested installing PyAv by compiling against installed build of FFmpeg (XUbuntu 20.04LTS) as describe here: Bring your own FFmpeg

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100

With this configuration it seams to work for me. I hope this information is helpful.

[shankara-n]

^^^ works, here is the wheel i built that has no conflicts https://github.com/shankara-n/pyav/raw/master/av-9.2.0-cp38-cp38-linux_x86_64.whl

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.