PyAV build fails with p4a develop branch recipe

[sdas92]

Background

• I am trying to build an apk with ffmpeg and av. In my case, ffmpeg was built without any issues (without av earlier).
• Now, av throws below error (as of now, I did not use custom recipe)

Error

  RAN: /home/somnath/codes/git/my-org/image-to-animation-offline/kivy/.env/bin/python3 '-cimport sys; from Cython.Compiler.Main import setuptools_main; sys.exit(setuptools_main());' ./av/dictionary.pyx

  STDOUT:
/home/somnath/codes/git/my-org/image-to-animation-offline/kivy/.env/lib/python3.11/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/somnath/codes/git/my-org/image-to-animation-offline/kivy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/av-openssl/arm64-v8a__ndk_target_28/av/av/dictionary.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)

Error compiling Cython file:
------------------------------------------------------------
...
cimport libav as lib
       ^
------------------------------------------------------------

av/dictionary.pxd:1:8: 'libav.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
cimport libav as lib


cdef class _Dictionary:

    cdef lib.AVDictionary *ptr
        ^
------------------------------------------------------------

av/dictionary.pxd:6:9: 'AVDictionary' is not a type identifier

My observations

1. dictionary.pxd gets created in: <codebase>/.buildozer/android/platform/build-arm64-v8a/build/other_builds/av-openssl/arm64-v8a__ndk_target_28/av/av

2. libav.pxd gets created in different folder: <codebase>/.buildozer/android/platform/build-arm64-v8a/build/other_builds/av-openssl/arm64-v8a__ndk_target_28/av/include which creates av/dictionary.pxd:1:8: 'libav.pxd' not found

Custom recipe to copy entire include dir into av (under av)

This is placed before get_recipe_env function, this fixes the not found issue

    def prebuild_arch(self, arch):
        super().prebuild_arch(arch)

        # Paths
        build_dir = self.get_build_dir(arch.arch)
        av_pkg_dir = os.path.join(build_dir, "av")
        include_dir = os.path.join(build_dir, "include")
        # Copy all .pxd files from include -> av/
        info(f"Copying PyAV packages from {include_dir} to {av_pkg_dir}")
        shutil.copytree(include_dir, av_pkg_dir, dirs_exist_ok=True)

3. Now getting:

[INFO]:    av first build failed (as expected)
[INFO]:    Running cython where appropriate
[INFO]:    Cythonize av/buffer.pyx
[DEBUG]:   -> running python3 -cimport sys; from Cython.Compiler.Main import setuptools_main; sys.exit(setuptools_main()); ./av/buffer.pyx
[DEBUG]:        /home/somnath/codes/git/my-org/image-to-animation-offline/kivy/.env/lib/python3.11/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/somnath/codes/git/my-org/image-to-animation-offline/kivy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/av-openssl/arm64-v8a__ndk_target_28/av/av/buffer.pxd
[DEBUG]:          tree = Parsing.p_module(s, pxd, full_module_name)
[INFO]:    Cythonize av/dictionary.pyx
[DEBUG]:   -> running python3 -cimport sys; from Cython.Compiler.Main import setuptools_main; sys.exit(setuptools_main()); ./av/dictionary.pyx
[DEBUG]:        /home/somnath/codes/git/my-org/image-to-animation-offline/kivy/.env/lib/python3.11/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/somnath/codes/git/my-org/image-to-animation-offline/kivy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/av-openssl/arm64-v8a__ndk_target_28/av/av/dictionary.pxd
[DEBUG]:          tree = Parsing.p_module(s, pxd, full_module_name)
[DEBUG]:   
[DEBUG]:        Error compiling Cython file:
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:        ...
[DEBUG]:            def __dealloc__(self):
[DEBUG]:                if self.ptr != NULL:
[DEBUG]:                    lib.av_dict_free(&self.ptr)
[DEBUG]:   
[DEBUG]:            def __getitem__(self, str key):
[DEBUG]:                cdef lib.AVDictionaryEntry *element = lib.av_dict_get(self.ptr, key, NULL, 0)
[DEBUG]:                                                                               ^
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:   
[DEBUG]:        av/dictionary.pyx:18:72: 'str' objects do not support coercion to C types (use 'bytes'?).
Exception in thread background thread for pid 835789:

Please let me know if any other details are needed from my end. Thank you.

[sdas92]

Update on this issue

My env details

• av version: 13.1.0 (taken from develop branch)
• ffmpeg version: n6.1.2 (develop branch)
• My own venv python version: 3.11.13
• My venv Cython version: 0.29.36

Custom av recipe

I have worked on the issue for almost three days & was able to fix the build issue with custom recipe (which might not be most efficient). Details are given below.

• Here is the working custom recipe which fixes the problems which I faced during the build process.

• Instead of copying the include folder into av folder, used CYTHONPATH env variable as show below.

        build_dir = self.get_build_dir(arch.arch)
        av_pkg_dir = os.path.join(build_dir, "av")
        include_dir = os.path.join(build_dir, "include")
        env['CYTHONPATH'] = include_dir

• There were few char* -> str or str -> bytes convert issues with the cython code which are being fixed under prebuild_arch function in the class.

• There was also an error from _distutils in setuptools during the build which is fixed by sending custom_build_ext as a cmdclass argument.

    raise DistutilsClassError(
distutils.errors.DistutilsClassError: command class <class 'setuptools.command.build_ext.build_ext'> must subclass Command

My working app

The app creates a sketch style animation from an uploaded image. Get the latest release.

Final thought

I am very sure that, there would much more efficient way (may be a custom patch or something) to do the PyAV build and I will let the developers close this issue if they find my analysis helpful. A great thanks to the Kivy world.