pymupdf
Build for linux/arm64 fails
Description of the bug
Problem
Installing PyMuPDF on linux/arm64 fails while install dependencies on linux/arm64 docker build. To make multiarch image, I've made a github workflow that builds the image for both amd64 and arm64. The build for amd64 is successful, but the build for arm64 stuck with the following error:
https://github.com/comfuture/glados/actions/runs/9674016697/job/26688926160#step:8:848
#16 37.91 Downloading PyMuPDF-1.24.6.tar.gz (30.4 MB)
#16 39.94 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.4/30.4 MB 14.2 MB/s eta 0:00:00
#16 41.66 Getting requirements to build wheel: started
#16 42.61 Getting requirements to build wheel: finished with status 'done'
#16 42.63 Installing backend dependencies: started
#16 57.14 Installing backend dependencies: finished with status 'done'
#16 57.14 Preparing metadata (pyproject.toml): started
#16 117.9 Preparing metadata (pyproject.toml): still running...
... (Repeats forever) ...
... (Repeats forever) ...
How to reproduce the bug
The project has files as follows:
├──.github
│ └── workflows
│ └── deploy.yaml
├── pymupdf_on_arm64
│ └── __init__.py (empty)
├── main.py
├── Dockerfile
└── pyproject.toml
pyproject.toml is as follows:
[build-system]
requires = ["flit_core >=3.9,<4"]
build-backend = "flit_core.buildapi"
[project]
name = "pymupdf_on_arm64"
requires-python = ">=3.10"
dependencies = [
"PyMuPDF>=1.24.6",
]
Dockerfile is as follows:
FROM python:3.11
ADD . /app
WORKDIR /app
ENV FLIT_ROOT_INSTALL=1
RUN pip install flit
RUN flit install -s
CMD [ "python", "main.py" ]
The workflow deploy.yaml is as follows:
name: deploy
on:
push:
branches:
- main
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: build and push
uses: docker/[email protected]
with:
tags: ghcr.io/${{ github.repository }}:${{ github.ref_name }},ghcr.io/${{ github.repository }}:latest
context: .
file: ./Dockerfile
platforms: linux/amd64,linux/arm64
push: false
outputs: type=image,name=target,annotation-index.org.opencontainers.image.description=PyMuPDF Reproduce build error multi-arch image
Then push the project to the repository, and the workflow will be triggered.
PyMuPDF version
1.24.6
Operating system
Linux
Python version
3.11
I'm not sure we should assume the build is broken after just [20m] 40m; ARM builds on Github often run very slowly - i've seen PyMuPDF ARM builds take several hours. I think Github may be using an emulator.
Suggest you let it run until Github times out (6h i think).
My Arm build won't run any version greater than 1.24.5. What do you want us to test so we can give you feedback. I had to update our requirements.txt file to make sure it only uses 1.24.5
@julian-smith-artifex-com what information are you waiting for? I'm happy to do whatever you need to help resolve this. I am stuck at version 1.24.5 until this is resolved. All our docker containers run on an arm server.
@julian-smith-artifex-com what information are you waiting for? I'm happy to do whatever you need to help resolve this. I am stuck at version 1.24.5 until this is resolved. All our docker containers run on an arm server.
The first line of the log at https://github.com/comfuture/glados/actions/runs/9674016697/job/26688926160#step:8:848 is:
2024-06-26T05:22:06.1409863Z Current runner version: '2.317.0'
The end of the log is:
2024-06-26T05:39:00.8320591Z #16 962.0 Preparing metadata (pyproject.toml): still running...
2024-06-26T05:40:01.1951329Z #16 1022.4 Preparing metadata (pyproject.toml): still running...
2024-06-26T05:41:03.5579643Z #16 1084.8 Preparing metadata (pyproject.toml): still running...
2024-06-26T05:42:04.2682315Z #16 1145.5 Preparing metadata (pyproject.toml): still running...
2024-06-26T05:42:08.1229545Z ##[error]The operation was canceled.
2024-06-26T05:42:08.1467241Z Post job cleanup.
...
2024-06-26T05:42:24.2017181Z Cleaning up orphan processes
So it looks like the Github workflow was manually cancelled about 20 min after it started.
So i'd like you to run the build again and leave it to run to completion without cancelling it. [Github will kill it if it hasn't finished after 6 hours]
I'm not familiar with flit. There may be an option to make it show the diagnostics from the underlying build, which would help us to figure out what's happening. With pip one can use -v or -vv for this; maybe there's something similar for flit?
I think this is independent of flit. I have local poetry as the built system and on version 1.24.5 my example docker image is building in 20 secconds with flag --no-cache while with the 1.24.9 it is stuck forever with preparing metadata as shown in the log.
pyproject.toml
[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.12"
pymupdf = "1.24.9"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
poetry.lock
# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand.
[[package]]
name = "pymupdf"
version = "1.24.9"
description = "A high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents."
optional = false
python-versions = ">=3.8"
files = [
{file = "PyMuPDF-1.24.9-cp310-none-macosx_10_9_x86_64.whl", hash = "sha256:da5d9699472bfd1de52975de3eb7efaf5190ac5801b9fc6bcccde603afbe6937"},
{file = "PyMuPDF-1.24.9-cp310-none-macosx_11_0_arm64.whl", hash = "sha256:3d1133983c7ac388a35bbab8dfc4c26a874c05edc47d2038961add2efa4639a8"},
{file = "PyMuPDF-1.24.9-cp310-none-manylinux2014_x86_64.whl", hash = "sha256:5199567353d1543e6c21c626148f8ac9ebb14ce553f2c434fcb9b00e195e1e52"},
{file = "PyMuPDF-1.24.9-cp310-none-musllinux_1_2_x86_64.whl", hash = "sha256:c97f0b2fb201c9d9bc0f15a901641174e8896a9ae9fbe0d5bb1a6f2315cc3ced"},
{file = "PyMuPDF-1.24.9-cp310-none-win32.whl", hash = "sha256:00499b864a56a2168254dce3d0f12048b96e9b3bdd43fecace18a1572342c8d4"},
{file = "PyMuPDF-1.24.9-cp310-none-win_amd64.whl", hash = "sha256:f074e501e883428e7d5480f732ea6a6bd17146f10ebefb9b84957fd32b79f0d4"},
{file = "PyMuPDF-1.24.9-cp311-none-macosx_10_9_x86_64.whl", hash = "sha256:caf43ce86790f95049a5849f2802b5c412b865cd368ece89a39a54fc84aa45cd"},
{file = "PyMuPDF-1.24.9-cp311-none-macosx_11_0_arm64.whl", hash = "sha256:13d06161176e1d4e337f5b5e053b628e4531bab5effb269a83dc38d4deb8e659"},
{file = "PyMuPDF-1.24.9-cp311-none-manylinux2014_x86_64.whl", hash = "sha256:042ad205c7ef615d9fbab7078f6fa8d14f020ed2dfe3a79d803b6171318565b5"},
{file = "PyMuPDF-1.24.9-cp311-none-musllinux_1_2_x86_64.whl", hash = "sha256:b4495833bb0300fc885491928f2cbdf96afb569205dcc256bb4c43e3d1fde7cb"},
{file = "PyMuPDF-1.24.9-cp311-none-win32.whl", hash = "sha256:e53370f3679a7b013c2abb801bb566882dab1fb59646d4b0a717ee0d350c5ab1"},
{file = "PyMuPDF-1.24.9-cp311-none-win_amd64.whl", hash = "sha256:454932e9c7b9cd3057ee83dfe805f551a1382b9e216e87a32eb44c6d6843f966"},
{file = "PyMuPDF-1.24.9-cp312-none-macosx_10_9_x86_64.whl", hash = "sha256:93cc4908259f133c9dc88f5e77329c4b2dbc03fca83126b1efffedb67ade0fb9"},
{file = "PyMuPDF-1.24.9-cp312-none-macosx_11_0_arm64.whl", hash = "sha256:84e1516d4b3e40711b9a6dbaedd30e0a89d6a054ca408a56114ceb5a1461f0d1"},
{file = "PyMuPDF-1.24.9-cp312-none-manylinux2014_x86_64.whl", hash = "sha256:de8b330900c194efeedeb97adab25520479d101fc9aed50d7323dde08698ae24"},
{file = "PyMuPDF-1.24.9-cp312-none-musllinux_1_2_x86_64.whl", hash = "sha256:41c92d69993e7614730205b75d7999b21ca0f929d31b2bb86a4b58d3b1b0451a"},
{file = "PyMuPDF-1.24.9-cp312-none-win32.whl", hash = "sha256:a04af6f3f5f35cb62bc7b3c2e9cfff510aa56c39c53355ecfff40b7cb9773fef"},
{file = "PyMuPDF-1.24.9-cp312-none-win_amd64.whl", hash = "sha256:e2828a79415ae3dd90c629697ace51db7f1e81f426fc2fc034c2151dbe58be6e"},
{file = "PyMuPDF-1.24.9-cp38-none-macosx_10_9_x86_64.whl", hash = "sha256:241913d0c76aacb05acdd8a0e82b1105883ffe6ef3bb4d9742b41d3c5e84d2db"},
{file = "PyMuPDF-1.24.9-cp38-none-macosx_11_0_arm64.whl", hash = "sha256:ff70e26625b6cdd036e2c63b5d6c1897949c0e8b205cd756276f27baadaad340"},
{file = "PyMuPDF-1.24.9-cp38-none-manylinux2014_aarch64.whl", hash = "sha256:8e29bc817afad511072371f24624c7c3b7485a9e656b6a65dc58fecdf5043b08"},
{file = "PyMuPDF-1.24.9-cp38-none-manylinux2014_x86_64.whl", hash = "sha256:d17ec6920f91c43b6e777a017f3aaf44b205a3216771db9e8aa46e78a703f8f6"},
{file = "PyMuPDF-1.24.9-cp38-none-musllinux_1_2_x86_64.whl", hash = "sha256:5cec9d17fdcbd83fa2c90190c22f652a0a51275cf75a29068eea025fff076829"},
{file = "PyMuPDF-1.24.9-cp38-none-win32.whl", hash = "sha256:4f7b19f5c0026db49b7be17901728ed15761c5aa2031f04b01f9eb2e54f1b50e"},
{file = "PyMuPDF-1.24.9-cp38-none-win_amd64.whl", hash = "sha256:e4c867f1cde68ff0e9c7889ea27c4c2c67df80e776f82619888bb69d1e1b27cf"},
{file = "PyMuPDF-1.24.9-cp39-none-macosx_10_9_x86_64.whl", hash = "sha256:b4f85c24050e3778be6c7c1f4d4965fd4385281264798df7b4301b78895053fd"},
{file = "PyMuPDF-1.24.9-cp39-none-macosx_11_0_arm64.whl", hash = "sha256:4e807010ef4e63cfb70dd88fe1fcd1d7e2b4e62ffa2b1dc53b35bc18bf939d8e"},
{file = "PyMuPDF-1.24.9-cp39-none-manylinux2014_aarch64.whl", hash = "sha256:5dac888cc16981e385c886c26de6aabf914059215e028d14cd67767ff0c1288c"},
{file = "PyMuPDF-1.24.9-cp39-none-manylinux2014_x86_64.whl", hash = "sha256:de55817c02e06ff75233ce2487cc5ebcbf585acd694bb69500825ee37789ac79"},
{file = "PyMuPDF-1.24.9-cp39-none-musllinux_1_2_x86_64.whl", hash = "sha256:49cb22196f11c2327f6345554db48cfb2e31ed4f073ca6a872f21ddc4b0619c1"},
{file = "PyMuPDF-1.24.9-cp39-none-win32.whl", hash = "sha256:46b1f84816c666e1c82f4249c1e815e92c462633255d72da20751eaad125d0f0"},
{file = "PyMuPDF-1.24.9-cp39-none-win_amd64.whl", hash = "sha256:4fa45474d63715c707e3c3a6ebeeee75fd7aaa180512b75863e437f6876dfa86"},
{file = "PyMuPDF-1.24.9.tar.gz", hash = "sha256:3692a5e824f10dc09bbddabab207f7cd5979831e48dd2f4de1be21e441767473"},
]
[package.dependencies]
PyMuPDFb = "1.24.9"
[[package]]
name = "pymupdfb"
version = "1.24.9"
description = "MuPDF shared libraries for PyMuPDF."
optional = false
python-versions = ">=3.8"
files = [
{file = "PyMuPDFb-1.24.9-py3-none-macosx_10_9_x86_64.whl", hash = "sha256:3c9e694b1fb1bde37a8d3c953fbd0916e7dee8a4650142547d4f832105b17689"},
{file = "PyMuPDFb-1.24.9-py3-none-macosx_11_0_arm64.whl", hash = "sha256:3fd74ee7969712ab457495465da0a61aab44d8cf9b71b9ef51910a8c6a90ad57"},
{file = "PyMuPDFb-1.24.9-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:198f6b3713b6f980fa96c1099be0d5459c7d43c593299948f0ba528577e6bf46"},
{file = "PyMuPDFb-1.24.9-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:ae044ebc8299f5a3ba822a6dfe97285dffd6c66cba194bc39180aa189a2755c9"},
{file = "PyMuPDFb-1.24.9-py3-none-win32.whl", hash = "sha256:20ea17fd5799dcf7813ec099c0ce303f763e6e4ba8d0f54d5f84e4df90c3a340"},
{file = "PyMuPDFb-1.24.9-py3-none-win_amd64.whl", hash = "sha256:c6b8adc0b9c91ff0f657440a816ad2130429a808cd53ff273f3e72532e526bdc"},
{file = "PyMuPDFb-1.24.9.tar.gz", hash = "sha256:5505f07b3dded6e791ab7d10d01f0687e913fc75edd23fdf2825a582b6651558"},
]
[metadata]
lock-version = "2.0"
python-versions = "^3.12"
content-hash = "08949dcff98f79abd27d217e6b983b85b9f0f289b9291648c6eb6903694e2a48"
Dockerfile
FROM python:3.12-bullseye
RUN pip install poetry
# Set the working directory
WORKDIR /app
COPY ./pyproject.toml .
COPY ./poetry.lock .
RUN poetry install
ENV PYTHONPATH=/app
# Expose the application port
EXPOSE 8000
Linux/aarch64 wheels were missing from pypi.org until a few days ago.
So it's possible that pip or poetry or other Python packaging tools have been trying to build from source, which can take a long time (e.g. hours).
The wheels are now available on pypi.org, so please retry, and report back here with the results.
That makes sense
it still took 160s instead of 20s with prior version(1.24.5), which is an 8x increase, but it completed the build and worked on my local machine.
for my use case this is fine, thank you