Failing DOI builds on `arm32` architectures

[yosifkit]

There are a few different causes on the arm32 architectures, but I wanted to open this to make you aware of the build failures. Happy to help where I can.

• satosa:8.4.0-bookworm on arm32v7
  • logs of recent build: https://doi-janky.infosiftr.net/job/multiarch/job/arm32v7/job/satosa/2368/execution/node/72/log/

      Building wheels for collected packages: maturin
        Building wheel for maturin (pyproject.toml): started
        Building wheel for maturin (pyproject.toml): still running...
        Building wheel for maturin (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error
    ...
              Downloaded allocator-api2 v0.2.16
            cargo build --manifest-path Cargo.toml --message-format=json-render-diagnostics --release -v --no-default-features --locked
            error: package `anstream v0.6.11` cannot be built because it requires rustc 1.70.0 or newer, while the currently active rustc version is 1.63.0
            Either upgrade to rustc 1.70.0 or newer, or use
            cargo update -p anstream@0.6.11 --precise ver
            where `ver` is the latest version of `anstream` supporting rustc 1.63.0
            error: `cargo build --manifest-path Cargo.toml --message-format=json-render-diagnostics --release -v --no-default-features --locked` failed with code 101
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for maturin
      Failed to build maturin
      ERROR: Could not build wheels for maturin, which is required to install pyproject.toml-based projects
      [end of output]

  • there isn't a newer rustc available in the stable Debian release (https://packages.debian.org/bookworm/rustc)
• satosa:8.4.0-bookworm on arm32v5
  • logs of recent build: https://doi-janky.infosiftr.net/job/multiarch/job/arm32v5/job/satosa/2531/execution/node/72/log/

      Building wheels for collected packages: maturin
        Building wheel for maturin (pyproject.toml): started
        Building wheel for maturin (pyproject.toml): still running...
        Building wheel for maturin (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error
    ...
              Downloaded ahash v0.8.7
            cargo build --manifest-path Cargo.toml --message-format=json-render-diagnostics --release -v --no-default-features --locked
            error: package `toml_edit v0.22.6` cannot be built because it requires rustc 1.70 or newer, while the currently active rustc version is 1.63.0
            Either upgrade to rustc 1.70 or newer, or use
            cargo update -p toml_edit@0.22.6 --precise ver
            where `ver` is the latest version of `toml_edit` supporting rustc 1.63.0
            error: `cargo build --manifest-path Cargo.toml --message-format=json-render-diagnostics --release -v --no-default-features --locked` failed with code 101

  • same as above, there isn't a newer rustc available in the stable Debian release (https://packages.debian.org/bookworm/rustc)
• satosa:8.4.0-alpine3.19 on arm32v6
  • this one just takes too long and times out the 3 hour limit we have on the Jenkins pipeline build step.
    • the host is a c7g.2xlarge on aws (though it shares processing power with arm32v5 and arm32v7 builds)
  • I tried building locally using qemu and it worked in less than 3 hours (nproc of 20 with 32GB ram), so not sure what can be done 😞

They've been failing for a while; I think since Satosa 8.4. Apologies for the delay in reporting these. 🙇 ❤️ satosa:8.4.0-alpine3.19 builds fine on arm32v7 and the builds work on all the other arches: amd64, i386, arm64v8, ppc64le and s390x.

[yosifkit]

It looks like something changed in build for i386 so that it too now fails for satosa:8.4.0-bookworm. See a recent GitHub actions run here:

#12 167.4 Building wheels for collected packages: cryptography, pyjwkest
#12 167.4   Building wheel for cryptography (pyproject.toml): started
#12 167.8   Building wheel for cryptography (pyproject.toml): finished with status 'error'
#12 167.8   error: subprocess-exited-with-error
#12 167.8   
#12 167.8   × Building wheel for cryptography (pyproject.toml) did not run successfully.
#12 167.8   │ exit code: 1
#12 167.8   ╰─> [14 lines of output]
#12 167.8       Running `maturin pep517 build-wheel -i /usr/local/bin/python --compatibility off`
#12 167.8       📦 Including license file "/tmp/pip-install-m5k7bv16/cryptography_0cacdfd8ed694b70880c2ce3c446d276/LICENSE"
#12 167.8       📦 Including license file "/tmp/pip-install-m5k7bv16/cryptography_0cacdfd8ed694b70880c2ce3c446d276/LICENSE.APACHE"
#12 167.8       📦 Including license file "/tmp/pip-install-m5k7bv16/cryptography_0cacdfd8ed694b70880c2ce3c446d276/LICENSE.BSD"
#12 167.8       🍹 Building a mixed python/rust project
#12 167.8       🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
#12 167.8       🐍 Not using a specific python interpreter
#12 167.8       📡 Using build options features, locked from pyproject.toml
#12 167.8       error: package `cryptography-cffi v0.1.0 (/tmp/pip-install-m5k7bv16/cryptography_0cacdfd8ed694b70880c2ce3c446d276/src/rust/cryptography-cffi)` cannot be built because it requires rustc 1.65.0 or newer, while the currently active rustc version is 1.63.0
#12 167.8       
#12 167.8       💥 maturin failed
#12 167.8         Caused by: Failed to build a native library through cargo
#12 167.8         Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.12-64bit" PYO3_PYTHON="/usr/local/bin/python" PYTHON_SYS_EXECUTABLE="/usr/local/bin/python" "cargo" "rustc" "--features" "pyo3/abi3-py37" "--message-format" "json-render-diagnostics" "--locked" "--manifest-path" "/tmp/pip-install-m5k7bv16/cryptography_0cacdfd8ed694b70880c2ce3c446d276/src/rust/Cargo.toml" "--release" "--lib"`
#12 167.8       Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/usr/local/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
#12 167.8       [end of output]

[tianon]

Oof, and ppc64le: 😞

 > [2/6] RUN set -eux; 	savedAptMark="$(apt-mark showmanual)"; 	apt-get update; 	apt-get install -y --no-install-recommends 		cargo 		dpkg-dev 		gcc 		gnupg dirmngr 		libbluetooth-dev 		libbz2-dev 		libc6-dev 		libexpat1-dev 		libffi-dev 		libgdbm-dev 		liblzma-dev 		libncursesw5-dev 		libreadline-dev 		libsqlite3-dev 		libssl-dev 		make 		python3-dev 		tk-dev 		uuid-dev 		wget 		xz-utils 		zlib1g-dev 	; 	pip install --no-cache-dir 		satosa==8.4.0 	; 	apt-mark auto '.*' > /dev/null; 	[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark > /dev/null; 	apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; 	rm -rf /var/lib/apt/lists/*; 	mkdir /etc/satosa; 	chown -R satosa:satosa /etc/satosa:
  Building wheel for pyjwkest (setup.py): started
  Building wheel for pyjwkest (setup.py): finished with status 'done'
  Created wheel for pyjwkest: filename=pyjwkest-1.4.2-py3-none-any.whl size=48061 sha256=c7d505cb47169157495fe809d9c9b35f2343078a0150131fb878be8e8b59b99a
  Stored in directory: /tmp/pip-ephem-wheel-cache-rr5yxbx4/wheels/81/bc/df/8a2c4efeaac7663d28487b0bcebfebafb7432e0272b5828360
Successfully built pycryptodomex MarkupSafe pyjwkest
Failed to build cryptography
ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip