pip fails to build wheels for cffi (gcc not found) in ubuntu.1804.armarch.open using image alpine-3.15-helix-arm32v7

[carlossanlop]

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=469944

Build leg reported

https://dev.azure.com/dnceng-public/public/_build/results?buildId=469944&view=logs&j=fd49b2ea-3ee3-5298-d793-bdb9fd631e7e&t=26d44400-a1b8-5ed1-d719-0cc35b38251b

Pull Request

dotnet/runtime#93906

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "Building wheel for cffi (pyproject.toml) did not run successfully.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

Happens in (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.15-helix-arm32v7-20230807201723-7ea784e

The repo had some recent PRs modifying the alpine images:

• main branch: Update alpine arm32 in CI runtime#94683
• release/8.0-staging branch: [release/8.0-staging] Update alpine arm32 in release 8.0 CI runtime#94609
• Change file handle limit in main: Add file handle limit setting for Alpine runtime#94439
• And this one is still open but incoming: update Alpine test images runtime#94487

My PR is not making any changes that could be causing this.

This error is only visible when there's an actual test error happening in these machines, otherwise, the CI leg passes.

Log file with the error: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-9e1b570d2b3441d698/System.IO.Compression.Brotli.Tests/1/console.b000a501.log?helixlogtype=result

Collecting cryptography>=39.0.1
  Downloading cryptography-41.0.5.tar.gz (630 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 630.5/630.5 kB 10.1 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [124 lines of output]
      Collecting setuptools>=61.0.0
        Downloading setuptools-68.2.2-py3-none-any.whl (807 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 807.9/807.9 kB 13.9 MB/s eta 0:00:00
      Collecting wheel
        Downloading wheel-0.41.3-py3-none-any.whl (65 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 kB 3.7 MB/s eta 0:00:00
      Collecting cffi>=1.12
        Downloading cffi-1.16.0.tar.gz (512 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 512.9/512.9 kB 12.1 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting setuptools-rust>=0.11.4
        Downloading setuptools_rust-1.8.1-py3-none-any.whl (26 kB)
      Collecting pycparser
        Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
      Collecting tomli>=1.2.1
        Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
      Collecting semantic-version<3,>=2.8.2
        Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
      Building wheels for collected packages: cffi
        Building wheel for cffi (pyproject.toml): started
        Building wheel for cffi (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error
      
        × Building wheel for cffi (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [82 lines of output]
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
      
                No working compiler found, or bogus compiler options passed to
                the compiler from Python's standard "distutils" module.  See
                the error messages above.  Likely, the problem is not related
                to CFFI but generic to the setup.py of any Python package that
                tries to compile C code.  (Hints: on OS/X 10.8, for errors about
                -mno-fused-madd see http://stackoverflow.com/questions/22313407/
                Otherwise, see https://wiki.python.org/moin/CompLangPython or
                the IRC channel #python on irc.libera.chat.)
      
                Trying to continue anyway.  If you are trying to install CFFI from
                a build done in a different context, you can ignore this warning.
      
            /tmp/pip-build-env-bcny16u4/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
            !!
      
                    ********************************************************************************
                    The license_file parameter is deprecated, use license_files instead.
      
                    This deprecation is overdue, please update your project and remove deprecated
                    calls to avoid build errors in the future.
      
                    See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
                    ********************************************************************************
      
            !!
              parsed = self.parsers.get(option_name, lambda x: x)(value)
            running bdist_wheel
            running build
            running build_py
            creating build
            creating build/lib.linux-aarch64-cpython-39
            creating build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/ffiplatform.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/vengine_cpy.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/recompiler.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/__init__.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/commontypes.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/cparser.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/_shimmed_dist_utils.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/_imp_emulation.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/lock.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/api.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/backend_ctypes.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/error.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/model.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/pkgconfig.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/cffi_opcode.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/verifier.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/setuptools_ext.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/vengine_gen.py -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/_cffi_include.h -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/parse_c_type.h -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/_embedding.h -> build/lib.linux-aarch64-cpython-39/cffi
            copying src/cffi/_cffi_errors.h -> build/lib.linux-aarch64-cpython-39/cffi
            running build_ext
            building '_cffi_backend' extension
            creating build/temp.linux-aarch64-cpython-39
            creating build/temp.linux-aarch64-cpython-39/src
            creating build/temp.linux-aarch64-cpython-39/src/c
            gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fomit-frame-pointer -g -fno-semantic-interposition -fomit-frame-pointer -g -fno-semantic-interposition -fomit-frame-pointer -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000 -fPIC -DFFI_BUILDING=1 -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.9 -c src/c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-39/src/c/_cffi_backend.o
            error: command 'gcc' failed: No such file or directory
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for cffi
      Failed to build cffi
      ERROR: Could not build wheels for cffi, which is required to install pyproject.toml-based projects
      
      [notice] A new release of pip available: 22.2.2 -> 23.3.1
      [notice] To update, run: pip install --upgrade pip
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=469944
Error message validated: Building wheel for cffi (pyproject.toml) did not run successfully.
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/15/2023 12:17:17 AM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[dougbu]

on one hand, I'm not sure why we set $HELIX_PYTHONPATH and $PYTHONPATH or otherwise mess w/ Python inside Docker containers. @dotnet/dnceng does anyone remember why we do this in every "contained" work item❔ (MattGal's !23438 made a small change in this area but history starts w/ alexperovich's !3044.) my best guess is we decided to validate queues using Python even inside containers.

on the other hand, the recent runtime changes were back to a previously-used Alpine Docker image. seems like something changed w/ the various Python packages while they were using a different (newer) container.

bottom line, we don't have much experience w/ the floating pip version we download and install in these containers.

"fix" choices may include

1. remove the runtime_python_requirements.txt installation from Helix work items that execute in a container. https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/alpine/3.15/helix/arm32v7/Dockerfile installs helix-scrips/ and sets up an old-style virtualenv but it's not clear customers need anything more than some Python version in the container. our testing should also work fine w/o those packages installed if we stop checking the helix-scripts/ version.
2. explicitly control the pip version installed in our *-helix-* containers. the ^^ Dockerfile just grabs the latest version at line 29
3. change resources/helix-scripts/runtime_python_requirements.txt to limit the cffi version to one that has a usable .whl available for this platform. might be doable but could require some experimentation

/cc @epananth and @davfost due to Docker implications /fyi @lbussell as well
/cc @missymessa since doing something here seems urgent and you're on SSA
/cc @ilyas1974 and @garath in case you instead want to handle this issue in Operations

[garath]

since doing something here seems urgent

Affects customers, so I believe the proecss is that SSA should triage and establish impact

[dougbu]

set aside my general comments about why we need helix-scripts/ or to meet its Python requirements in Docker containers. removing that everywhere could break some customers. we could add an issue to the Docker epic around finding whether our customers need Python in containers and (maybe) removing it more broadly. a total removal wasn't intended to be my (1) choice.

---

rethinking the options above, we seem to have four choices. three involve updates to https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/alpine/3.15/helix/arm32v7/Dockerfile and the runtime team picking up a new Docker tag; the last changes https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/resources/helix-scripts/runtime_python_requirements.txt and requires a helix-machines rollout (currently planned for the 29th). updating the Dockerfile likely involves reverting @hoyosjs' recent change which may have broken the runtime team (so that the next tag doesn't cause similar issues).

1. remove the # Install Helix Dependencies portion of this particular Dockerfile. the intersection of pipelines testing in this container w/ those using Python in their tests should be very small, if it exists at all. we don't use this container in DncEng testing and have no other requirements on our side for helix-scripts/ in any container
   • I don't know if we already have a "non Helix" Mariner 3.1.5 image that provides what runtime needs. finding such an image would avoid everything except a change to the container the runtime team uses
2. move the # build MsQuic as we don't have Alpine packages (yet) portion of the Dockerfile above the # Install Helix Dependencies portion. this at least has a chance of working if the former portion doesn't delete as much, leaving gcc and similar OS packages around for the cffi build
3. reduce the pip version to 20.3.4 (changing the python3 -m pip install --upgrade pip==22.2.2 line). this should automatically limit the cffi version (see below) because lower pip versions narrow what package versions are available
4. explicitly request cffi==1.15.1 in our runtime_python_requirements.txt file. Build #20231115.10 created a Mariner image, found this version of cffi, and got a compatible wheel i.e., it didn't need to compile anything. one caveat: that was a Mariner 2 image, not 3.15 but this shouldn't be relevant here

(1) sounds fastest but is also somewhat "dirty" in that it makes a -helix- container non-Helix and could be considered a breaking change. (1) gets cleaner if there's already a satisfactory MCR image or the edits create a new image w/o -helix- in its name.

[missymessa]

@carlossanlop can you tell us what the overall impact of this error is for your PR?

[carlossanlop]

I am unsure of the impact to my team, I was hoping dnceng could help initially determine if this was something to worry about or not. But maybe there's someone from @dotnet/runtime-infrastructure that might know if this could affect runtime.

I only happened to notice it in an expected test failure in an unrelated PR when reading the console log, but it probably happens in all runs using the same queue, it just does not get reported as an error at all.

[CarnaViire]

rethinking the options above, we seem to have four choices. three involve updates to https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/alpine/3.15/helix/arm32v7/Dockerfile and the runtime team picking up a new Docker tag

We unfortunately cannot update the tag at the moment, as we are blocked by an external issue (microsoft/msquic#3958) which heavily impacts our CI (dotnet/runtime#91757) -- this was the reason we pinned the tag in the first place (dotnet/runtime#94609)

cc @wfurt @ManickaP

I wonder if this is a problem only 3.15 has? Because 3.15 is EOL now. We were looking into updating to 3.16 (but it is now blocked by the issue above as well)

[wfurt]

Building and installing dependencies when the container start is going to be expensive and unreliable IMHO. I was under impression that the whole prereq repo exists to have everything ready to run the tests.

If the images are meant for testing, we should perhaps add step to verify that Helix is able to run.

And I think we can temporarily disable Quic tests on Alpine arm32 @CarnaViire. Not great but we should not block everything else IMHO. (or we can investigate and fix msquic ours selves or move back to released version instead of tracking current main)