[question] what makes ansible-builder v3 takes so long as compared to ansible-builder v1 ?

[nodje]

I can't explain why it now takes me 45~50 mns for a new build with juste 10 python deps 3 collections and python 3.11.

It use to take minutes only with ansible-builder 1.0

It's not that there are any step that takes very long but there seems to be so many steps and each seems to take at least 10s, even to process an ARG or run the simplest of RUN command.

[nitzmahone]

In most cases it should be comparable or faster than 1.x, so I'm curious as well... I'd guess something's going on with your build cache or package management, because we actually reduced several redundant package installation steps from 1.x. Can you share your EE definition?

[Shrews]

Keep in mind that processing of ARG and RUN commands is performed by the container system (docker or podman), and not builder itself. Builder is mostly a front end to building the instruction file that will be fed to either the podman build or docker build command.

[nitzmahone]

Yeah, if even basic no-op things like ARG have a noticeable delay, I'd suspect there's a problem with your container runtime's storage driver (common issues: antivirus, building to a network FS, using a storage driver that assumes cheap snapshots with a FS that doesn't, busted build cache). Obviously building from scratch can take a little longer than 1.x did if you're installing Python/core/runner, since those were previously baked into our base images. You can make your own base image for your EEs so you don't have to do that on every build if that's a problem, but the build cache usually makes that moot anyway, unless you're doing things that drastically change the early stages as you iterate.

[nitzmahone]

... as a data point, the following EE def builds for me via ansible-builder build in:

• 55s with no layers/images cached
• 4.2s on repeated cached builds with no changes
• 33s when adding a new collection to a cached build
  (ansible-builder 3.0.0, Fedora 38, podman 4.7.0):

version: 3

images:
  base_image:
    # completely vanilla image, works with most anything RHELish (Fedora, CentOS Stream)
    name: registry.access.redhat.com/ubi9/ubi:latest

dependencies:
  # install and use a bleeding-edge Python!
  python_interpreter:
    # (optional) provide the name of an OS package to install
    package_system: python3.11
    # (optional) provide the path of the Python interpreter to use for everything else.
    # defaults to /usr/bin/python3; we'll do some sanity checking with this later...
    python_path: /usr/bin/python3.11
  ansible_core:
    # anything valid to `pip install` - a git ref, whatever 
    package_pip: ansible-core<2.16
  ansible_runner:
    package_pip: ansible-runner
  galaxy:
    collections:
    - ansible.utils

[nodje]

thanks for your answers.
Here the exec env I use:

---
version: 3

build_arg_defaults:
  ANSIBLE_GALAXY_CLI_COLLECTION_OPTS: '--pre -c'

dependencies:
  ansible_core:
    package_pip: ansible-core==2.15.4
  ansible_runner:
    package_pip: ansible-runner==2.3.4
  galaxy: requirements.yml
  python: requirements.txt
  system: bindep.txt
  python_interpreter:
    package_system: "python311"
    python_path: "/usr/bin/python3.11"

images:
  base_image:
    name: automationhub.tld/ansible-automation-platform-24/ee-minimal-rhel9:latest

# Custom package manager path for the RHEL based images
options:
  package_manager_path: /usr/bin/microdnf

additional_build_files:
  - src: ansible.cfg
    dest: configs
  - src: ubi9.repo
    dest: configs
  - src: pip.conf
    dest: configs
  - src: rootCA.crt
    dest: configs

additional_build_steps:
  prepend_base:
    - RUN rm -rf /etc/yum.repos.d/*
    - COPY _build/configs/ubi9.repo /etc/yum.repos.d/ubi9.repo
    - COPY _build/configs/pip.conf /etc/pip.conf
    - COPY _build/configs/rootCA.crt /etc/pki/ca-trust/source/anchors/rootCA.crt
    - RUN update-ca-trust
    # enable SHA1 in rhel9 & gcc & python3-devel
    - RUN $PKGMGR install -y crypto-policies-scripts gcc python3.11-devel
    #- RUN update-crypto-policies --set DEFAULT:SHA1

  prepend_galaxy:
    - COPY _build/configs/ansible.cfg /etc/ansible/ansible.cfg
    - ENV ANSIBLE_GALAXY_SERVER_AUTOMATION_HUB_URL=https://s01vl9989234.fr.net.intra
  append_base:
    - RUN alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 0
    - RUN $PKGMGR install -y crypto-policies-scripts gcc python3.11-devel
    - RUN update-crypto-policies --set DEFAULT:SHA1

  prepend_final: |
    RUN whoami
    RUN cat /etc/os-release
  append_final:
    - RUN echo This is a post-install command!

We use this for different EE creations.
the requirements.* files never have more than a couple of deps and collections. This is not what makes the difference.

To illustrate the change, which occurred 3 weeks ago, we were building in 4 to 13mns with ansible-builder v1 (but the Container file was pre-generated and a podman build cmd was used.)

Now with v3, using the same CI, but using a single cmd ansible-builder build -t name:tag -v3 13mns is the fastest build we have, and that's when we rebuild a successfully built image, i.e. only using cache.
A new or partly new image build takes from 30 to 50 mns.

BTW

• we have to use ANSIBLE_GALAXY_CLI_COLLECTION_OPTS: '--pre -c' because or rootCA.crt doesn't seem to be taken into account
• we have issues using update-crypto-policies --set DEFAULT:SHA1 to deactivate the ban on SHA1 on RHEL9

I think both issues must stem from the fact that prepend_base actions are run "in a builder image", I suppose, that is not the finally delivered image.
If I'm guessing alright, I should put those directives somewhere else, but I'm unsure where since the ContainerFile is hidden when using the ansible-builder build cmd. Maybe the doc could make clearer what is happening.