Local build failed with isaac-ros activate --build-local

[Vorwerk-Uwe-Bombosch]

We recently upgraded from the release 3 run_dev.sh development environment to the release 4 isaac-ros and face two issues.

Our goal is building a Docker image that contains the ros-bridge (was: humble, now: jazzy) and a few NVIDIA models on top, namely: isaac-ros-ess, isaav-ros-rtdetr, and isaac-ros-foundationpose. In release 3, we managed this layered image with run_dev.sh and a few Docker-files that we had built to automate the installation and activation of the models. The sequence of layers was controlled by .isaac_ros_common-config, as specified in the NVIDIA documentation. It worked well.

In release 4, we should use isaac-ros activate [option] instead of run_dev.sh and ${ISAAC_ROS_WS}/.isaac-ros-cli/config.yaml instead of .isaac_ros_common-config. This is how I understood it, but here the trouble starts:

1. When running isaac-ros activate, the base image is downloaded, but the additional packages (ess, rtdetr, fp) are not installed. Therefore, we have to run isaac-ros activate --build-local to add these layers. But this triggers a local build - not of noble and jazzy, but of ess, rtdtr, and fp. The end-note of the Conepts Page tells us that this is by intention. In our view, this is a huge downside in comparison to run_dev.sh, since it takes ages even on a g6e.2xlarge Ubuntu 24.04 ES2 host. Is there a feature that enables us to avoid local builds for NVIDIA-delivered packages?
2. The local build of Foundation Pose fails, with an error message we can't interpret (see below). To be clear, this is a build where we ran isaac-ros activate --build-local with just one layer on top - the docker file for FP.

I'll be happy to provide more details if needed. Thank you in advance for any help!

Environment

• AWS E2C g6e.2xlarge with Ubuntu 24.04 (the Isaac SIM 5 AMI)
• Development environment 4.latest set up according to these Instructions
• Docker file for the foundation pose layer:

#
# Dockerfile layer that installs the pose estimation package foundationpose
#

ARG BASE_IMAGE=nvcr.io/nvidia/isaac/ros:noble-ros2_jazzy
FROM ${BASE_IMAGE}

# Install only the FoundationPose package
# Cache mount preserves apt cache, so no apt-get update needed
RUN --mount=type=cache,target=/var/cache/apt \
    echo "Installing Isaac ROS FoundationPose packages..." && \
    apt-get install -y \
        ros-jazzy-isaac-ros-foundationpose \
    && rm -rf /var/lib/apt/lists/*

# Copy setup script to a standard PATH location and ensure it's executable
COPY scripts/installation/setup_foundationpose_model.sh /usr/local/bin/setup_foundationpose.sh

# Ensure script is executable
RUN chmod +x /usr/local/bin/setup_foundationpose.sh

RUN echo "FoundationPose package installed. Run 'setup_foundationpose.sh' at runtime to download assets and convert model files."

Error message (excerpt)

#44 exporting to image
#44 exporting layers
#44 exporting layers 331.5s done
#44 exporting manifest sha256:8650d1bb9b7559d49983674de11d68dd499bb2c9cff262334b23b14057d1bf68 0.0s done
#44 exporting config sha256:6b260abc140834cf3c84d060c5d51dcd57105e7226555b3a114bc8ad05d2a038 0.0s done
#44 pushing layers
#44 pushing layers 1.8s done
#44 ERROR: failed to push nvcr.io/nvidia/isaac/ros:noble_a36082d996bc9582db03aa92fed6d54c-amd64: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://authn.nvidia.com/token?scope=repository%3Anvidia%2Fisaac%2Fros%3Apull%2Cpush&service=registry: 401 Unauthorized
------
 > exporting to image:
------

 1 warning found (use docker --debug to expand):
 - NoEmptyContinuation: Empty continuation line (line 267)
ERROR: failed to solve: failed to fetch anonymous token: unexpected status from GET request to https://authn.nvidia.com/token?scope=repository%3Anvidia%2Fisaac%2Fros%3Apull%2Cpush&service=registry: 401 Unauthorized
docker buildx rm isaaceks-x86_64
Traceback (most recent call last):
  File "/usr/lib/isaac-ros-cli/run_dev.py", line 552, in <module>
    main()
  File "/usr/lib/isaac-ros-cli/run_dev.py", line 541, in main
    build_image_layers(**build_args)
  File "/usr/lib/isaac-ros-cli/build_image_layers.py", line 736, in main
    raise e
  File "/usr/lib/isaac-ros-cli/build_image_layers.py", line 733, in main
    run_shell(build_cmd, capture_output=False, env=env_dict, check=True)
  File "/usr/lib/isaac-ros-cli/build_image_layers.py", line 70, in run_shell
    completed_process = subprocess.run(
                        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'docker  buildx bake noble_a36082d996bc9582db03aa92fed6d54c  --progress=plain --builder isaaceks-x86_64 --provenance=false --push --file /tmp/tmp1efd1cn6/docker-bake.hcl' returned non-zero exit status 1.