bundler: git gems missing from runtime image in multi-stage container builds

[romanblanco]

Describe the issue

When a Gemfile declares a git dependency (via github:, git:, or branch:/ref: options), Hermeto configures bundler to use the pre-fetched local clone via BUNDLE_LOCAL.* config entries.

This causes bundler to use the gem in-place from Hermeto's output directory rather than installing it into BUNDLE_PATH.
In a multi-stage container build, the runtime image ends up missing those gems, causing load errors at startup.

This is a problem for repositories depending on a fork or an unreleased branch:

# Gemfile                                                                                                                                                                                                         
source 'https://rubygems.org'                                                                                                                                                                                     
gem 'json-schema', github: '3scale/json-schema'                                                                                                                                                                   

After hermeto fetch-deps, the generated bundler config contains:

BUNDLE_DISABLE_LOCAL_BRANCH_CHECK: "true"                                                                                                                                                                         
BUNDLE_DISABLE_LOCAL_REVISION_CHECK: "true"                                                                                                                                                                       
BUNDLE_LOCAL.JSON___SCHEMA: "${output_dir}/deps/bundler/json-schema-26487618a684" 
#                              ^^^ points into Hermeto's output directory                                                                                                                                

With BUNDLE_LOCAL.* active, bundler reads the gem in-place from Hermeto's output directory and never copies it into BUNDLE_PATH:

def requires_checkout?
    allow_git_ops? && !local? && !locked_revision_checked_out?
    #                 ^^^
    #                 false when BUNDLE_LOCAL.* is set -> checkout never runs
end

In a multi-stage build the build stage can reach the output directory, but the runtime stage copies only .bundle/ — so the gem disappears:

FROM ruby:3.3 AS build
COPY . .
RUN bundle install   # hermeto supplies BUNDLE_APP_CONFIG

FROM ruby:3.3
COPY --from=build .bundle/ .bundle/
# json-schema is missing here → LoadError at startup

Version

• I have reproduced this issue on the latest version from the main branch

Steps to reproduce

# /tmp/myapp/Gemfile
source 'https://rubygems.org'
gem 'json-schema', github: '3scale/json-schema'

bundle lock # generates Gemfile.lock with a GIT section
hermeto fetch-deps --source /tmp/myapp --output /tmp/hermeto-output bundler

# Inspect the generated config
grep -A1 BUNDLE_LOCAL /tmp/hermeto-output/bundler/config_override/config
# Simulate hermetic bundle install
BUNDLE_APP_CONFIG=/tmp/hermeto-output/bundler/config_override BUNDLE_PATH=/tmp/mybundle bundle install --local

# Gem is not in BUNDLE_PATH
ls /tmp/mybundle/ruby/*/bundler/gems/

Environment

• OS: Fedora Linux 43
• Deployment: localhost
• Ruby: 3.3.10
• Bundler: 2.5.22
• git: 2.54.0

How does this issue affect you, or how did you find it?

Since merging RedHatInsights/compliance-backend#2753, our service is not starting properly.

We're using gem from a forked repository: https://github.com/RedHatInsights/compliance-backend/blob/a5cf3f037fb25772aa47bf7bb447d27b34ceeeae/Gemfile#L71

[eskultety]

@slimreaper35 looking at our design there's no mention of using bundle cache to vendor gems as an alternative approach which we use e.g. in the cargo backend. In fact the design is quite detailed on how bundler is making our lives harder with creation of the bundle/cache directory and we're going an extra mile to prevent it. Do we have a confirmed arbitrary code execution with bundler which wasn't mentioned in the design that we rejected the option of vendoring? I'm asking because bundler seems to have seamless integration with vendoring and using the cache which might be a solution to this issue.

[slimreaper35]

https://github.com/hermetoproject/hermeto/blob/main/docs/design/bundler.md#prefetching

Running a bundler command to fetch the dependencies always executes the Gemfile, which is arbitrary Ruby code.

[eskultety]

https://github.com/hermetoproject/hermeto/blob/main/docs/design/bundler.md#prefetching

Running a bundler command to fetch the dependencies always executes the Gemfile, which is arbitrary Ruby code.

Should have used AI to summarize rather than reading it myself, I clearly forgot how to read.

[eskultety]

@romanblanco I did some investigation on our side.

TL;DR we will likely need to handle this with git configuration as you suggest,yes.

Long version
So, the reason for us setting BUNDLE_LOCAL.* (and the related git check disablements) was that bundler kept contacting the internet and fetched git deps all the time (that's what's in our design doc). What the design doesn't tell is why that is - the way we clone git deps is the "new" cache format with a proper git clone as re-introduced in ruby/rubygems@7d6b6316 . That however, only appears in Bundler 2.6+ . When we were implementing and testing the backend, we used Ruby 3.3 and the bundler that shipped with it - 2.5.21. Older bundler expects the git dependencies to be bare git clones as done in ruby/rubygems@68698892. So, when bundler failed to process our cloned git dep as a bare repository it defaulted to a clean 'git fetch' which would fail, hence the env variables we used and which are also documented as the suggested approach.
In fact, when I switched our mechanism to bare git clones for git deps (just a snippet):

diff --git hermeto/core/package_managers/bundler/gem_models.py hermeto/core/package_managers/bundler/gem_models.py
index c13fcd04..7053a97f 100644
--- hermeto/core/package_managers/bundler/gem_models.py
+++ hermeto/core/package_managers/bundler/gem_models.py
@@ -146,17 +146,13 @@ class GitDependency(_GemMetadata):
         git_repo_path.path.mkdir(parents=True)

         log.info("Cloning git repository %s", self.url)
-        repo = GitRepo.clone_from(
+        GitRepo.clone_from(
             url=str(self.url),
             to_path=git_repo_path.path,
+            bare=True,
             env={"GIT_TERMINAL_PROMPT": "0"},
         )

-        if self.branch is not None:
-            repo.git.checkout(self.branch)
-
-        repo.git.reset("--hard", self.ref)
-

I was able to run our bundler e2e tests just fine. However, I don't like how we're tied to bundler's cache implementation which has clearly changed in the past and now you'd get a warning to migrate to the new format which is what our impl. does . FWIW we also don't serialize gemspec to pure Ruby in our impl. but that doesn't seem to be a big problem with bundler.