Resolve and install content-addressable gems in Bundler

Encode content-addressable gems in the v1 compact index (no v2 endpoint): the
per-gem info line carries the content address in the version token's platform
slot and the real platform in a platform: requirement token, gated by a
rubygems:>= requirement so old clients ignore the rows:

  1.0.0-<sha> |checksum:<sha256>,ruby:<req>,rubygems:>= 4.1.0.dev,platform:= <real>

- endpoint_specification.rb: parse the platform: token into platform_requirement;
  content_addressable?; installable_on_platform? matches on the real platform
  (since @platform holds the sha); version_suffix reconstructs the download name.
- match_platform.rb: when a skinny binary compatible with the running Ruby
  exists, prefer it exclusively; otherwise fall back to the fat/ruby variant so a
  usable binary is never stranded. Per-minor ~> ranges are disjoint, so at most
  one skinny qualifies (no skinny-vs-skinny tie-break needed).
- lazy_specification.rb: carry platform_requirement/version_suffix; reconstruct
  full_name as name-version-<sha>; on a --local exact-match miss, retry by
  name+version so the portable lockfile entry (name-version-platform) resolves to
  the on-disk name-version-<sha> gem.
- stub_specification.rb: delegate full_name/version_suffix to the RubyGems stub.

Old clients drop the skinny rows because their RubyGems version doesn't satisfy
the rubygems:>= gate (and the <sha> token doesn't match a real platform anyway),
so they fall back to the fat binary.

Assisted-By: devx/4ab7951d-76be-4b93-8ffb-c3581711ac1f

Author: jenshenny

bundler/lib/bundler/endpoint_specification.rb

@@ -5,7 +5,7 @@ module Bundler
   class EndpointSpecification < Gem::Specification
     include MatchRemoteMetadata
 
-    attr_reader :name, :version, :platform, :checksum, :created_at
+    attr_reader :name, :version, :platform, :checksum, :created_at, :platform_requirement
     attr_writer :dependencies
     attr_accessor :remote, :locked_platform
 
@@ -21,11 +21,37 @@ def initialize(name, version, platform, spec_fetcher, dependencies, metadata = n
       @loaded_from          = nil
       @remote_specification = nil
       @locked_platform = nil
+      @platform_requirement = nil
 
       parse_metadata(metadata)
     end
 
+    # A content-addressable ("skinny") binary: the version token carries a SHA
+    # prefix instead of a platform, and the real platform is supplied via the
+    # compact index `platform:` requirement.
+    def content_addressable?
+      !@platform_requirement.nil?
+    end
+
+    # The version suffix (SHA prefix) carried in the version token's platform
+    # slot, e.g. "9f3c1a2b". Used to reconstruct the download/full name. nil for
+    # ordinary gems, whose @platform is a real Gem::Platform or the "ruby" string.
+    def version_suffix
+      @platform.version_suffix if @platform.is_a?(Gem::Platform)
+    end
+
+    # For content-addressable specs, platform matching must use the platform
+    # requirement from metadata, not the SHA prefix stored in @platform.
+    def installable_on_platform?(target_platform)
+      return super unless content_addressable?
+
+      target_platform.nil? ||
+        target_platform == Gem::Platform::RUBY ||
+        @platform_requirement === Gem::Platform.new(target_platform)
+    end
+
     def insecurely_materialized?
+      return false if content_addressable?
       @locked_platform.to_s != @platform.to_s
     end
 
@@ -162,6 +188,10 @@ def parse_metadata(data)
           @required_rubygems_version = Gem::Requirement.new(v)
         when "ruby"
           @required_ruby_version = Gem::Requirement.new(v)
+        when "platform"
+          # e.g. v == ["= x86_64-linux-musl"]; strip the requirement operator.
+          platform_string = Array(v).last.to_s.sub(/\A\s*=?\s*/, "")
+          @platform_requirement = Gem::Platform.new(platform_string) unless platform_string.empty?
         when "created_at"
           value = v.is_a?(Array) ? v.last : v
           if value.is_a?(String)

bundler/lib/bundler/lazy_specification.rb

@@ -9,7 +9,7 @@ class LazySpecification
     include ForcePlatform
 
     attr_reader :name, :version, :platform, :materialization
-    attr_accessor :source, :remote, :force_ruby_platform, :dependencies, :required_ruby_version, :required_rubygems_version, :overrides
+    attr_accessor :source, :remote, :force_ruby_platform, :dependencies, :required_ruby_version, :required_rubygems_version, :overrides, :platform_requirement, :version_suffix
 
     #
     # For backwards compatibility with existing lockfiles, if the most specific
@@ -31,9 +31,25 @@ def self.from_spec(s)
       lazy_spec.required_ruby_version = s.required_ruby_version
       lazy_spec.required_rubygems_version = s.required_rubygems_version
       lazy_spec.overrides = s.overrides if s.is_a?(LazySpecification)
+      lazy_spec.platform_requirement = s.platform_requirement if s.respond_to?(:platform_requirement)
+      lazy_spec.version_suffix = s.version_suffix if s.respond_to?(:version_suffix)
       lazy_spec
     end
 
+    def content_addressable?
+      !@platform_requirement.nil?
+    end
+
+    # Match using the platform requirement from metadata when content-addressable,
+    # since @platform holds the SHA prefix rather than a real platform.
+    def installable_on_platform?(target_platform)
+      return super unless content_addressable?
+
+      target_platform.nil? ||
+        target_platform == Gem::Platform::RUBY ||
+        @platform_requirement === Gem::Platform.new(target_platform)
+    end
+
     def initialize(name, version, platform, source = nil, **materialization_options)
       @name          = name
       @version       = version
@@ -64,7 +80,9 @@ def source_changed?
     end
 
     def full_name
-      @full_name ||= if platform == Gem::Platform::RUBY
+      @full_name ||= if @version_suffix
+        "#{@name}-#{@version}-#{@version_suffix}"
+      elsif platform == Gem::Platform::RUBY
         "#{@name}-#{@version}"
       else
         "#{@name}-#{@version}-#{platform}"
@@ -140,6 +158,17 @@ def materialize_for_installation
 
       if use_exact_resolved_specifications?
         spec = materialize(self) {|specs| choose_compatible(specs, fallback_to_non_installable: false) }
+
+        # The exact locked full_name wasn't found in the index (materialize returns
+        # self). This happens for content-addressable gems: the installed full_name
+        # is name-version-<sha>, while the lockfile pins the portable
+        # name-version-platform. Retry by name+version and let platform/ruby
+        # selection pick the right binary.
+        if spec.equal?(self)
+          by_name = materialize([name, version]) {|specs| resolve_best_platform(specs) }
+          return by_name unless by_name.nil? || by_name.equal?(self)
+        end
+
         return spec if spec
 
         # Exact spec is incompatible; in frozen mode, try to find a compatible platform variant

bundler/lib/bundler/match_platform.rb

@@ -22,9 +22,34 @@ def self.select_best_local_platform_match(specs, force_ruby: false)
       Gem::Platform.sort_best_platform_match(matching, local)
     end
 
+    def self.content_addressable?(spec)
+      spec.respond_to?(:content_addressable?) && spec.content_addressable?
+    end
+
+    # A skinny binary targets exactly one Ruby minor (e.g. `~> 3.2.0`). Those
+    # ranges are disjoint, so at most one skinny variant per (gem, version,
+    # platform) is compatible with the running Ruby. Only a Ruby-compatible
+    # skinny may displace the fat/ruby fallback.
+    def self.usable_skinny?(spec)
+      return false unless content_addressable?(spec)
+
+      req = spec.required_ruby_version
+      req.nil? || req.none? || req.satisfied_by?(Gem.ruby_version)
+    end
+
     def self.select_all_platform_match(specs, platform, force_ruby: false, prefer_locked: false)
       matching = specs.select {|spec| spec.installable_on_platform?(force_ruby ? Gem::Platform::RUBY : platform) }
 
+      # Prefer the skinny (content-addressable) binary: when a skinny variant
+      # compatible with the running Ruby exists, choose it exclusively. If no
+      # skinny matches this Ruby, keep the fat/ruby variants as a fallback so we
+      # never strand a usable binary. Because per-minor `~>` ranges are disjoint,
+      # at most one skinny qualifies -- no skinny-vs-skinny tie-break is needed.
+      unless force_ruby
+        skinny = matching.select {|spec| usable_skinny?(spec) }
+        matching = skinny if skinny.any?
+      end
+
       specs.each(&:force_ruby_platform!) if force_ruby
 
       if prefer_locked

bundler/lib/bundler/stub_specification.rb

@@ -13,6 +13,19 @@ def insecurely_materialized?
       false
     end
 
+    # Delegate to the underlying RubyGems stub so content-addressable binaries
+    # use their content-addressed name (name-version-<sha>), matching the install
+    # directory, rather than recomputing name-version-platform.
+    def full_name
+      stub.full_name
+    end
+
+    # The version suffix lives on the stub line, not in the gemspec body, so read
+    # it from the RubyGems stub rather than method_missing'ing to the full spec.
+    def version_suffix
+      stub.version_suffix
+    end
+
     attr_reader :checksum
     attr_accessor :stub, :ignored