Add content-addressable v1 index demo and tests

Self-contained, native end-to-end tests for content-addressable ("skinny")
binaries served from a v1 compact index (no v2 endpoint, no rubygems.org, no
containers):

- demo/: a tiny native gem whose required_ruby_version pins the building Ruby's
  minor, so it builds as a skinny binary.
- fake_compact_index.rb: a dependency-free threaded HTTP server that serves a
  directory as a v1 compact index (/versions, /info/<gem>, /gems/*).
- prove_resolution.rb / test_resolution.sh: using the real parser, the real
  EndpointSpecification, and the real MatchPlatform selection, prove a new client
  picks the skinny variant over source+fat, and an old client drops it via the
  rubygems:>= gate.
- test_local.sh: build, gem install (gems/name-version-<sha>/ + sha in the stub),
  bundle install --local (lockfile round-trip), require, idempotent re-install.
- test_remote_v1.sh: bundle install against the fake v1 server; the skinny gem
  is selected and downloaded as /gems/name-version-<sha>.gem.
- test_remote_v1_oldclient.sh: a stock RubyGems/Bundler older than the gate
  installs the fat binary and never requests the skinny gem, while the patched
  client picks the skinny one, against the same index.

README.md documents what each proves and how this differs from PR #168.

Assisted-By: devx/4ab7951d-76be-4b93-8ffb-c3581711ac1f

Author: jenshenny

dev/content-addressable-v1-demo/README.md

@@ -0,0 +1,146 @@
+# Content-addressable gems in the **v1** compact index — resolution proof
+
+This prototype implements the proposal where
+content-addressable ("skinny") gems are **directly in the v1 compact index**
+, gated by a `rubygems:>=` requirement so that:
+
+- **old clients ignore** the skinny rows (they don't satisfy `rubygems:>=`), and
+- **new clients process** them, match on the `platform:` metadata token, and
+  download the content-addressed `name-version-<sha>.gem`.
+
+A skinny `/info/hola` entry looks like (content-addressable rows last):
+
+```
+---
+1.0.0                      |checksum:…,ruby:>= 3.1,rubygems:>= 3.3.22
+1.0.0-x86_64-linux         |checksum:…,ruby:>= 3.1
+1.0.0-ef716ba7a6           |checksum:…,ruby:~> 4.0.0,rubygems:>= 4.1.0.dev,platform:= x86_64-linux
+```
+
+- The version slot's "platform" (`ef716ba7a6`) is the **content address** =
+  `sha256(.gem)[0,10]`. It becomes the gem's `full_name` and download path.
+- The `platform:` **metadata token** carries the real platform, used for
+  compatibility matching.
+- `rubygems:>= 4.1.0.dev` is the gate: this is the RubyGems version
+  content-addressable support is assumed to ship in.
+
+## Run it
+
+```bash
+# 1. Resolution proof — no compiler/network needed (pure parser + selection)
+./dev/content-addressable-v1-demo/test_resolution.sh
+
+# 2. Local install path — build a native gem, gem install, bundle install --local
+./dev/content-addressable-v1-demo/test_local.sh
+
+# 3. Remote install path — serve a v1 index from a fake server, bundle install
+PORT=8920 ./dev/content-addressable-v1-demo/test_remote_v1.sh
+
+# 4. Old-client compatibility — stock RubyGems/Bundler ignores the skinny rows
+PORT=8920 ./dev/content-addressable-v1-demo/test_remote_v1_oldclient.sh
+```
+
+Each ends with `ALL GOOD`. `test_local.sh` / `test_remote_v1.sh` build a tiny
+native gem (`demo/`) and need a C toolchain plus a one-time `rake-compiler`
+install. They use the Ruby on your `PATH` (the demo gemspec pins `~>` that
+Ruby's minor, so it is "skinny" for whatever Ruby you run); override with
+`RUBY_PREFIX=/opt/rubies/X`.
+
+## What it proves
+
+### Resolution (`test_resolution.sh`)
+
+For a single `hola 1.0.0` published as **source**, **fat** (regular
+precompiled), and **skinny** (content-addressable) variants, using the *real*
+client code (`Gem::Resolver::APISet::GemParser`, `Bundler::EndpointSpecification`
+built exactly like `Bundler::Fetcher#specs`, and
+`Bundler::MatchPlatform.select_best_platform_match`):
+
+1. **New client picks the skinny one** and its download name reconstructs to
+   `hola-1.0.0-ef716ba7a6.gem`.
+2. **Old client ignores the skinny rows.** The `rubygems:>= 4.1.0.dev`
+   requirement is not satisfied by a pre-CA RubyGems (e.g. 3.5.0), so
+   `matches_current_rubygems?` is false and the row is dropped; the old client
+   falls back to the fat binary.
+
+### Install paths (`test_local.sh`, `test_remote_v1.sh`)
+
+These exercise the install machinery ported from
+[PR #168](https://github.com/Shopify/rubygems/pull/168) on top of the v1 branch:
+
+- **Build:** `rake native gem` content-addresses the skinny gem, renaming
+  `hola-1.0.0-<platform>.gem` → `hola-1.0.0-<sha>.gem` in `Gem::Package.build`.
+- **`--local`:** `gem install` installs under `gems/hola-1.0.0-<sha>/` and
+  records the sha in the stub line (`# stub: hola 1.0.0 <platform> lib <sha>`),
+  so `full_name` reconstructs the content-addressed name offline.
+  `bundle install --local` resolves it (the lockfile stays portable —
+  `hola (1.0.0-<platform>)` — and Bundler bridges back to the on-disk
+  `name-version-<sha>` gem), `bundle exec require` + `bundle list` work, and
+  re-install is idempotent.
+- **Remote (v1):** Bundler fetches `/versions` and `/info/hola` (unprefixed —
+  **no `/v2/` endpoint**), selects the skinny variant, downloads
+  `/gems/hola-1.0.0-<sha>.gem`, installs it, and `require` works.
+
+### Old-client compatibility (`test_remote_v1_oldclient.sh`)
+
+The most important backwards-compat guarantee: a **new publisher** serving
+content-addressable gems must not break **old consumers**. This test serves a
+real fat binary plus a gated skinny row, then installs with two clients against
+the same v1 index:
+
+- **Old client** (the stock RubyGems/Bundler on `PATH` — here 4.0.10, older than
+  the `4.1.0.dev` gate and without the patches): ignores the skinny row
+  (`rubygems:>= 4.1.0.dev` is unsatisfied, and the `<sha>` token doesn't match
+  the local platform anyway), installs `hola-1.0.0-<platform>.gem`, and
+  `require` works. It **never requests** the skinny `.gem`, and doesn't choke on
+  the `<sha>` version token or the `platform:` metadata token.
+- **New client** (patched): selects and downloads the skinny
+  `hola-1.0.0-<sha>.gem`.
+
+The server request log shows exactly which `.gem` each client fetched, proving
+the split.
+
+## Client changes that make this work (vs. master)
+
+All on top of `master`; **no v2 endpoint involved**. Naming convention on this
+branch: the sha identity is `version_suffix`; the real platform from metadata is
+`platform_requirement`.
+
+### RubyGems core (build + install)
+
+| File | Change |
+| --- | --- |
+| `lib/rubygems/platform.rb` | Preserve a 10-hex-char **version suffix** verbatim instead of normalizing it to `os="unknown"`. Kept in its own `version_suffix` field; `==`/`hash`/`===` treat it as an exact-match token. |
+| `lib/rubygems/specification.rb` | `content_addressable?` / `content_addressable_ruby_abi` (skinny detection: `~> X.Y.Z` and rake-compiler's `>= X.Y, < X.(Y+1).dev`); `to_ruby` writes the version suffix into the stub line. |
+| `lib/rubygems/package.rb` | `Gem::Package.build` renames skinny gems to `name-version-<sha>.gem`. |
+| `lib/rubygems/package_task.rb` | Move the (renamed) built file to the package dir. |
+| `lib/rubygems/basic_specification.rb` | `version_suffix` accessor; `full_name` returns `name-version-<sha>` when set. |
+| `lib/rubygems/stub_specification.rb` | Read the optional 5th stub-line field (the sha); `full_name` reconstructs the content-addressed name; `to_spec` carries the suffix onto the loaded full spec. |
+| `lib/rubygems/installer.rb` | `assign_version_suffix` derives the sha from the gem's bytes before any path is computed. |
+
+### Bundler (resolution + install bridge)
+
+| File | Change |
+| --- | --- |
+| `bundler/lib/bundler/endpoint_specification.rb` | Parse the `platform:` metadata token into `platform_requirement`; add `content_addressable?`, `version_suffix`, and a platform match that uses the platform requirement. |
+| `bundler/lib/bundler/match_platform.rb` | When a skinny variant compatible with the running Ruby exists, prefer it **exclusively** over fat/source (per-Ruby-minor `~>` ranges are disjoint, so at most one qualifies). |
+| `bundler/lib/bundler/lazy_specification.rb` | Carry `platform_requirement`/`version_suffix`; reconstruct `full_name` as `name-version-<sha>`; on a `--local` exact-match miss, retry by name+version so the portable lockfile entry resolves to the on-disk `name-version-<sha>` gem. |
+| `bundler/lib/bundler/stub_specification.rb` | Delegate `full_name`/`version_suffix` to the underlying RubyGems stub. |
+
+## Differences from PR #168
+
+- **No v2 endpoint.** PR #168 negotiates a `/v2/` compact-index namespace to hide
+  content-addressable entries from old clients. This branch keeps everything in
+  the **v1** index and hides skinny rows from old clients with a `rubygems:>=`
+  gate instead, so `compact_index_client.rb` / `cache.rb` /
+  `fetcher/compact_index.rb` are left untouched.
+- **Naming.** PR #168 uses `content_address` / `real_platform`; this branch uses
+  `version_suffix` / `platform_requirement`.
+
+## Known nuance
+
+On the **remote** path the gem currently installs into
+`gems/hola-1.0.0-<real-platform>/`, whereas the **`--local`** path installs into
+`gems/hola-1.0.0-<sha>/`. Both are internally consistent and work for a single
+active Ruby, but they disagree on the on-disk directory name (same nuance noted
+in PR #168).

dev/content-addressable-v1-demo/demo/.gitignore

@@ -0,0 +1,5 @@
+/tmp/
+/pkg/
+/lib/hola/*.bundle
+/lib/hola/*.so
+*.gem

dev/content-addressable-v1-demo/demo/Rakefile

@@ -0,0 +1,5 @@
+require "rake/extensiontask"
+spec = Gem::Specification.load("hola.gemspec")
+Rake::ExtensionTask.new("hola", spec) do |ext|
+  ext.lib_dir = "lib/hola"
+end

dev/content-addressable-v1-demo/demo/ext/hola/extconf.rb

@@ -0,0 +1,2 @@
+require "mkmf"
+create_makefile("hola/hola")

dev/content-addressable-v1-demo/demo/ext/hola/hola.c

@@ -0,0 +1,9 @@
+#include <ruby.h>
+
+static VALUE hola(VALUE self) {
+  return rb_str_new_cstr("hola from native");
+}
+
+void Init_hola(void) {
+  rb_define_global_function("hola", hola, 0);
+}

dev/content-addressable-v1-demo/demo/hola.gemspec

@@ -0,0 +1,13 @@
+Gem::Specification.new do |s|
+  s.name        = "hola"
+  s.version     = "1.0.0"
+  s.summary     = "content-addressable native gem demo"
+  s.authors     = ["poc"]
+  s.files       = Dir["lib/**/*.rb"] + Dir["ext/**/*"]
+  s.extensions  = ["ext/hola/extconf.rb"]
+  # Pin to a single Ruby ABI (the Ruby building this gem) so it is a "skinny"
+  # binary -> content-addressable. Computed at build time so the demo is skinny
+  # for whatever Ruby you run it with, not just 3.3.
+  minor = RUBY_VERSION.split(".").first(2).join(".")
+  s.required_ruby_version = "~> #{minor}.0"
+end

dev/content-addressable-v1-demo/demo/lib/hola.rb

@@ -0,0 +1 @@
+require "hola/hola"

dev/content-addressable-v1-demo/fake_compact_index.rb

@@ -0,0 +1,60 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+#
+# Minimal compact-index gem server for testing content-addressable ("skinny")
+# binaries end-to-end over the **v1** compact index, with no external deps
+# (raw TCPServer, threaded).
+#
+# Serves a directory tree as static files over HTTP/1.1:
+#   GET /versions       -> <root>/versions
+#   GET /info/<gem>     -> <root>/info/<gem>
+#   GET /gems/<file>.gem -> <root>/gems/<file>.gem
+#
+# Always returns the full body (200); ignores Range (Bundler's compact-index
+# updater handles a full response even when it sent a Range header). Logs every
+# request to stderr so you can see exactly what Bundler asks for.
+#
+# Usage: ruby fake_compact_index.rb <root_dir> <port>
+
+require "socket"
+
+root = File.expand_path(ARGV[0] || ".")
+port = Integer(ARGV[1] || "8899")
+
+server = TCPServer.new("127.0.0.1", port)
+warn "[fake-index] serving #{root} on http://127.0.0.1:#{port}"
+
+loop do
+  conn = server.accept
+  Thread.new(conn) do |c|
+    begin
+      request_line = c.gets
+      next unless request_line
+      method, path, = request_line.split(" ")
+      # drain headers
+      while (line = c.gets) && line != "\r\n"; end
+
+      clean = path.split("?", 2).first.to_s
+      file = File.join(root, clean)
+      warn "[fake-index] #{method} #{clean} -> #{File.exist?(file) ? "200" : "404"}"
+
+      if File.file?(file)
+        body = File.binread(file)
+        ctype = clean.end_with?(".gem") ? "application/octet-stream" : "text/plain"
+        head = +"HTTP/1.1 200 OK\r\n"
+        head << "Content-Type: #{ctype}\r\n"
+        head << "Content-Length: #{body.bytesize}\r\n"
+        head << "Accept-Ranges: none\r\n"
+        head << "Connection: close\r\n\r\n"
+        c.write(head)
+        c.write(body) unless method == "HEAD"
+      else
+        c.write("HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
+      end
+    rescue => e
+      warn "[fake-index] error: #{e.class}: #{e.message}"
+    ensure
+      c.close rescue nil
+    end
+  end
+end

dev/content-addressable-v1-demo/prove_resolution.rb

@@ -0,0 +1,130 @@
+# frozen_string_literal: true
+
+# Proof: content-addressable ("skinny") gems encoded in the **v1** compact index
+# are (a) chosen by a new client over the fat and source variants, and (b)
+# ignored by old clients via the `rubygems:` requirement gate.
+#
+# This deliberately uses the *real* client code paths -- the host RubyGems
+# compact-index line parser (Gem::Resolver::APISet::GemParser), the real
+# Bundler::EndpointSpecification (built exactly like Bundler::Fetcher#specs
+# does), and the real Bundler::MatchPlatform platform selection -- so the
+# demo proves the production behaviour, not a reimplementation.
+#
+# Run with the patched RubyGems + Bundler from this checkout:
+#
+#   ruby --disable-gems -Ilib -rrubygems -Ibundler/lib \
+#     dev/content-addressable-v1-demo/prove_resolution.rb
+#
+# (test_resolution.sh wires that up for you.)
+
+require "bundler"
+require "bundler/endpoint_specification"
+require "bundler/match_platform"
+require "rubygems/resolver"
+
+# The RubyGems version at which content-addressable support ships. New clients
+# satisfy this; old clients do not and therefore drop the skinny rows.
+CA_SUPPORTED_FROM = "4.1.0.dev"
+
+LOCAL = Gem::Platform.local                       # e.g. arm64-darwin-24 / x86_64-linux
+RUBY_MINOR = RUBY_VERSION.split(".").first(2).join(".") # e.g. "4.0"
+SHA10 = "ef716ba7a6"                              # sha256(.gem)[0,10]
+
+# A single gem `hola 1.0.0` published in three shapes, exactly as it would
+# appear in the v1 `/info/hola` file. Order mirrors the Slack proposal: the
+# content-addressable rows come last and carry `rubygems:>=` + `platform:`.
+INFO = <<~INFO
+  ---
+  1.0.0 |checksum:#{"a" * 64},ruby:>= 3.1,rubygems:>= 3.3.22
+  1.0.0-#{LOCAL} |checksum:#{"b" * 64},ruby:>= 3.1
+  1.0.0-#{SHA10} |checksum:#{"c" * 64},ruby:~> #{RUBY_MINOR}.0,rubygems:>= #{CA_SUPPORTED_FROM},platform:= #{LOCAL}
+INFO
+
+# Minimal spec fetcher stand-in: EndpointSpecification only needs #uri (for
+# checksum attribution). #fetch_spec is never reached in this offline demo.
+FakeFetcher = Struct.new(:uri) do
+  def fetch_spec(*) = raise("network access not expected in this demo")
+end
+FETCHER = FakeFetcher.new("https://example.test")
+
+# Build EndpointSpecifications from the info file using the real parser, the
+# same way Bundler::Fetcher#specs does (name, version, platform, deps, metadata).
+def build_specs(info)
+  parser = Gem::Resolver::APISet::GemParser.new
+  lines = info.split("\n")
+  body = lines[(lines.index("---") + 1)..]
+  body.map do |line|
+    version, platform, deps, reqs = parser.parse(line)
+    Bundler::EndpointSpecification.new("hola", version, platform, FETCHER, deps, reqs)
+  end
+end
+
+def describe(spec)
+  kind =
+    if spec.content_addressable? then "SKINNY (content-addressable)"
+    elsif spec.platform == Gem::Platform::RUBY then "source  (ruby platform)"
+    else "fat     (precompiled platform)"
+    end
+  "#{spec.full_name.ljust(28)} -> #{kind}"
+end
+
+specs = build_specs(INFO)
+
+puts "Running RubyGems #{Gem::VERSION}, Ruby #{RUBY_VERSION}, platform #{LOCAL}"
+puts
+puts "Parsed v1 /info/hola into #{specs.size} candidate specs:"
+specs.each {|s| puts "  #{describe(s)}" }
+puts
+
+# ---------------------------------------------------------------------------
+# Part 1: a NEW client (this RubyGems) picks the skinny variant.
+# ---------------------------------------------------------------------------
+chosen = Bundler::MatchPlatform.select_best_platform_match(specs, LOCAL)
+raise "expected exactly one winner, got #{chosen.size}" unless chosen.size == 1
+winner = chosen.first
+
+puts "[new client] select_best_platform_match(#{LOCAL}) chose:"
+puts "  #{describe(winner)}"
+puts "  download name: #{winner.full_name}.gem  (reconstructed from version + sha)"
+puts
+
+unless winner.content_addressable?
+  abort "FAIL: expected the skinny (content-addressable) gem to be chosen"
+end
+
+# ---------------------------------------------------------------------------
+# Part 2: an OLD client drops the skinny rows via the `rubygems:` gate.
+# `matches_current_rubygems?` is exactly the lever the resolver uses to
+# exclude metadata-incompatible candidates. We simulate an old client by
+# checking the gate against a pre-CA RubyGems version.
+# ---------------------------------------------------------------------------
+old_rubygems = Gem::Version.new("3.5.0")
+skinny = specs.find(&:content_addressable?)
+gate = skinny.required_rubygems_version
+old_client_keeps_skinny = gate.satisfied_by?(old_rubygems)
+new_client_keeps_skinny = skinny.matches_current_rubygems?
+
+puts "[gate] skinny row declares rubygems:#{gate}"
+puts "  old client (RubyGems #{old_rubygems}): keeps skinny? #{old_client_keeps_skinny}  -> ignores it"
+puts "  new client (RubyGems #{Gem::VERSION}): keeps skinny? #{new_client_keeps_skinny}  -> processes it"
+puts
+
+if old_client_keeps_skinny
+  abort "FAIL: an old client should NOT satisfy the rubygems gate"
+end
+unless new_client_keeps_skinny
+  abort "FAIL: this (new) client should satisfy the rubygems gate"
+end
+
+# What an OLD client would actually resolve: drop gated-out rows first, then
+# run the same platform selection. It must fall back to the fat binary.
+old_visible = specs.reject {|s| !s.required_rubygems_version.satisfied_by?(old_rubygems) }
+old_winner = Bundler::MatchPlatform.select_best_platform_match(old_visible, LOCAL).first
+puts "[old client] after dropping gated rows, select_best_platform_match chose:"
+puts "  #{describe(old_winner)}"
+if old_winner.content_addressable?
+  abort "FAIL: old client should fall back to the fat binary, not the skinny one"
+end
+puts
+
+puts "ALL GOOD: new clients choose the skinny gem; old clients fall back to fat."