Add content addressable gems support - #9773
Merged
Merged
Conversation
jenshenny
force-pushed
the
feature-branch-ca-changes-rubygems
branch
2 times, most recently
from
August 10, 2026 18:27
4a5def3 to
d849a56
Compare
OughtPuts
force-pushed
the
feature-branch-ca-changes-rubygems
branch
3 times, most recently
from
August 24, 2026 15:53
ce7e319 to
8d050c3
Compare
tenderlove
reviewed
Aug 24, 2026
tenderlove
left a comment
tenderlove
left a comment
Member
There was a problem hiding this comment.
I didn't review the specs super closely, I trust they are covering useful scenarios.
It looks like we've got a lot of array / hash manipulation going on in this PR. Should we be thinking about making real, named objects?
The direction looks good here IMO
|
|
||
| def hash | ||
| @set.hash ^ @name.hash ^ @version.hash ^ @platform.hash | ||
| @set.hash ^ @name.hash ^ @version.hash ^ @platform.hash ^ @content_address.hash |
Member
There was a problem hiding this comment.
No impact on this PR, but this is a bad hash and we should fix it upstream.
We should be doing [@set, @name, ...].hash
|
|
||
| def hash # :nodoc: | ||
| name.hash ^ version.hash | ||
| [name, version, platform, content_address].hash |
| end | ||
|
|
||
| def spec_platforms(entry, platforms) | ||
| platforms = platforms.transform_values(&:uniq) |
Member
There was a problem hiding this comment.
This is because it's now a hash of hashes? What is the structure of platforms?
Contributor
There was a problem hiding this comment.
platforms comes from line 213 in output_versions. It's a one-level hash
platforms = Hash.new {|h,version| h[version] = [] }
In reality it would look something like this:
{
Gem::Version.new("1.0.0") => [
Gem::Platform::RUBY,
Gem::Platform.new("x86_64-linux"),
Gem::Platform.new("x86_64-linux")
],
Gem::Version.new("0.9.0") => [
Gem::Platform::RUBY
]
}
And then `transform_values(&:uniq) would remove any dups in the values.
|
|
||
| Gem::NameTuple.new(name, version, platform || "ruby") | ||
| suffix ||= "ruby" | ||
| content_address = suffix if Gem::ContentAddress.match?(suffix) |
Member
There was a problem hiding this comment.
This logic is because we're using this same loop with the CA and non-CA RubyGems endpoints?
Contributor
There was a problem hiding this comment.
Yep. The compact-index /versions response uses the same suffix field for both formats, legacy entries contain a platform, while CA entries contain a content-address token. This handles both, so it detects CA suffixes and stores them as content_address. The actual platform is decoded later from the gem’s /info metadata!
| platform: platform, | ||
| ruby_abi: ruby_abi_from(requirements[:ruby]), | ||
| } | ||
| end |
Member
There was a problem hiding this comment.
It feels like we should make a real object here. Just spitballing but like:
class GemInfo < Struct.new(:version, :suffix, :platform, :ruby_abi)
def hash; suffix; end
def eql?(other); other.version == version && other.suffix == suffix; end
endThough now that I type this out, it seems very similar to NameTuple? It feels like we could be doing more simple code here with set intersections. e.g. wanted_rows.map { make_obj(_1) } & compact_index_info_rows(name).map { make_object(_1) }
Contributor
There was a problem hiding this comment.
@tenderlove have implemented a solution in this commit, interested in your thoughts!
This was referenced Aug 27, 2026
jenshenny
force-pushed
the
feature-branch-ca-changes-rubygems
branch
8 times, most recently
from
August 30, 2026 22:13
2b193ae to
467a7a5
Compare
Contributor
Author
TestingWith: https://rubygems.org/gems/content_addressable_test 1.1
|
jenshenny
marked this pull request as ready for review
August 30, 2026 22:15
jenshenny
commented
Aug 30, 2026
jenshenny
force-pushed
the
feature-branch-ca-changes-rubygems
branch
4 times, most recently
from
August 30, 2026 23:28
6dc3207 to
f854b32
Compare
jenshenny
added a commit
to rubygems/rubygems.org
that referenced
this pull request
Aug 31, 2026
Replace Object#present? (ActiveSupport) with a plain truthy check on content_address in CompactIndex::GemVersionMethods, so the vendored lib/compact_index* files no longer rely on Rails being loaded. content_address is either nil or a non-empty hex string (the Version model's CONTENT_ADDRESS_FORMAT validation rejects empty/other values, allow_nil: true), so a truthy check is equivalent to present? here. Ref: ruby/rubygems#9773 (comment)
Contributor
Author
Added a follow up issue to explore refactoring away the hashes and arrays #9834 - but it's not a blocker for this initial support as we discussed privately. cc: @tenderlove |
jenshenny
force-pushed
the
feature-branch-ca-changes-rubygems
branch
3 times, most recently
from
August 31, 2026 23:16
27472f4 to
8b25dd2
Compare
Co-authored-by: Harriet Oughton <harriet.oughton@shopify.com> Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
Co-authored-by: Gira Chawda <gira.chawda@shopify.com> Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
…al cache Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
Gem::Indexer (rubygems-generate_index) loads the V1-only CompactIndex constants, either from the compact_index gem or from its own embedded copy, so the artifice's copy of rubygems.org's V2-only implementation can never share that name safely. Check in a copy of rubygems.org's lib/compact_index, renamed to the VendoredCompactIndex namespace, under spec/support/vendor/compact_index, and load it from the artifice with a plain require. Checking the copy in (rather than downloading it during the test run) keeps the suite hermetic and offline, makes the namespace rewrite visible in review, and lets parallel workers and CI runners that skip the test-deps setup load it without falling back to the incompatible gem. Refresh the copy with `rake vendor:compact_index`; `rake vendor:compact_index_check` fails if the checked-in copy drifts from the pinned upstream ref.
A content-addressable gem was locked inline as name (version-address)
platform. Bundler 4.0 parses the token after the version as the platform,
gets unknown, materializes name-version-unknown, and fails: it then either
re-resolves and rewrites the lockfile or fails outright under frozen.
Lockfiles are committed and read by many Bundler versions during
co-publication, so the inline form breaks the exact clients the transition
is supposed to protect.
The spec line is now an ordinary platform pin, and the content address moves
to its own section:
GEM
specs:
nokogiri (1.19.4-x86_64-linux)
CONTENT ADDRESSES
nokogiri (1.19.4-x86_64-linux) 86e5e59f sha256=<content-addressable build's sha>
CHECKSUMS
nokogiri (1.19.4-x86_64-linux) sha256=<platform build's sha>
Verified against the released Bundler 4.0.9:
- unknown unindented sections set @parse_method = nil and are skipped
silently (lockfile_parser.rb:137)
- Definition#lockfiles_equal? subtracts unknown sections before comparing
(definition.rb:1196), so a plain 4.0 bundle install does not rewrite the
lockfile and the section survives; it is only dropped on a genuine 4.0
re-lock, and restored on the next 4.1 re-lock
- 4.0 classifies a locked spec with a missing or empty CHECKSUMS entry as a
lockfile change (definition.rb:619-620): a plain install re-resolves and
frozen mode hard-fails. The platform lock name line must therefore carry a
real checksum, and it must be the platform build's, since 4.0 attributes
it to the artifact it installs for that lock name
That last point drives the checksum rules:
- the checksum store is keyed by full name instead of lock name, because a
content-addressable build and the platform build of the same name,
version, and platform share a lock name while being different files.
Store#register accepts anything responding to full_name and lock_name, so
the parser registers CHECKSUMS entries under the name tuple parsed from
the line rather than the (possibly content-addressed) spec object
- the content-addressable build's checksum is serialized next to its
address in CONTENT ADDRESSES, the only place older Bundler never parses,
and verifies the downloaded gem via the existing install-time registration
- CHECKSUMS carries the platform build's checksum, which the compact index
supplies for every row the fetcher sees, so it is captured during
resolution without downloading the platform gem
- when no platform build exists (skinny-only publication), the CHECKSUMS
line is omitted entirely rather than written bare, so every CHECKSUMS
line describes an artifact installable by its lock name
The section registers in SECTIONS_BY_VERSION_INTRODUCED under 4.1.0, and
its line format requires a platform, since content addressing only applies
to platformed gems.
…ute every eligibility, naming, lockfile, plugin directory, and spec construction decision through its shared predicates
Assisted-By: devx/9c2464e2-74cd-4d36-a6c0-50aef8c2ad01
Assisted-By: devx/9c2464e2-74cd-4d36-a6c0-50aef8c2ad01
Assisted-By: devx/9c2464e2-74cd-4d36-a6c0-50aef8c2ad01
jenshenny
force-pushed
the
feature-branch-ca-changes-rubygems
branch
from
September 5, 2026 23:16
3ffc2fb to
f2c0576
Compare
Contributor
Author
|
Another round of testing after changes 1. Query commands:
|
Contributor
Author
|
@hsbt I tested this branch throughly with the new changes. The suffix widening is still WIP so it would be great to get this merged and we'll create a follow up PR when that's ready. I also added fixes for gem doctor and bundle clean wrt the new abi specification directories and adjusted the platform field in the compact index spec since the existing syntax had an |
hsbt
approved these changes
Sep 6, 2026
hsbt
left a comment
hsbt
left a comment
Member
There was a problem hiding this comment.
Thank you for working through the design and for implementing both the server and the client side 🙏
Merging this now, with time to stabilize it before the final release.
Member
|
@jenshenny @OughtPuts Sorry for my late review. I think that
Users need to build If users just use |
Member
|
Related: #9863 |
Contributor
Author
|
@kou yes, the |
Member
|
Thanks for confirming my questions. Why do you choose this design? I feel that a tool can change |
Contributor
Author
|
I believe it was @hsbt 's suggestion as someone can compile the extension themselves and use btw @OughtPuts and @Edouard-chin are working on adding support to the available tooling right now and mentioned they possibly need to make some additional changes to the build command. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#9654
TL;DR
Adds RubyGems and Bundler client support for content-addressable ("skinny") binary gems: one artifact per Ruby ABI, named with a SHA-256 prefix.
The branch covers build, discovery, install, display, yank, lockfiles, caching, and
bundle install --local. Existing source and platform gems are unchanged.Why
"Fat" binary gems contain every supported Ruby ABI and keep growing. Skinny binaries are smaller, but builds for the same gem, version, and platform need distinct filenames. A content-derived suffix gives each artifact a unique identity.
A content address must be 8–64 lowercase hexadecimal characters. RubyGems only treats it as one when the gem also has a non-Ruby platform and constrained
required_ruby_version, avoiding false matches with ordinary filenames.Follow ups
gem buildref: Shopify/rubygems#171.
User behaviour
gem build nokogiri.gemspec --ruby-abi 3.4builds a skinny gem namednokogiri-1.18.9-78be552b.gem, where the suffix is the SHA-256 digest of the gem contents.--ruby-abi, behaviour is unchanged.Details
--ruby-abivalidates the ABI format (X.Y), requires a non-Ruby platform to be set, and constrainsrequired_ruby_version: if unset it defaults to~> X.Y.0; a mismatched existing requirement is rejected.gem installref: Shopify/rubygems#172 (local) and #173 (remote).
Local
gem install --local GEMNAMEis content-addressable aware.Every install of a CA gem writes a gemspec stub whose
# stub:suffix is the hash (so thename-version-<sha>directory resolves). The real platform rides on a separate# stub-target:line that older RubyGems ignore — backwards compatible, while current RubyGems recover both. File:specifications/mygem-1.0-78be552b.gemspec:Remote
platform:=metadata separately. Distinct hashes remain distinct candidates.For example, a server
info/nokogiriresponse with two skinny variants (different Ruby ABIs) plus a platform fallback:The hash is carried in the version token (
1.18.9-78be552b); the real platform and Ruby requirement travel in theplatform:=/ruby:metadata. The two hashes stay distinct resolver candidates, and1.18.9-x86_64-linuxis the platform fallback.gem pushref: Shopify/rubygems#174.
User behaviour
gem push name-*.gem --platform x86_64-linux --ruby-abi 3.4reads the specs of the SHA-named files and pushes the single matching artifact.Details
--platformand--ruby-abiselectors. Given multiple SHA-named files, RubyGems reads each specification and selects the one whose platform andrequired_ruby_versionsatisfy both selectors.ruby_matches?does not check platform (a RUBY-platform gem with a matching~> X.Y.0can be selected by--ruby-abi); this is documented in the tests rather than special-cased.gem yankref: Shopify/rubygems#176.
User behaviour
gem yank mygem -v 1.0.0 --platform x86_64-linux --ruby-abi 3.4sends gem name, version, platform, and ABI so the server can select one skinny variant.--ruby-abi.Remote queries (
gem list/search/info -r)ref: Shopify/rubygems#175.
User behaviour
gem dependencyref: Shopify/rubygems#189.
User behaviour
gem dependency GEM --remotelists content-addressable gems with their content-addressed full name plus the real platform and Ruby ABI, instead of treating the hash as the platform:Details
fetch_remote_specsruns the detected tuples throughGem::SpecFetcher#decode_content_addressable_tuples, which splits the hash from the real platform via the source, fetches each spec, and carriescontent_addressonto it (dependency_command.rb:69-73).content_address_annotationappends(Platform: <plat>, Ruby ABI: <abi>)only whenGem::ContentAddress.content_addressed?(spec)is true (dependency_command.rb:155-165).bundle install(lockfile + local cache)ref: Shopify/rubygems#177 (remote) and #178 (lockfile + local cache).
Remote
Lockfile and local cache
Gemfile.lockand parses both on the next run.vendor/cachewithbundle install --local. Remote and local paths produce the same installed directory; checksums remain keyed by the platform lock name. A CA gem locks with the hash in the version and the real platform beside it.vendor/cache