[Bundler] RFC Proposal: “include gem as” feature

[byroot]

Context

A problem that has always existed forever, but is of course becoming bigger as time passes, is abandoned gems. You may depend on a gem, and one day when trying to upgrade Ruby or another gem realize it now has a very simple compatibility issue, but the original maintainer is no longer active, hence the gem will probably never receive an update.

If the gem is a “leaf”, meaning no other gems depend on it, it’s easy to just fork it and publish it under another name. However if other gems declare a dependency on it, the solutions are much more limited. You can reference a git fork and include the gem that way, which works but tends to be a lone solution rather than a community one. It’s rare, and rather not recommended to point your Gemfile at someone else’ fork of a repository.

Another possibility is to try to reclaim that gem name to become the new maintainer, but it’s slow and rarely works.

Proposal

I believe this community problem could largely be eased if it was possible to declare that a gem is standing up for another one in your Gemfile.

gem "console-formatter" # has a dependency on "left-pad"
gem "left-padder", as: "left-pad"

In the above example we assume that left-pad, a very useful and popular gem that is pulled as a dependency by dozens of bigger gems, is no longer working with a future version of Ruby, and the maintainer is no longer active.

I, as another open source contributor, can fork that gem, give it another name (left-padder) and then instruct users that they can replace left-pad with it, by specifying it in their Gemfile.

If the situation persists and the left-pad maintainer remains inactive, the various gems that depend on left-pad can progressively decide to update their dependency declaration, but in the meantime users have a way out of the problem, and are unblocked.

Additional Use Cases

Such a feature would also allow for secondary use cases.

For instance, maintaining alternative versions of active gems that include features or changes the maintainer doesn’t wish to include upstream.

Another use case is for community maintained versions of EOL releases. If left-pad is currently at version 2, and the version 1 is EOL but a large community of users are still on version 1, they can maintain a left-pad-lts gem if they want.

[indirect]

Some form of this idea has been floating around for a long time, and I think we need more details to know whether this seems worth the effort that would be required. The RFC template document contains a list of the kinds of questions that we would want answered in order to consider accepting an RFC for this feature. Could you add answers to those questions?

The biggest worry that comes immediately to mind for me is "how do we keep a feature like this from harming maintainers?" For example, if I maintain left-pad, but someone else is using left-padder, how do we ensure that errors are reported to left-padder and not to left-pad?

[byroot]

Could you add answers to those questions?

Many don't seem to really apply here, but let's try:

Why are we doing this? What use cases does it support? What is the expected outcome?

I think I mostly covered this already. I want to make it less of a pain for the community when a project stop being maintained, but remain largely used.

Why should we not do this?

I can't think of any reason not to, aside from implementation challenge if it turns out it's really tricky to implement.

Why is this design the best in the space of possible designs?

I don't really see any other solutions to this problem.

What other designs have been considered and what is the rationale for not choosing them?

I haven't considered any other design, because I can't think of any other.

What is the impact of not doing this?

The Ruby community will continue to have to fight whenever a popular, but no longer maintained gem stop working. I could give examples, but I really don't want to look like I'm pointing fingers. It's totally fine for a maintainer to just move on, they don't owe their users anything.

What parts of the design do you expect to resolve through the RFC process before this gets merged?

I think the design is quite simple, so I don't really see much needed discussion on it. It's more about wanting or not wanting this feature, and how to implement it.

The biggest worry that comes immediately to mind for me is "how do we keep a feature like this from harming maintainers?" For example, if I maintain left-pad, but someone else is using left-padder, how do we ensure that errors are reported to left-padder and not to left-pad?

It would probably happen once in a while, but for the most part:

• The as: can only be done in the Gemfile, so supposedly the reporter knows they replaced the gem. It can't happen silently via pulling a 3rd gem.
• Assuming the report starts with a backtrace, the path will contain left-padder-xxx, not left-pad.

[byroot]

the various gems that depend on left-pad can progressively decide to update their dependency declaration, but in the meantime users have a way out of the problem, and are unblocked.

So one thing I overlooked a bit here, is that while this process is going on, some users may have both left-pad and left-padder pulled as transitive dependencies, so they'll likely run into a name clash until they explictly add gem "left-padded", as: "left-pad" in their Gemfile.

So this part of the workflow isn't exactly seamless, but I can't really think of a solution here. Perhaps the gemspec could include a list of "conflicting" gems like .deb has, so that bundler can give a clearer error. But adding to gemspec is I believe much more involved than "just" adding an option to bundler.

[indirect]

Maybe I am not as optimistic as you, but when you say "while the process is going on", it sounds to me like you are describing how things will end up permanently in the end. I imagine most applications will eventually have 2 gems that depend on left-pad, 3 gems that depend on left-padder, and 1 gem that depends on left-pad-lts.

That means you, the application developer, are personally responsible for troubleshooting the unexpected interactions between those 9 gems, and the way that all three left-pad gems work differently. The only escape I know of is for the application developer to fork those gems and update them to all use the same dependency. And that is already the current situation, without this feature existing.

Since it seems impossible to know right now if I am right, what if we tried it as an experiment? For example, we could add plugin hooks to Bundler to allow this feature to be offered as a plugin, or we could ship this feature behind a setting named BUNDLER_UNSTABLE_ALPHA_GEM_REPLACEMENT_MAY_DISAPPEAR_ANY_TIME or something like that.

Hopefully having some users actually try it out would give us enough information to know if we should release the feature widely or if changes need to be made for that to be a good idea.

[simi]

@byroot the left-pad allusion is clear, but would you mind to share any real problem this is going to solve? From history, I do remember mostly only mail gem Ruby 3.1 support delay (if I remember well, it was fixing just warnings).

My opinion is negative on this feature, since IMHO we should prefer the dependency update and release as a preferred way to fix the problem, which works well in RubyGems.org ecosystem and keeps it "healthy" without dramatic fork rate. And if dependency you rely on is not maintained, clearly you should not use it.

Feature like this must be the last chance to solve the problem, not the easiest one. If introduced, I would prefer to make it not easy to enable and leave it annoying when kept as permanent solution. Strangely long ENV variable + huge warning on each bundle operation could be the good starting point.

Btw. similar discussions already happened few times at ruby/rubygems#1746, rubygems/bundler#1549, rubygems/bundler-features#20 and rubygems/bundler#4552.

[byroot]

The only escape I know of is for the application developer to fork those gems and update them to all use the same dependency.

Well, no because gem as: allow to solve this without forking.

what if we tried it as an experiment?

If the feature was available as a plugin or behind some feature flag I'd happily use it and report back any pain points etc.

would you mind to share any real problem this is going to solve?

Alright, I didn't want to point any fingers, but here a few examples:

• httpclient is broken because it embeds some old CA roots, and is still to this day a dependency of google-apis-core: https://rubygems.org/gems/httpclient/reverse_dependencies. It's somewhat working because now they configure the CA roots, but is also broken on Ruby 3.4 because it requires mutex_m without declaring it in gemspec: WIP - Switch to Faraday googleapis/google-api-ruby-client#11087 (comment)
• unicorn is preventing changes in the MRI C API https://bugs.ruby-lang.org/issues/19057, the patch has been sent upstream but never got released, and I suppose you know the whole situation about Unicorn and MFA. Also see: How to handle reverse dependencies on unicorn? unicorn-ruby/unicorn#1

But more generally, whenever a gem is no longer maintained, it creates a mess, and if it breaks on a newer version of Ruby of another gems, it create problems for the whole community and can be source of stagnation (e.g. Ruby giving up on a positive changes because it break important, but no longer maintained gems).

Also importantly, in the next couple years, Ruby is scheduled to default to frozen string literals, and I fear this will unfortunately uncover a number of abandoned gems, and I'd like for the community to have a way to handle this, and for new maintainers to emerge.

we should prefer the dependency update and release as a preferred way to fix the problem, which works well in RubyGems.org ecosystem and keeps it "healthy" without dramatic fork rate.

Do we have examples of this working properly before? How many time abandoned gems have been handed over to new maintainers by the rubygems team?

If tomorrow morning I get hit by a bus, a large number of important gems will have no maintainer any more, and no-one will be able to update them so they keep working. That's a problem.

[byroot]

Also a simpler implementation of this feature could be to just mark a gem being "de-activated", e.g.:

gem "some-gem-that-depend-on-left-pad"
gem "left-pad", disabled: true
gem "left-padder"

disabled: true would:

• Not install the gem.
• Not add it to load path.

This would essentially provide the same capabilities.

[simi]

... unicorn ...

Yes, I'm aware of this whole situation. Gem still could be released without MFA, but it is not happening. This gem is clearly in strange state. There is current maintainer is even claiming gem is not recommended (I would read deprecated). If there's no way to help this gem maintain, clearly fork is the way to go. Luckily this gem is not part of gem dependencies and should be easy to fork and migrate projects to.

... httpclient ...

It seems there is some activity by maintainer and things are slowly moving forward. AFAIK there is still enough time before Ruby 3.4 release. Indeed would be great to provide PR removing whole bundling CA stuff to make it easily maintainable for the future.

But more generally, whenever a gem is no longer maintained, it creates a mess, and if it breaks on a newer version of Ruby of another gems, it create problems for the whole community and can be source of stagnation (e.g. Ruby giving up on a positive changes because it break important, but no longer maintained gems).

Let's fight this with actions preventing this from happening. If gem is no longer maintained, IMHO you should not use it. If RubyGems/Bundler provides easy way to replace gems with alternative ones, IMHO that would be mess since forks and alternative releases will start to randomly be pushed and used with various level (IMHO usually none) of maintenance.

Do we have examples of this working properly before? How many time abandoned gems have been handed over to new maintainers by the rubygems team?

I don't remember any actually. I mean Ruby ecosystem seems healthy enough on its own (without RubyGems.org team intervention) to prepare majority of used gems before or quickly after Ruby new version release.

If tomorrow morning I get hit by a bus, a large number of important gems will have no maintainer any more, and no-one will be able to update them so they keep working. That's a problem.

That's clearly problem and you should invite more maintainers to those projects. If I remember well, something similar is checked by https://securityscorecards.dev/#the-checks. We can potentially scan for gems in similar situation (reaching some downloads threshold having 1 maintainer) and suggest to owners to invite more people. Currently, in similar situation, we would receive request for adding maintainer to gem which is first validated and somehow resolved by RubyGems.org support team.

[byroot]

Luckily this gem is not part of gem dependencies and should be easy to fork and migrate projects to.

It is, See: https://rubygems.org/gems/unicorn/reverse_dependencies and unicorn-ruby/unicorn#1

It seems there is some activity by maintainer and things are slowly moving forward.

Didn't noticed a few things got merged 3 months ago, but still no release in sight and the issue has been going for years now.

If gem is no longer maintained, IMHO you should not use it

Easier said than done when there is often a whole dependency tree pulling it. For some important gems it means dozens of other gems coordinating etc. It would be much better for everyone if taking over maintenance under another name was simplified.

IMHO that would be mess since forks and alternative releases will start to randomly be pushed and used with various leve

I don't see how it's much different from deciding to depend on a new gem. You look at the maintainer and their track record and decide if you trust them.

I mean Ruby ecosystem seems healthy enough on its own

I'm not saying it's unhealthy. But to give you context, I take care of testing Shopify's monolith against ruby-head and clearing compatibility issue with our 700+ transitive dependencies. This is a very frequent problem for me. I believe you may not notice it, but at any point in time I always have dozens of pull requests open on repos in various state of maintenance. Most eventually get merged, but when it isn't, then it become a mess of instead contributing to reverse dependencies to try to eliminate the package.

That's clearly problem and you should invite more maintainers to those projects.

Competent and trusted maintainers don't exactly grow on trees. But to be very frank with you this answer annoys me a bit, because I point a problem, propose a solution for when it happens, and I'm basically told the problem don't exist, and that if it existed it would be my fault...

To try to come up to the problem from another angle, I, as the developer of an application with a Gemfile, would like the capability to override some gemspecs. It's great that gems automatically pull their dependencies, but I don't see why I'm simply not allowed to override that if I want to do it for whatever reason.

[simi]

But to be very frank with you this answer annoys me a bit, because I point a problem, propose a solution for when it happens, and I'm basically told the problem don't exist, and that if it existed it would be my fault...

I'm sorry for this, it wasn't my intention. I'm maintainer of big project (I mean big Gemfiles) and I do understand this problem exists. I'm just afraid everyone will roll own fork on initial problem and I'm trying to find out balance in between those two approaches.

I'm not saying it's unhealthy. But to give you context, I take care of testing Shopify's monolith against ruby-head and clearing compatibility issue with our 700+ transitive dependencies. This is a very frequent problem for me. I believe you may not notice it, but at any point in time I always have dozens of pull requests open on repos in various state of maintenance. Most eventually get merged, but when it isn't, then it become a mess of instead contributing to reverse dependencies to try to eliminate the package.

Would in this case the proposed solution (keep the feature behind some strange ENV and provide warnings or so) works for you for this usecase? That way it could potentially prevent mis-usage of this feature.

[byroot]

Would in this case the proposed solution (keep the feature behind some strange ENV and provide warnings or so) works for you for this usecase?

It would solve the problem for me yes, I'd like to also solve it for the community, but if you don't want to I'll take what I can get.

[simi]

It is, See: https://rubygems.org/gems/unicorn/reverse_dependencies and unicorn-ruby/unicorn#1

It seems there are 2-3 main gems which could be forked next to unicorn. 🙏

[simi]

It would solve the problem for me yes, I'd like to also solve it for the community, but if you don't want to I'll take what I can get.

Clearly we all would like to solve this problem for everyone, but we need to find out how. IMHO this could be good starting point. By I'd like to also solve it for the community, is your intention to maintain forks (but keeping the name) so everyone can use them using "overriding feature"?

[byroot]

is your intention to maintain forks (but keeping the name) so everyone can use them using "overriding feature"?

Yes, and also to allow others to do the same.

Basically the capability I'm asking for already exist via the gem git: feature, and it's not rare other companies are running Shopify forks of gems to solve this class of problems. But as you know this isn't great because:

• I could break people apps if I force push my branch and the commit in their Gemfile.lock is garbage collected.
• git gems are slower to install and take more disk space (annoying in docker).
• I generally don't think it's good practice to pull someone else repo like that, for security reasons (even though very practically speaking it isn't that different).

But more importantly, git gems don't open the door to reverse dependencies migrating to the new maintained fork.