Rubygems should not install test files

[anatol]

I see that gemspec has test_files and many gemfiles include tests. I believe it is artifact from gem test times. I think tests in gemfiles are OK (e.g. in case if you decide to reintroduce gem test back #730). But gem install should not install these files, gem tests are useless for users (unless I missed something).

Bottomline: gem install should not install files from test_files.

[drbrain]

This is not something I can change for RubyGems 2.x due to backward compatibility. I have placed this in the Future milestone.

[anatol]

In addition to test files other types of files unneeded at runtime are installed. gems that have native extensions should not install object files, C source files, Makefile. I believe only shared library (*.so) is needed.

[luislavena]

That part depends on the gem author excluding build artifacts from the gems
they build.

Normally pre compiled binaries only ship .so files, and if not, is a bug to
report the gem author.

Sorry for top posting. Sent from mobile.
On Dec 12, 2013 10:05 PM, "Anatol" notifications@github.com wrote:

In addition to test files other types of files unneeded at runtime are
installed. gems that have native extensions should not install object
files, C source files, Makefile. I believe only shared library (*.so) is
needed.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/735#issuecomment-30478268
.

[anatol]

Normally pre compiled binaries only ship .so files

Not sure what do you mean here, but all the examples of gems with native extensions that I see build *.so at install time. Just a few examples: nokogiri, unicorn, rmagic, byebug,... They have ./ext/extconf.rb file that with help of mkmf generates a Makefile. make is called and it builds shared libraries. As far as I see these *.so are copied to ./lib folder where they later loaded into ruby library via 'require'.

Thus you cannot exclude extconf and C sources from gem file. These files are needed at installation time. But these C sources and build artifacts aren't needed after installation. I believe gemspec does not allow to specify such files ("needed in gem but only until installation").

It looks like ./ext folder can be just removed. I tested it with a dozen gems with native extensions on my machine and ruby libraries still work.

[drbrain]

The ext folder cannot be removed at this time for backward compatibility. Some gems may require files out of it through relative path manipulation.

[anatol]

Some gems may require files out of it through relative path manipulation.

Yep, one thing I can think of is a gem that has multiple shared libraries but copies only part of it to lib. Other shared libraries are loaded via absolute path in RPATH. nokogiri compiled without system libraries is one example here.

That is wrong thing IMO - I think all of the shared libraries should be copied from ext/ to lib/ and relative paths should be used. So ext/ can be completely removed. But as you said it is non-compatible change.

[drbrain]

RubyGems 2.2 will copy built extensions to a platform-specific extension directory to support sharing of gems across ruby versions and platforms. RubyGems 3.0 will not copy built extensions to lib.

It is possible RubyGems 3.0 can delete ext/, but that is some time off so I don't want to make a decision now.

[luislavena]

@anatol

Not sure what do you mean here, but all the examples of gems with native extensions that I see build *.so at install time.

You need to see beyond install-time compilation. There are platforms and workflows that will use precompiled gems and not compiled at installation.

As far as I see these *.so are copied to ./lib folder where they later loaded into ruby library via 'require'.

That is mkmf behavior combined with RubyGems.

That is wrong thing IMO - I think all of the shared libraries should be copied from ext/ to lib/ and relative paths should be used.

Talk to gem authors, even better, send patches that correct those issues. Nobody likes a fat gem directory, so I believe any contribution to put that on diet will be very welcome.

As said by @drbrain, these changes cannot be introduced without breaking the entire ecosystem and only can be introduced gradually.

In the meantime, you can send patches to these gem authors 😄

[maxlinc]

Could --no-test (similar to --no-document) be added without breaking backwards compatibility?

Ideally it'd be a flag on both install and build. I agree with @anatol and @luislavena that I don't want a fat gem directory, but ideally I'd also keep gem downloads and unpackaging fast. That would help with the Travis-CI recommendation to speed up your build by excluding non-essential dependencies.

Case in point: yajl-ruby currently produces a 500k gem which expands to 4M. If it excluded test files (spec, examples and benchmark directories) it would be a 50k gem that expands to 528k.

[anatol]

In fact we can go extreme here. At runtime only folders from require_paths are needed. Everything else can be avoided.

Would it be possible to have some kind of --thin installation flag that install only bare minimum needed at runtime?

[maxlinc]

@anatol I like the idea, but it might be too extreme. Two concerns come to mind:

• IANAL - so I'm cautious about skipping the installation of LICENSE files
• Many gems have non-code dependencies that aren't on require_paths, like templates in twbs/bootstrap-sass or resources in hoxworth/json-schema.

[drbrain]

I see nothing that would prevent this from appearing in a 2.x feature release offhand as it merely adds new behavior. Besides the stumbling points mentioned above I also see:

• how many gems fill in the test_files list? This should be populated automatically if possible to make it easier for authors.
• examples are not usually included in a special category so maybe metadata should be used to describe required directories for install instead?
• how easy will it be for authors to be sure a thin or test-only install works (since they typically test with all the files present)

[anatol]

If majority of people need only runtime files then maybe thin installation should be used by default?

And gem spec should contain additional information about runtime files that are not in require_paths (to handle bootstrap-sass example above). Something like

spec.runtime_files = ["templates/**"]

And maybe the same for license files: spec.license_files = 'LICENSE'. (While we are here - it makes sense to make this field mandatory).

Saying this I still think that having tests in *.gem file might be useful to check package at installation time, something like #730.

[drbrain]

A thin installation can't be default in rubygems 2.x