Random build failures in CI

[skalee]

Tests do fail from time to time in CI due to some "bad file descriptor" error (Errno::EBADF). For example:

• https://travis-ci.org/riboseinc/enmail/jobs/460756775
• https://travis-ci.org/riboseinc/enmail/jobs/477826746

It is unclear whether this issue is related to current project, or to rspec-pgp_matchers.

[skalee]

Handling temporary files in riboseinc/rspec-pgp_matchers has been improved, and it helped a lot. However, the problem is not fully resolved — this failures still happen, though very rarely. See this comment: riboseinc/rspec-pgp_matchers#31 (comment).

[skalee]

I become more and more convinced that these issues are caused by some race condition in Ruby's Open3#popen3. This method uses UNIX pipes for capturing standard input and output, and I have a strong suspicion that file descriptors returned from IO.pipe method are not guaranteed to be ready for immediate use.

[skalee]

After a lot of thinking I came to conclusion that the current CI setup does not suit us well. I am going to replace it with a Docker image.

1. Firstly, these race condition issues happen only in Travis CI. I cannot reproduce them elsewhere, and they are almost certainly related to this particular environment. Therefore, running builds in Docker container should help. Also, backtrace inspection clearly indicates that the problem is not a bug in EnMail, but is rather specific to our custom RSpec matchers. So it's not like kicking a can into a long grass.
2. Secondly, Travis build environment is very difficult to replicate on local computer, which makes build debugging quite difficult. A big advantage of Docker is that images are painless to host on any computer.
3. Thirdly, although proof-of-concept builds took up to twenty minutes each, that is twice longer than on current master, I expect that this change will eventually result with a substantial speed-up, perhaps by an order of magnitude. This can be achieved by caching prebuilt images (e.g. pushing them to Docker Hub), or by using more recent Linux distributions, which provide more recent versions of required software, so we don't need to build that ourselv
4. Travis CI environment is capable of running Docker containers. However, we are not bound to that, and can easily move to another provider if we want to.
5. Finally, I am hoping for easier re-using this Docker images across various Ribose projects.

[skalee]

Meanwhile I have tried with OS X in Travis. I was hoping for an easy fix for this issue as well as for a substantial speed-up (GnuPG and all RNP's dependencies can be installed with Homebrew in no time). Unfortunately, the same file descriptor failure has occurred: https://travis-ci.org/riboseinc/enmail/jobs/545273825.

[ronaldtse]

@skalee any updates in this area? Thanks!