Add integration tests to docker image

[agjohnson]

This would still be a manual process, as I don't know how we would automate the integration tests, but it would at least give maintainers and contributors a quick way of testing changes to the build image.

Tests would include:

• Test cloning repository
• Test sphinx html build and that output looks correct
• Test sphinx pdf build and that output looks correct
• Test sphinx epub build and that output looks correct
• Test conda installation
• Test pip installation
• Test virtualenv?

Any more here?

Refs #48

[humitos]

I think this could be complicated (time consuming at the beginning) but what about having a repository with one branch per scenario (under https://github.com/readthedocs/template for example), including a readthedocs.yaml with Py3 another with Py2, another with python setup.py install another without it and more...

So, then in this repository a tox file that runs all of those scenarios. Finally, the contributor can take a quick look to a couple of index.html and pdf to check that the overall is working and check more granulary the change that he/she just incorporate?

[agjohnson]

Perhaps, though I'd avoid manual review as a requirement. I suppose we can move the output files to a docker shared volume for review afterwards though.

testinfra could be a useful addition here, and has a connector for docker:
https://testinfra.readthedocs.io/en/latest/examples.html#test-docker-images

I'd say we could just include fixtures here and automate testing of the output, no need for a separate repo.

[humitos]

I think this could be complicated (time consuming at the beginning) but what about having a repository with one branch per scenario (under readthedocs/template for example), including a readthedocs.yaml with Py3 another with Py2, another with python setup.py install another without it and more...

This is already done in the https://github.com/readthedocs/test-builds repository and we can use here to make some simple builds on the just created docker images. We can just check the exit code is 0 in all the commands for now.

• Test conda installation
• Test pip installation
• Test virtualenv?

There are already done in #143. We can easily add the rest one once that PR gets merged and we are happy with the results.