Automated testing of examples in the documentation

[richtja]

Before the LTS release, we spent a lot of time testing and fixing the documentation examples to make them compatible with changes in the avocado code. Because of that, we should create automated tests for checking the compatibility between documentation and code. This issue should bring a discussion about what will be the best solution to fo this problem. Here are some possible solutions:

• Use sphinx with doctest. This does exactly what we want but it can work just with python interpreter and for the command-line examples we have to use subprocess module.

• For command-line examples uses some doctest extensions like doctest-cli from people who were dealing with the same problem. But here we cannot ensure reliability of such tools.

• We can create interaction tests for command-line examples and then these tests import into the documentation by sphinx.

These are ideas about how we can deal with this problem, but I am not familiar with documentation testing. So any ideas are welcome.

[willianrampazzo]

I was thinking when this problem appeared, before the LTS release, can't we import the tests from examples/ dir? Then, we test only once, and the user will have the test ready to run when they want.

If this is possible, but we find that many examples are not in the directory right now, we can create something like examples/docs and reference that as the examples from our documentation.

Just throwing the idea here :)

[richtja]

I was thinking when this problem appeared, before the LTS release, can't we import the tests from examples/ dir? Then, we test only once, and the user will have the test ready to run when they want.

@willianrampazzo Thanks for the reply, because this is exactly what I mean by the third option; maybe I didn't express myself clearly.

Here I can see a problem that we have to import all those examples into rst documentation, and I think that won't be easy to maintain in the future. But maybe I am wrong.

[willianrampazzo]

I was thinking when this problem appeared, before the LTS release, can't we import the tests from examples/ dir? Then, we test only once, and the user will have the test ready to run when they want.

@willianrampazzo Thanks for the reply, because this is exactly what I mean by the third option; maybe I didn't express myself clearly.

Okay, got it!

Here I can see a problem that we have to import all those examples into rst documentation, and I think that won't be easy to maintain in the future. But maybe I am wrong.

I think keeping the tests in one place and test them will make things easy for the users and us whether this is on the examples directory or into the documentation.

The idea to use the tests from the examples directory is to have the documentation examples ready to be executed and avoid the copy/paste from the documentation to a test file and then running on the user side.

This is the approach I would go for, without deeply evaluating it, so, as you mentioned, there may be drawbacks in the maintenance process.

[richtja]

Ok, we both agree on this. I will start working on it if @clebergnu or @beraldoleal don't have any comments on this.