[Docs] Run code snippets as integration tests #1189
Description
Some recent issues with docs: #1085 #1179 #1183 demonstrate that we have not tested our documentation examples against the latest Burrow release. It looks like some of this is code changing under the docs, but also the docs being changed without being tested. e.g. add-validator.md uses --pool
and --separate-genesid-doc
which looks like would never have worked.
We should find a way to get test coverage of some or all of our code snippets. The best way I can think of is to introduce a markdown pre-processor with support for file inclusion, such as: https://github.com/mmarkdown/mmark and to then ensure we have integration tests that touch the code snippets.
Some of our code snippets involve, shell, deploy scripts, JS, etc so this is not trivial. There is also a trade-of here between integrity of docs and the ease with which people can contribute. If fixing or contributing to docs involves understanding our test harnesses and potentially needing to fix any errors then the end result may be detrimental to the quality of the docs overall. On one extreme we could try and capture the full process of all tutorials via an automated script - which adds maintenance burden if we ever want to change that. More minimally we could just try and make sure we touch all config and deploy files. Hopefully we can take an incremental approach.
I suggest we attempt to get includes working, and apply it to a single doc item, e.g. #1085 to see how we can do this with minimal interruption
This could also fix #1187
#integration