Autogenerating Documentation

[mfripp]

I would really like to have documentation for each module with a number of features:

• compact but comprehensive list of components defined in that module
• descriptions of the components where needed
• concise representation of the rules used to define each component

We have a lot of this in the Supplemental Information file for the Switch 2.0 paper. But ideally these elements would be automatically extracted from the source code, to make sure we cover everything. In the near term, this would be helpful for cross-checking that everything is covered in the Supplemental Information file, and possibly to add some extra detail there (cross-reference Python names for the Latex terms, list the tables that parameters are defined in, etc.) In the longer term, this could help us create web-based documentation that uses Python terms rather than Latex, is more readable than our current source code, and allows cross-referencing terms between modules.

I've been playing around this week to see what might be possible along these lines. I'm pretty sure now that we could do this by inserting our detailed comments throughout the main code in each module, using docstring format (triple-quotes). This text would be similar (often identical) to the comments currently written at the top of each module, but would be dispersed throughout the module instead. Once that is done, I'm pretty sure we (I) could automatically generate documentation pages for each module by following these steps:

• read the module using AST, then scan through all the standard callback functions in the module and...
• add docstrings directly to a reStructuredText (rst) file (in order)
• convert Constraint, Expression, Set, Param, Param.default and Var.bound rules into easier-to-read Python-style expressions and insert them into the rst file (in sequence with the docstrings)
• insert function definitions into the rst file (in sequence with the docstrings)
• move component definition expressions just above the rule functions they rely on and add "where Constraint_rule is given by:" to improve readability
• accumulate lists of Params, Expressions, Vars and Sets and relevant descriptors (indexing set, domain, has default value, etc.)
• accumulate extra information on Params from the load_aug calls in the load_inputs function (filename where params come from)
• write summary tables of all components defined in the module at the top of the rst file.

Once this is done, it's fairly easy to convert the rst file into HTML, Tex, PDF, etc.

At a later stage, we may be able to use standard translations to convert the Python component names (and eventually maybe even the sum()-type expressions) from this rst file into equivalent Tex terms (i.e., shorter variable names), and write additional Tex-oriented rst files. Then it might be pretty quick work to tweak that to good Tex code and/or we could use a system to retain any translated code that has been manually tweaked, until the corresponding Python code changes (this would probably be something for later in the year).

To help you see how this could work, I have attached a zip file commit_autodoc.zip (but see new version in comment below) containing three example files:

• a Python module (generators.core.commit.operate) which has had about 2/3 of the comments interleaved in the body
• an rst file which I created manually from this, but which I think I could produce automatically after a day or two of work
• an html file generated from the rst file (needs a better stylesheet, but you get the idea)

So my questions are:

1. does this sound like a good idea?
   • pro: unified management of code and documentation, "literate programming", less brain requirement to link the documentation (formerly at top of file) to code
   • con: code is a little more cluttered (but this may be made up by having easier to read documentation)
2. Rodrigo has done amazing work to prepare the Tex documentation and Josiah did amazing work to write documentation at the top of all the module files. This would potentially unify those two threads of work. Is that helpful?

[rodrigomha]

Looks amazing in my opinion. Probably this will be really helpful for new users to have detailed information on every module. However, one important aspect for new users is the order of the modules are presented. That's the reason in the supplementary document I started with timescales, financials and so on. Maybe we can have a README or a manual where the documentation is located explaining how to start reading the files, that would recommend users to start with timescales, financials, etc.

Another point, is that some parameters/others may not be defined on the supplementary document, since those parameters are only used to construct sets or subsets, and in the tex file I only define that set. For a mathematical model is not really important how that set is constructed using code, and you just assume that your set is already well defined.

[mfripp]

Here's a newer version of the demo files, with some internal hyperlinks from the component tables to the definitions: commit_autodoc.zip

[josiahjohnston]

I like this goal. Our last discussion on these topics were in issue #76 and pull request #57. I'll have to get back with you on more specific comments.

[josiahjohnston]

This is a great prototype.

• It might be preferable to embed the component-specific documentation in it's doc attribute during construction. Pro: this attribute is supported by Pyomo and has a few hooks in other places for model inspection. Con: Our documentation may be longer than is convenient for places where that doc is displayed, in which case we may want to divide our documentation into a short description for the doc attribute and a longer description for our auto-generated documentation.
• We may want one or more separate attributes for tex symbols or equations.

Example:

model.foo = Param(doc="An example parameter")
model.foo.long_doc = "Long verbose explanation"
model.foo.tex_symbol = "\alpha"
model.foo.tex_equation = ...

It would be ideal for the code snippets to either be automatically extracted, or linked to the relevant section of source code on github. Manually pasting those snippets into documentation for core sections that change slowly might be workable, but I'd steer clear manually pasting in code for rapidly-changing modules. I generally prefer having less exhaustive documentation than risking out-of-date documentation. Same ideas hold true for the potential tex_equation attribute.

Auto-generating tex equations from parsing pyomo expressions & constraints sounds cool but challenging.