Support EEP 48 Docs chunk

[erszcz]

EEP 48 proposes a documentation format for BEAM languages. I've authored docsh - a library compiling source code documentation to the proposed Docs chunk format. docsh also extends erl with helpers enabling access to this documentation.

Here's a screenshot showing the interop between Elixir (today's 1.7.0-rc.0) and Erlang with docsh (>=0.6.0):

One hypothetic future use case, thanks to the shared format, is using ExDoc with non-Elixir projects.

Now once EEP 48 support is in place I think it's time to expose the project to the broader community - a Hex package is already available. Listing rebar3_docsh in the official Rebar3 documentation would definitely help, too.

It's important to point out that the majority of OTP documentation is not stored in EDoc (the only format docsh can read currently), but in out-of-source OTP SGML. OTP team might change their build system to bundle the docs in EEP 48 format - it would benefit not just Erlang, but Elixir, LFE, Efene, and Alpaca users, too - but they probably have to be convinced that there's enough interest.

What do you think?

cc @josevalim @yurrriq @rvirding @marianoguerra @j14159

[ferd]

Yeah I'm all in favor for this work. Is there anything needed aside from putting rebar3_docsh in the plugins list?

[erszcz]

It's enough for now. Of course, any kind of feedback is welcome, too.

[ferd]

Yeah it'll take a while before I have enough time to personally try it. Will have to see what the artifacts look like and so on. Ideally, after the plugin matures, it would be neat to be able to pull that kind of functionality within rebar3, but we'll have to take the time to take a more mature look.

So far the instructions seem clear, so that's good.

[josevalim]

Hopefully one day the functionality will also be in edoc (and the h helper will be in Erlang). For example, we could ask edoc to generate the chunk for given a module, and we could choose if we want to embed it in the beam file or write it to the doc directory.

[ferd]

Yeah. We have no direct control over that in this project, though.

[michalmuskala]

Would it be reasonable to have this as a default in rebar?

If it's going to be opt-in, nobody's going to opt into it, because there's no direct benefit for the maintainer of the library - the benefit is for the consumers, but they can't do anything about it.

[garazdawi]

OTP team might change their build system to bundle the docs in EEP 48 format - it would benefit not just Erlang, but Elixir, LFE, Efene, and Alpaca users, too - but they probably have to be convinced that there's enough interest.

It is my intention to make this happen for OTP-22. Most likely we'll implement the .chunk part of the EEP.

[lackac]

@garazdawi I would be happy to help :)

[marianoguerra]

hi, I intend to support this EEP in efene, but more interesting to anyone else, I've built a project that parses the erlang documentation and allows to translate it to any format, I have a half backed restructured text generator here: https://github.com/marianoguerra/beamdocs#build

the resulting restructured text is really ugly, I just wanted to see if I could handle all of OTP docs and as of now it works with OTP master, the most interesting part is the xml -> erlang data structures part, after that we can translate it to ExDoc markdown format and use ExDoc for it :)

[marianoguerra]

here's HTML generated using sphinx: http://marianoguerra.org/tmp/html/

(I generated the index with a bash script, now I'm adding support for the book.xml format to improve the index)

[garazdawi]

@lackac We would appreciate any help you can give. The way I imagine it working is to basically take a copy of the xslt rules used to generate the man pages and rework that to output the EEP48 format into a plaintext file and then use a small escript to convert the text to the external term format. Feel free to contact me on irc, slack, email, pigeon post if you want to discuss this further.

[josevalim]

@garazdawi I think we can even store the xml in the chunk because we want to convert the data in the chunk to multiple formats. For example, ansi (or man) in the terminal, HTML for browser docs and so on.

In this case, the erl -man could even lookup and transform on the fly, although I am not sure if this is desirable really.

[marianoguerra]

@garazdawi I'm working on a exml to anything translator, right now it translate to erlang terms, markdown (pretty decent) and restructured text, I can make it translate to anything that's needed.

@josevalim exdoc is markdown right?

here you can see a quick build using mkdocs: http://marianoguerra.org/tmp/site/erts/book/

There are some xml entity scaping issues and encoding issues that I have to look for, but it's evolving pretty fast.

maybe we can store them as erlang terms and do the translations later?

[erszcz]

The EEP-48 item entries stored by docsh are also Markdown (sans Elixir extensions) formatted by a poor man's EDoc converter. My hunch is that converging to a single format for the whole platform would reduce the amount of work necessary for support in different tools.

[josevalim]

ex_doc works with markdown right now but it doesn't care about it, the important is that we can emit HTML at some point. The chunks were designed exactly so we don't have to converge on a single format and I think standardizing on markdown would be a loss generally speaking.

Erlang/OTP itself is in a privileged position because whatever format it picks will be supported by everybody, as you can simply make the tools part of the Erlang/OTP distribution.

[garazdawi]

I created an issue here https://bugs.erlang.org/browse/ERL-708 for further discussion. Feel free to chime in.

Regarding the format, one thing that I think is important is to have the same format for edoc and erl_docgen documentation, so that all Erlang libraries (OTP and non-OTP) follow the same format.

[lackac]

I agree with @josevalim. I have no experience with edoc or the exml files but skimmed through the documentation of the former and looked into some examples of the latter. It looks to me that translating to Markdown will result in some loss of information along the way. We would need to decide what to do with each of the special tags. Some of that would be perfect for the docs metadata sections, but others would make more sense to represent in the text documentation. If we translate them into markdown we loose structural information that might be useful in some contexts.

[OvermindDL1]

RestructuredText (reST) might be a better format 'if' one was to standardize on one. It's very pluggable by design unlike markdown, has no ambiguous bits like markdown, is about as easy as markdown for the parts markdown supports (although it supports a LOT more), has lots and lots of tools for it, and is already well used and been around for a very long time. In addition it can output to about anything in existence including HTML and man pages. It's even the default format used by popular documentation engines like Sphinx.

[KennethL]

The Erlang/OTP documentation is based on our own XML DTDs. There are several types of DTDs to be used depending on if it is a User Guide, an Erlang Module reference manual, a commanline utility manual etc.

The few uses of Edoc markup in OTP is translated to the XML format mentioned above and the same goes for a few Markdown files on top level. When we have everything in XML format we apply XSLT transforms to produce HTML, Man and PDF.

Erl_docgen is the application we use for generation of the documentation it has a backend for edoc which produces the OTP XML format.

So when translating the OTP documentation to nicer html, epub, etc. only XML should be used as input. There might be edoc markup in .erl files but it is only used for a few applications.

We are thinking of introducing other format for writing documentation and there I think Asciidoc seems attractive as an alternative or complement to XML.

When you compare Markdown, RST and Asciidoc with our needs for markup and the ease of writing documentation I think Asciidoc seems the most attractive. Markdown seems to be the most popular but is unfortunately weak when it comes to standardization, markup features etc.

[erszcz]

FYI, I've released docsh 0.7.0 - https://github.com/erszcz/docsh/blob/master/CHANGELOG.md

Alas, the pretty screenshot with erl/iex interop is outdated now (temporarily?), since this version stores unmodified EDoc in docs_v1 chunk. Currently there's only one formatter able to consume this - docsh itself - as EDoc-the-library does not, as of now, provide any means to format its internal XML representation to plain text.

[tank-bohr]

I have some plugins about this

https://hex.pm/packages/rebar3_eep48
https://hex.pm/packages/rebar3_ex_doc

It's kind of PoC. I tried to make the erlang documentation compatible with elixir ex_doc generators

[garazdawi]

The edoc PR for EEP-48 is nearing completion. It includes documentation about how to configure rebar3 to generate chunks by calling rebar3 edoc.

In the long run, do we want to make EEP-48 doc generation part of the compile step so that it will always be available in the rebar3 shell? Or would that impact the compilation speed too much?