Support EEP 48 Docs chunk

[erszcz]

EEP 48 proposes a documentation format for BEAM languages. I've authored docsh - a library compiling source code documentation to the proposed Docs chunk format. docsh also extends erl with helpers enabling access to this documentation.

Here's a screenshot showing the interop between Elixir (today's 1.7.0-rc.0) and Erlang with docsh (>=0.6.0):

One hypothetic future use case, thanks to the shared format, is using ExDoc with non-Elixir projects.

Now once EEP 48 support is in place I think it's time to expose the project to the broader community - a Hex package is already available. Listing rebar3_docsh in the official Rebar3 documentation would definitely help, too.

It's important to point out that the majority of OTP documentation is not stored in EDoc (the only format docsh can read currently), but in out-of-source OTP SGML. OTP team might change their build system to bundle the docs in EEP 48 format - it would benefit not just Erlang, but Elixir, LFE, Efene, and Alpaca users, too - but they probably have to be convinced that there's enough interest.

What do you think?

cc @josevalim @yurrriq @rvirding @marianoguerra @j14159

[ferd]

Yeah I'm all in favor for this work. Is there anything needed aside from putting rebar3_docsh in the plugins list?

[erszcz]

It's enough for now. Of course, any kind of feedback is welcome, too.

[ferd]

Yeah it'll take a while before I have enough time to personally try it. Will have to see what the artifacts look like and so on. Ideally, after the plugin matures, it would be neat to be able to pull that kind of functionality within rebar3, but we'll have to take the time to take a more mature look.

So far the instructions seem clear, so that's good.

[josevalim]

Hopefully one day the functionality will also be in edoc (and the h helper will be in Erlang). For example, we could ask edoc to generate the chunk for given a module, and we could choose if we want to embed it in the beam file or write it to the doc directory.

[ferd]

Yeah. We have no direct control over that in this project, though.

[michalmuskala]

Would it be reasonable to have this as a default in rebar?

If it's going to be opt-in, nobody's going to opt into it, because there's no direct benefit for the maintainer of the library - the benefit is for the consumers, but they can't do anything about it.

[garazdawi]

OTP team might change their build system to bundle the docs in EEP 48 format - it would benefit not just Erlang, but Elixir, LFE, Efene, and Alpaca users, too - but they probably have to be convinced that there's enough interest.

It is my intention to make this happen for OTP-22. Most likely we'll implement the .chunk part of the EEP.

[lackac]

@garazdawi I would be happy to help :)

[marianoguerra]

hi, I intend to support this EEP in efene, but more interesting to anyone else, I've built a project that parses the erlang documentation and allows to translate it to any format, I have a half backed restructured text generator here: https://github.com/marianoguerra/beamdocs#build

the resulting restructured text is really ugly, I just wanted to see if I could handle all of OTP docs and as of now it works with OTP master, the most interesting part is the xml -> erlang data structures part, after that we can translate it to ExDoc markdown format and use ExDoc for it :)

[marianoguerra]

here's HTML generated using sphinx: http://marianoguerra.org/tmp/html/

(I generated the index with a bash script, now I'm adding support for the book.xml format to improve the index)

[garazdawi]

@lackac We would appreciate any help you can give. The way I imagine it working is to basically take a copy of the xslt rules used to generate the man pages and rework that to output the EEP48 format into a plaintext file and then use a small escript to convert the text to the external term format. Feel free to contact me on irc, slack, email, pigeon post if you want to discuss this further.

[josevalim]

@garazdawi I think we can even store the xml in the chunk because we want to convert the data in the chunk to multiple formats. For example, ansi (or man) in the terminal, HTML for browser docs and so on.

In this case, the erl -man could even lookup and transform on the fly, although I am not sure if this is desirable really.

[marianoguerra]

@garazdawi I'm working on a exml to anything translator, right now it translate to erlang terms, markdown (pretty decent) and restructured text, I can make it translate to anything that's needed.

@josevalim exdoc is markdown right?

here you can see a quick build using mkdocs: http://marianoguerra.org/tmp/site/erts/book/

There are some xml entity scaping issues and encoding issues that I have to look for, but it's evolving pretty fast.

maybe we can store them as erlang terms and do the translations later?

[erszcz]

The EEP-48 item entries stored by docsh are also Markdown (sans Elixir extensions) formatted by a poor man's EDoc converter. My hunch is that converging to a single format for the whole platform would reduce the amount of work necessary for support in different tools.