Skip to content

Commit 19cebbb

Browse files
Michael-HenneckeMichael-Hennecke
authored
DAOS-18878 doc: add developer info for DAOS documentation (18285) (#18379)
add developer info for DAOS documentation Signed-off-by: Michael Hennecke <michael.hennecke@hpe.com>
1 parent 5105c7f commit 19cebbb

2 files changed

Lines changed: 357 additions & 1 deletion

File tree

‎docs/dev/documentation.md‎

Lines changed: 354 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,354 @@
1+
# DAOS Documentation
2+
3+
Documentation on DAOS is available in the following places:
4+
5+
* The [https://daos.io/](https://daos.io/) website contains general
6+
information about the DAOS Foundation, a subproject of the
7+
Linux Foundation. This website is maintained by the
8+
DAOS Foundation and uses Wordpress for content management.
9+
10+
* The DAOS Community Wiki [https://wiki.daos.io/](https://wiki.daos.io/)
11+
is a redirection to
12+
[https://daosio.atlassian.net/wiki/spaces/DC](
13+
https://daosio.atlassian.net/wiki/spaces/DC),
14+
an Atlassian Wiki space.
15+
16+
* Documentation for end users and administrators of DAOS is
17+
available at [https://docs.daos.io/](https://docs.daos.io/).
18+
This website provides versioned documentation for each DAOS release.
19+
Its web pages are created from the contents in the
20+
[daos-stack/daos/docs/](https://github.com/daos-stack/daos/tree/master/docs)
21+
trees of the DAOS release branches in Github.
22+
This website also cross-reference the "DAOS Internals" documentation
23+
(in the "Developer Zone" section).
24+
25+
* The `dmg` and `daos` commands have man-pages that are created at
26+
DAOS build time through hidden `manpage` options of these commands.
27+
The contents of these man-pages is embedded in the Go code for
28+
the commands. The man-pages currently only output concatenated
29+
lists of all subcommands and their options.
30+
See
31+
[src/control/cmd/dmg/README.md](https://github.com/daos-stack/daos/blob/master/src/control/cmd/dmg/README.md)
32+
and
33+
[src/control/cmd/daos/README.md](https://github.com/daos-stack/daos/blob/master/src/control/cmd/daos/README.md).
34+
35+
* Most DAOS commands and subcommands have a `--help` option which
36+
prints one-line descriptions of the (sub)command and its options.
37+
The contents of these help texts is embedded in the Go code
38+
(or C code) for the commands.
39+
40+
* Developer-facing "DAOS Internals" documentation of the
41+
DAOS software is maintained as Markdown files within the
42+
[daos-stack/daos/src/](https://github.com/daos-stack/daos/tree/master/src)
43+
source tree of the DAOS Github repository.
44+
Make sure to access the release branch (`master`, `release/2.8`, etc.)
45+
for the version of DAOS you are working with.
46+
47+
* DAOS API documentation is generated from the DAOS source files using
48+
`doxygen`. Running `doxygen` during the website deployment process for
49+
[https://docs.daos.io/](https://docs.daos.io/)
50+
creates HTML pages in `docs/doxygen/html/`,
51+
which are then made available in the "Developer Zone" section of
52+
[https://docs.daos.io/](https://docs.daos.io/).
53+
54+
The following sections describe how the multi-versioned documentation
55+
on [https://docs.daos.io/](https://docs.daos.io/)
56+
is generated from the documentation Markdown files in the Github
57+
`daos-stack/daos/docs/` tree (using the `mkdocs` and `mike` tools,
58+
and the `doxygen` tool for the API documentation).
59+
60+
## Configuration files in Github
61+
62+
### Mkdocs configuration
63+
64+
The [MkDocs](https://www.mkdocs.org/) tool is a static site generator
65+
that creates web pages from Markdown files.
66+
The `mkdocs` command is driven by a single configuration file,
67+
[mkdocs.yml](https://github.com/daos-stack/daos/tree/master/mkdocs.yml)
68+
in the toplevel directory of the DAOS project on github.
69+
Among other project information, this file defines the
70+
menu structure for the navigation bar of the website that is being created.
71+
Each DAOS release branch uses its own version of `mkdocs.yml`,
72+
as some information in that file is version specific or uses URLs
73+
that are pointing to that specific DAOS version.
74+
75+
!!! note
76+
Absolute paths or hardcoded version numbers in the `mkdocs.yml` file
77+
should **not** be changed to relative URLs/paths,
78+
as that will break some of the processing by downstream tools.
79+
These specifics need to be updated once when a new release branch is
80+
created, and again when an existing release branch that was under
81+
development reaches General Availability (GA).
82+
Examples with detailed steps can be found below.
83+
84+
The `mkdocs` tool creates a `sites` subdirectory in the toplevel
85+
directory of the DAOS project on github,
86+
in which the HTML documents for the static website are created.
87+
To prevent this temporary subdirectory to be accidentally committed
88+
into github, the
89+
[.gitignore](https://github.com/daos-stack/daos/blob/master/.gitignore)
90+
file contains a `site/` line.
91+
92+
### Mike configuration
93+
94+
The `mike` plugin is used with
95+
[Materials for MkDocs](https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/)
96+
to generate versioned documentation for each of the DAOS release branches.
97+
Its configuration information is stored in the `versions.json` file
98+
that resides in a new `gh-pages` branch in a checked out version
99+
of the DAOS repository.
100+
Note that this file is **not** committed into the main DAOS-Stack
101+
github repo. It only exists in the local `gh-pages` branch
102+
on the system on which the webpages are generated.
103+
104+
The `versions.json` file should **not** be edited directly.
105+
The `mike` command should be used to define which versions of the
106+
documentation are being created, which aliases should be set up,
107+
and which version should be the default version to display.
108+
See below for details.
109+
110+
### Doxygen configuration
111+
112+
The configuration file for the `doxygen` tool is
113+
[Doxyfile](https://github.com/daos-stack/daos/blob/master/Doxyfile),
114+
located in the toplevel directory of the DAOS project on github.
115+
116+
## Installing the mkdocs Software
117+
118+
On the machine where the DAOS documentation is to be built,
119+
GIT needs to be set up so the `daos-stack/daos` project
120+
can be checked out.
121+
In addition, the MkDocs package and plugins need to be installed.
122+
Running `pip install mike` should install the prerequisite packages.
123+
124+
Depending on the operation system and Python environment,
125+
prerequisite software and the mkdocs plugins may have to be explicitly
126+
installed. For example, on a Windows laptop running Cygwin:
127+
128+
```
129+
pip3 install alabaster sphinxcontrib-applehelp sphinxcontrib-devhelp sphinxcontrib-htmlhelp sphinxcontrib-jsmath sphinxcontrib-qthelp
130+
131+
pip3 install --user mkdocs
132+
pip3 install --user mkdocs-material
133+
pip3 install --user mike
134+
export PATH="$HOME/.local/bin:$PATH"
135+
mike --version
136+
```
137+
138+
## Checking out DAOS from github
139+
140+
To create the DAOS documentation webpages, it is recommended to work
141+
on a clone of the DAOS project on github that is **separate** from any
142+
checked out version that may be used for code development work.
143+
144+
```
145+
cd ~/dev
146+
git clone git@github.com:daos-stack/daos.git daos-website
147+
cd daos-website
148+
git branch # should be on master branch
149+
grep site .gitignore # should show "site/"
150+
grep doxy .gitignore # should show "docs/doxygen/"
151+
mike list # should show nothing on a new setup
152+
```
153+
154+
## Creating the static website contents
155+
156+
To create the static website contents, the following process is used.
157+
For each DAOS release branch that should get included on the website:
158+
159+
* the git release branch is checked out
160+
161+
* running `doxygen` creates a temporary version of the doxygen documentation
162+
for this release in the `docs/doxygen/html` subdirectory.
163+
164+
!!! note
165+
To prevent these files from being accidentally committed into github,
166+
the [.gitignore](https://github.com/daos-stack/daos/blob/master/.gitignore)
167+
file contains a `docs/doxygen/` line.
168+
169+
* `mike deploy` is called with a title, a name for the release,
170+
and optional alias name(s) for this release.
171+
These will be used in the multi-versioned website's navigation system.
172+
173+
- `mike` first uses `mkdocs` to create a temporary copy of the
174+
static webpages for this release in the `site/` subdirectory.
175+
176+
- `mike` will then copy that `site/` contents into the temporary `gh-pages`
177+
branch, with a directory name that matches the release name
178+
specified on the `mike deploy` invocation.
179+
180+
- Any aliases for that release will be created as symlinks in the
181+
`gh-pages` branch, pointing to the directory with the main release name.
182+
183+
This process is repeated for each release. After all versions have been
184+
processed this way, invoking `mike default` will set the version that is
185+
displayed on the website by default.
186+
That command creates a toplevel `index.html` in the `gh-pages` branch which
187+
contains a redirect to the version that was selected as the default.
188+
189+
A complete example for a website where DAOS 2.6 is the current "latest"
190+
version, and the master branch is the "2.7" development branch for what
191+
will eventually become DAOS 2.8:
192+
193+
```
194+
mike delete --all
195+
196+
git checkout master
197+
rm -rf docs/doxygen/html 2>/dev/null
198+
doxygen
199+
mike deploy -t "v2.7 - master" master v2.7 v2.8 2>&1 | tee deploy-log.master.txt
200+
rm -rf docs/doxygen/html
201+
202+
git checkout release/2.6
203+
rm -rf docs/doxygen/html 2>/dev/null
204+
doxygen
205+
mike deploy -t "v2.6 - latest" v2.6 latest 2>&1 | tee deploy-log.v2.6.txt
206+
rm -rf docs/doxygen/html
207+
208+
git checkout release/2.4
209+
rm -rf docs/doxygen/html 2>/dev/null
210+
doxygen
211+
mike deploy -t "v2.4 - deprecated" v2.4 2>&1 | tee deploy-log.v2.4.txt
212+
rm -rf docs/doxygen/html
213+
214+
mike set-default latest
215+
mike list
216+
```
217+
218+
After `release/2.8` has been branched, but before it becomes generally
219+
available (GA), the website version structure should be updated like this:
220+
221+
```
222+
mike delete master # will also delete the v2.7 v2.8 aliases
223+
224+
git checkout master
225+
rm -rf docs/doxygen/html 2>/dev/null
226+
doxygen
227+
mike deploy -t "v2.9 - master" master v2.9 v3.0 2>&1 | tee deploy-log.master.txt
228+
rm -rf docs/doxygen/html
229+
230+
git checkout release/2.8
231+
rm -rf docs/doxygen/html 2>/dev/null
232+
doxygen
233+
mike deploy -t "v2.8 - rc" v2.8 rc 2>&1 | tee deploy-log.v2.8.txt
234+
rm -rf docs/doxygen/html
235+
236+
mike list
237+
```
238+
239+
And finally, when DAOS version 2.8 is released this will be set
240+
as the new `latest` release:
241+
242+
```
243+
mike delete rc
244+
mike delete latest
245+
246+
git checkout release/2.8
247+
rm -rf docs/doxygen/html 2>/dev/null
248+
doxygen
249+
mike deploy -t "v2.8 - latest" v2.8 latest 2>&1 | tee deploy-log.v2.8.txt
250+
rm -rf docs/doxygen/html
251+
mike set-default latest
252+
253+
mike list
254+
```
255+
256+
It is useful to save the logs from the `mike deploy` invocation.
257+
These logs will contain warnings about pages that exist but are not
258+
linked anywhere in the navigation structure, and other issues.
259+
260+
The new website can be tested locally by running `mike serve` on the machine
261+
where the website is being created. This will start mike's built-in webserver.
262+
The new contents can then be inspected by pointing a browser
263+
(running on the same machine) to `http://localhost:8000/`.
264+
265+
## Staging the new website contents to docs.daos.io
266+
267+
Because the version aliases are symlinks, and because the website contents
268+
consists of thousands of small files, copying it to the actual webserver
269+
is best done by creating and transferring a tarfile, which can then be
270+
un-tarred on the webserver and will keep the symlinks intact.
271+
272+
As a best practice, a new website version is first deployed in a
273+
staging area so it can be validated in the same webserver environment
274+
in which the live website is running:
275+
276+
```
277+
export WEBSERVER="docs.daos.io"
278+
export TARFILE="docs-daos-io.tgz"
279+
280+
git checkout gh-pages
281+
282+
tar czvf $TARFILE versions.json index.html v?.? latest master
283+
284+
ssh $WEBSERVER "rm -r docs.daos.io/staging && mkdir docs.daos.io/staging"
285+
scp $TARFILE $WEBSERVER:docs.daos.io/staging
286+
ssh $WEBSERVER "cd docs.daos.io/staging && tar xz $TARFILE && rm $TARFILE"
287+
288+
rm $TARFILE
289+
290+
git checkout master
291+
```
292+
293+
At this point, the new website can be inspected by using the
294+
[https://docs.daos.io/staging/](https://docs.daos.io/staging/) URL.
295+
296+
## Go-live of the new website contents on docs.daos.io
297+
298+
To make the website in the staging area the active website, there's a small
299+
`go-live.sh` script on the webserver that will archive the current live
300+
version and replace it with the one in the staging area:
301+
302+
```
303+
$ cat go-live.sh
304+
D=`date "+%Y-%m-%d"`
305+
mv docs.daos.io old-docs.daos.io.$D
306+
mv old-docs.daos.io.$D/staging docs.daos.io
307+
mkdir docs.daos.io/staging
308+
```
309+
310+
## Miscellaneous tips and tricks
311+
312+
### Beware of relative URLs in links
313+
314+
The `mkdocs` tool creates a *directory* (with an `index.html` in it)
315+
for each Markdown *file* that it processes.
316+
So a documentation file like `docs/user/container.md`
317+
in the github master branch will become this URL on the website:
318+
[https://docs.daos.io/master/user/container/index.html](https://docs.daos.io/master/user/container/index.html).
319+
320+
This additional directory level may break relative URLs in
321+
hyperlinks in the Markdown files. On the other hand, adding that
322+
additional directory level to those relative URLs will break the
323+
links when looking at the (un-processed) Markdown files directly
324+
in github.
325+
326+
There is no good fix. In some places absolute paths to website URLs
327+
are used to make sure those hyperlinks work in both the raw
328+
Markdown and the rendered HTML pages on the website. The drawback
329+
of this approach is that those URLs will contain hardcoded
330+
release numbers, but those can be quickly updated by a
331+
search and replace when a new release branch is created.
332+
333+
### Resolving Spell Checker Issues
334+
335+
The DAOS CI performs spell checking on PRs.
336+
If the spell checker reports an error for a legitimate word
337+
(or acronym), the error can be resolved by adding the term to the
338+
wordlist that is maintained in the `utils/cq/words.dict` file.
339+
340+
### Always call mike on a release branch, not gh-pages
341+
342+
You must be in an actual DAOS release branch,
343+
not in the `gh-pages` branch, to run mike commands.
344+
Otherwise the tool will report errors like this:
345+
346+
```
347+
$ git branch
348+
* gh-pages
349+
master
350+
351+
$ mike list
352+
error: [Errno 2] No such file or directory: 'mkdocs.yml'; pass --config-file or set --remote/--branch explicitly
353+
```
354+

‎mkdocs.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@ repo_name: daos-stack/daos
88
repo_url: https://github.com/daos-stack/daos
99
# edit_uri: blob/release/2.6/docs/
1010
edit_uri: blob/master/docs/
11-
copyright: Copyright 2016-2024 Intel Corporation
11+
# yamllint disable-line rule:line-length
12+
copyright: Copyright 2025-2026 Hewlett Packard Enterprise Development LP. Copyright 2016-2024 Intel Corporation.
1213

1314
# Theme
1415
theme:
@@ -107,6 +108,7 @@ nav:
107108
- Developer Zone:
108109
- 'Dev Environment': 'dev/development.md'
109110
- 'Contributing': 'dev/contributing.md'
111+
- 'Documentation': 'dev/documentation.md'
110112
- 'DAOS Internals': 'https://github.com/daos-stack/daos/blob/master/src/README.md'
111113
- 'DAOS API Documentation': 'https://docs.daos.io/v2.6/doxygen/html/index.html'
112114
# Attention: Don't change that doxygen path to a relative path, or mkdocs will stumble...

0 commit comments

Comments
 (0)