Skip to content

Commit 47c56c5

Browse files
jlenainjlenain
authored
Documentation (#149)
* Do not exclude DQM Bokeh app anymore, it is now mock-imported, and we may want to document it (e.g. by converting its README into a proper section in the doc). * Remove commented suppress_warnings, as we now use mock imports instead * Do not explicitlt write `nectarchain` in mock import settings * Changed README as reStructuredText. * Work on documentation * Lint * Fix references to API * Fix target ref, work on doc * Re-introduce default role for Sphinx, to fix reference to API * Some tips about DIRAC * Some doc formatting --------- Co-authored-by: Jean-Philippe Lenain <[email protected]>
1 parent bf6b1f8 commit 47c56c5

File tree

11 files changed

+180
-91
lines changed

11 files changed

+180
-91
lines changed

‎README.md‎

Lines changed: 0 additions & 23 deletions
This file was deleted.

‎README.rst‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
================================================
2+
nectarchain |pypi| |conda| |ci| |doc| |codecov|
3+
================================================
4+
5+
.. |ci| image:: https://github.com/cta-observatory/nectarchain/actions/workflows/ci.yml/badge.svg?branch=main
6+
:target: https://github.com/cta-observatory/nectarchain/actions/workflows/ci.yml?query=workflow%3ACI+branch%3Amain
7+
:alt: Test Status
8+
.. |pypi| image:: https://badge.fury.io/py/nectarchain.svg
9+
:target: https://pypi.org/project/nectarchain
10+
.. |conda| image:: https://anaconda.org/conda-forge/nectarchain/badges/version.svg
11+
:target: https://anaconda.org/conda-forge/nectarchain
12+
.. |codecov| image:: https://codecov.io/github/cta-observatory/nectarchain/graph/badge.svg?token=TDhZlJtbMv
13+
:target: https://codecov.io/github/cta-observatory/nectarchain
14+
.. |doc| image:: https://readthedocs.org/projects/nectarchain/badge/?version=latest
15+
:target: https://nectarchain.readthedocs.io/en/latest/?badge=latest
16+
:alt: Documentation Status
17+
18+
Repository for the high level analysis of the NectarCAM data, a camera to equip the
19+
medium-sized telescopes of `CTAO <https://www.ctao.org/>`_ in its Northern site.
20+
The analysis is heavily based on `ctapipe <https://github.com/cta-observatory/ctapipe>`_,
21+
adding custom code for NectarCAM calibration.
22+
23+
``nectarchain`` is currently pinned to ``ctapipe`` version 0.19.
24+
25+
This code is under rapid development. It is not recommended for production use unless you are an expert or developer!
26+
27+
* Code: https://github.com/cta-observatory/nectarchain
28+
* Docs: https://nectarchain.readthedocs.io/
29+
30+
Installation for Users
31+
======================
32+
33+
``nectarchain`` and its dependencies may be installed using the *Anaconda* or
34+
*Miniconda* package system. We recommend creating a conda virtual environment
35+
first, to isolate the installed version and dependencies from your main
36+
environment (this is optional).
37+
38+
The latest version of ``nectarchain`` can be installed via::
39+
40+
mamba install -c conda-forge nectarchain
41+
42+
or via::
43+
44+
pip install nectarchain
45+
46+
**Note**: to install a specific version of ``nectarchain``, take look at the documentation `here <https://nectarchain.readthedocs.io/en/latest/user-guide/index.html>`_.
47+
48+
**Note**: ``mamba`` is a C++ reimplementation of conda and can be found `here <https://github.com/mamba-org/mamba>`_.
49+
50+
Note this is *pre-alpha* software and is not yet stable enough for end-users (expect large API changes until the first stable 1.0 release).
51+
52+
Developers should follow the development install instructions found in the
53+
`documentation <https://nectarchain.readthedocs.io/en/latest/developer-guide/index.html>`_.
54+
55+
56+
Contributing
57+
============
58+
59+
All contribution are welcome.
60+
61+
Guidelines are the same as `ctapipe's ones <https://ctapipe.readthedocs.io/en/latest/developer-guide/getting-started.html>`_.
62+
See `here <https://ctapipe.readthedocs.io/en/latest/developer-guide/pullrequests.html#pullrequests>`_ how to make a pull request to contribute.
63+
64+
65+
Report issue / Ask a question
66+
=============================
67+
68+
Please use `GitHub Issues <https://github.com/cta-observatory/nectarchain/issues>`_ to report issues or `GitHub Discussions <https://github.com/cta-observatory/nectarchain/discussions>`_ for questions and discussions.

‎docs/conf.py‎

Lines changed: 7 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -75,19 +75,17 @@
7575
# source_suffix = ['.rst', '.md']
7676
source_suffix = ".rst"
7777

78+
# have all links automatically associated with the right domain.
79+
default_role = "py:obj"
80+
7881
templates_path = ["_templates"]
7982

80-
exclude_patterns = [
81-
"_build",
82-
"Thumbs.db",
83-
".DS_Store",
84-
"**/*.dqm.bokeh_app**",
85-
]
83+
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
8684

8785
autodoc_mock_imports = [
88-
"nectarchain.makers.extractor.charge_extractor",
89-
"nectarchain.makers.calibration.core",
90-
"nectarchain.dqm.bokeh_app",
86+
f"{project}.makers.calibration.core",
87+
f"{project}.makers.extractor.charge_extractor",
88+
f"{project}.dqm.bokeh_app",
9189
]
9290

9391
# intersphinx allows referencing other packages sphinx docs
@@ -101,11 +99,6 @@
10199
"ctapipe": ("https://ctapipe.readthedocs.io/en/v0.19.3/", None),
102100
}
103101

104-
# suppress_warnings = [
105-
# "autosummary",
106-
# "autosummary.import_cycle",
107-
# ]
108-
109102
# If true, `todo` and `todoList` produce output, else they produce nothing.
110103
todo_include_todos = True
111104

‎docs/developer-guide/getting-started.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ Please follow the `same conventions as ctapipe <https://ctapipe.readthedocs.io/e
3939
Optional DIRAC support
4040
----------------------
4141

42-
*Note*: this is **not** needed if you are using ``nectarchain`` as a container :ref:`as-a-container`, as DIRAC is already fully installed and configured within.
42+
*Note*: this is **not** needed if you are using ``nectarchain`` as a container (see :ref:`as-a-container`), as DIRAC is already fully installed and configured within.
4343

4444
To enable support for DIRAC within the same environment, do the following after the installation of ``nectarchain`` described above:
4545

@@ -64,13 +64,13 @@ Building the documentation
6464

6565
To locally build the documentation, optional dependencies should be installed with:
6666

67-
.. code-block::
67+
.. code-block:: console
6868
6969
$ pip install -e ".[docs]"
7070
7171
The documentation can then be compiled with:
7272

73-
.. code-block::
73+
.. code-block:: console
7474
7575
$ make -C docs html
7676

‎docs/user-guide/dqm.rst‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
.. _dqm:
2+
3+
Quick recipe for the Data Quality Monitoring script
4+
===================================================
5+
6+
Run locally
7+
-----------
8+
9+
To launch the Data Quality Monitoring (DQM), first activate the ``nectarchain`` ``conda`` environment:
10+
11+
.. code-block:: console
12+
13+
source activate nectarchain
14+
15+
Usage:
16+
17+
.. code-block:: console
18+
19+
$ python start_dqm.py -h
20+
21+
To automatically find and retrieve run files from DIRAC, use the ``-r`` option:
22+
23+
.. code-block:: console
24+
25+
$ python start_dqm.py -r 2720 $NECTARCAMDATA $NECTARCAMDATA
26+
27+
See :ref:`env-vars` for the usage of the ``$NECTARCAMDATA`` environment variable.
28+
29+
To manually use local run files, use the ``-i`` option **after** indicating the positional arguments for input and output directories:
30+
31+
.. code-block:: console
32+
33+
$ python start_dqm.py $NECTARCAMDATA $NECTARCAMDATA -i NectarCAM.Run2720.0000.fits.fz NectarCAM.Run2720.0001.fits.fz
34+
35+
As a DIRAC job
36+
--------------
37+
38+
The user script `nectarchain/user_scripts/jlenain/dqm_job_submitter/submit_dqm_processor.py` can be used to run the DQM as a DIRAC job:
39+
40+
.. code-block:: console
41+
42+
$ python submit_dqm_processor.py -h
43+
44+
Under the hood, it calls the ``dqm_processor.sh`` wrapper script, which itself launches an Apptainer instance of the ``nectarchain`` container on the DIRAC worker. This Apptainer image is automatically built and published in CI on releases.
45+
46+
The DQM runs one job per NectarCAM run. It is possible, for instance, to bulk-submit DIRAC jobs for all runs acquired during a given period, e.g.:
47+
48+
.. code-block:: console
49+
50+
$ d=2023-01-01
51+
$ while [ "$d" != 2023-03-01 ]; do python submit_dqm_processor.py -d $d; d=$(date -I -d "$d + 1 day"); done
52+
53+
The script will first assess whether DQM jobs have already been run for a given NectarCAM run, based on the output directory on DIRAC where this script stores its output. Look for the ``$DIRAC_OUTDIR`` environment variable in ``dqm_processor.sh``.

‎docs/user-guide/env-vars.rst‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
.. _env-vars:
2+
3+
Environment variables
4+
=====================
5+
6+
We strive for relying as little as possible on environment variables across ``nectarchain``.
7+
However, some environment variables are needed for ``nectarchain`` to work properly, especially to automatically fetch run files from the grid via DIRAC, or to store output results and plots.
8+
9+
Mandatory
10+
---------
11+
12+
:``NECTARCAMDATA``: path to local NectarCAM data. It can contain ``fits.fz`` run files, `~nectarchain.data.container.waveformsContainer.WaveformsContainer` or `~nectarchain.data.container.chargesContainer.ChargesContainer` HDF5 files. This is also where the `~nectarchain.data.management.DataManagement.findrun` method will automatically store NectarCAM run files when fetched from DIRAC.
13+
14+
Optional
15+
--------
16+
17+
:``NECTARCHAIN_LOG``: path for output logs for ``nectarchain``, generally defaulting to ``/tmp``.
18+
:``NECTARCHAIN_FIGURES``: path to store output figures for ``nectarchain``, generally defaulting to ``/tmp``, or configurable (see e.g. `nectarchain.makers.component.spe.spe_algorithm.SPEalgorithm`).

‎docs/user-guide/howto-dirac.rst‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,13 +75,39 @@ NectarCAM data can then be explored using ``dls``:
7575
NectarCAM.Run5568.0003.fits.fz
7676
NectarCAM.Run5568.0004.fits.fz
7777
78-
The `~nectarchain.data.management.DataManagement.findrun` method will
78+
Sometimes, it is useful to quickly find the DIRAC location of a given NectarCAM run. The command ``dfind`` comes in handy. Suppose you want to know where run 5326 is, or how much slice files it contains:
79+
80+
.. code-block:: console
81+
82+
$ dfind /vo.cta.in2p3.fr/nectarcam | grep 5326
83+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0000.fits.fz
84+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0001.fits.fz
85+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0002.fits.fz
86+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0003.fits.fz
87+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0004.fits.fz
88+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0005.fits.fz
89+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0006.fits.fz
90+
/vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0007.fits.fz
91+
$ dfind /vo.cta.in2p3.fr/nectarcam | grep 5326 | grep -e "fits.fz" | wc -l
92+
8
93+
94+
95+
The `nectarchain.data.management.DataManagement.findrun` method will
7996
automatically localize NectarCAM data on DIRAC, given a run number, and fetch
8097
the run files for you.
8198

8299
Tips
83100
====
84101

102+
Web portal
103+
^^^^^^^^^^
104+
105+
CTA-DIRAC has a web portal available at https://ctadirac-01.cscs.cta-observatory.org/DIRAC/ to interact with it. In particular, the *Job Monitor* application is very useful to interact with your jobs on DIRAC.
106+
107+
.. tip::
108+
109+
If you submitted your jobs using the ``cta_nectarcam`` role, you would need to re-authenticate in the portal with this role: Click on *Settings* in the bottom left, and select *cta_nectarcam* under *Group*.
110+
85111
Proxy error
86112
^^^^^^^^^^^
87113

‎docs/user-guide/index.rst‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,5 +7,7 @@ User Guide
77
:maxdepth: 2
88

99
getting-started
10+
env-vars
1011
howto-dirac
11-
troubleshooting
12+
dqm
13+
troubleshooting

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
55
[project]
66
name = "nectarchain"
77
description = "Analysis chain for the CTA MSTN NectarCAM prototype"
8-
readme = "README.md"
8+
readme = "README.rst"
99
authors = [{name = "CTAO NectarCAM collaboration"}]
1010
license = {text = "BSD-3-Clause"}
1111
classifiers = [

‎src/nectarchain/README.md‎

Lines changed: 0 additions & 7 deletions
This file was deleted.

0 commit comments

Comments
 (0)