Unit tests for makers, container and component modules (part 1/X) (#163)

* add arg to load only wfs and do not extract charges

* cleaner way to treat containers inside makers

* new methods to find computed calibration quatities
ds to find computed calibration quatities

* update to support new version of EvB v6

* -bug fix SPE combined : method use to update parameters was not the wanted one
-update pp and n values to last studies
-display wfs -> mean instead of sum
-Agg backend is not forced in Pstat calibnration

* add find photostat result method

* user scripts

* add workflow_display to CI

* change tests name in user_scripts which is not a test for pytest

* fix import errors in TRR
exluse user_scripts to pytest

* bugfix : logger
improvement split_run method

* static method decorator were missing

* bugfix test containers

* unit test fore nectarchain.makers .core

* bugfix with new version of LIghtNectarCAMEventSource from ctapipe_io_nectarcam

* very small code improvement

* Unit test for WaveformsNectarCAMCalibrationTool

* Unit test for the ChargesNectarCAMCalibrationTool

* formatting

* fix test core makers

* flake8 formatting

* refactoring following pep8

* continue refactoring

* dqm flake8

* refactoring containers

* fix bug formatting

* shitty python formatting

* test bugfix

* pedestal maker : implementation of I/O with containers

* remove LightNectarCamEventSource

* bugfix in toml file introduced in previous commits

* bugfix : __init__ file in tests repertory

* pytest don't like to have 2 test files with same names

* specific noqa for unused imports

---------

Co-authored-by: guillaume.grolleron <[email protected]>

Author: guillaumegrolleron

.github/workflows/ci.yml

@@ -2,6 +2,7 @@ name: CI
 
 # by not building all branches on push, we avoid the duplicated builds in PRs
 on:
+  workflow_dispatch:
   push:
   # TODO
   # Remove comment: just a test to force CI on push

docs/user-guide/env-vars.rst

@@ -9,7 +9,7 @@ However, some environment variables are needed for ``nectarchain`` to work prope
 Mandatory
 ---------
 
-:``NECTARCAMDATA``: path to local NectarCAM data. It can contain ``fits.fz`` run files, `~nectarchain.data.container.waveformsContainer.WaveformsContainer` or `~nectarchain.data.container.chargesContainer.ChargesContainer` HDF5 files. This is also where the `~nectarchain.data.management.DataManagement.findrun` method will automatically store NectarCAM run files when fetched from DIRAC.
+:``NECTARCAMDATA``: path to local NectarCAM data. It can contain ``fits.fz`` run files, `~nectarchain.data.container.waveforms_container.WaveformsContainer` or `~nectarchain.data.container.charges_container.ChargesContainer` HDF5 files. This is also where the `~nectarchain.data.management.DataManagement.findrun` method will automatically store NectarCAM run files when fetched from DIRAC.
 
 Optional
 --------

docs/user-guide/howto-pedestal.rst

@@ -25,7 +25,7 @@ processed in slices with a fixed number of events set by the
 
 The pedestal
 estimation tool inherits the configurable parameters of the
-`~nectarchain.makers.component.PedestalComponent.PedestalEstimationComponent`.
+`~nectarchain.makers.component.pedestal_component.PedestalEstimationComponent`.
 The data can be filtered based on time using the ``ucts_tmin`` and
 ``ucts_tmax`` parameters and to eliminate outlier waveforms using the ``filter_method`` parameter. Two different methods to exclude outlier
 waveforms are implemented:
@@ -46,11 +46,11 @@ To run the example script:
 Inspect the results
 =========================
 The results are stored in a
-`~nectarchain.data.container.pedestalContainer.NectarCAMPedestalContainer`. The
+`~nectarchain.data.container.pedestal_container.NectarCAMPedestalContainer`. The
 results include information on pixels that were flagged as having
 an abnormal behavior during the computation of the pedestals. The
 flags are defined in in
-`~nectarchain.data.container.pedestalContainer.PedestalFlagBits`. The
+`~nectarchain.data.container.pedestal_container.PedestalFlagBits`. The
 results are accessible on the fly if the tool is run interactively (as in the example above) and stored in a `.h5` file.
 
 The user script `nectarchain/user_scripts/ltibaldo/show_pedestal_output.py` provides an example of how to access the results from disk and produce some plots:

pyproject.toml

@@ -45,7 +45,6 @@ test = [
 dev = [
     "setuptools_scm",
 ]
-
 docs = [
     "sphinx",
     "sphinx-autodoc-typehints",
@@ -55,15 +54,30 @@ docs = [
     "numpydoc",
     "tomli; python_version < '3.11'"
 ]
-
 # we can use self-references to simplify all
 all = [
     "nectarchain[test,docs,dev]",
 ]
 
+
+[tool.pytest.ini_options]
+addopts = "--ignore=src/nectarchain/user_scripts"
+
+
+
 [tool.setuptools.packages.find]
 where = ["src"]
 exclude = ["nectarchain._dev_version"]
 
 [tool.setuptools_scm]
 write_to = "src/nectarchain/_version.py"
+
+[tool.ruff.lint]
+select = ["E", "F", "W"]  # This ensures it mimics Flake8's error codes.
+exclude = ["build/", "dist/", "*.pyc"] 
+
+[tool.ruff]
+line-length = 88
+
+[tool.black]
+line-length = 88

src/nectarchain/data/__init__.py

@@ -1,2 +1,34 @@
-from .container import *
-from .management import *
+"""Description: This file is used to import all the classes and functions
+from the data module."""
+
+from .container import (
+    ArrayDataContainer,
+    ChargesContainer,
+    ChargesContainers,
+    GainContainer,
+    NectarCAMContainer,
+    NectarCAMPedestalContainer,
+    SPEfitContainer,
+    TriggerMapContainer,
+    WaveformsContainer,
+    WaveformsContainers,
+    get_array_keys,
+    merge_map_ArrayDataContainer,
+)
+from .management import DataManagement
+
+__all__ = [
+    "ArrayDataContainer",
+    "NectarCAMContainer",
+    "TriggerMapContainer",
+    "get_array_keys",
+    "merge_map_ArrayDataContainer",
+    "ChargesContainer",
+    "ChargesContainers",
+    "WaveformsContainer",
+    "WaveformsContainers",
+    "GainContainer",
+    "SPEfitContainer",
+    "NectarCAMPedestalContainer",
+    "DataManagement",
+]

src/nectarchain/data/container/__init__.py

@@ -1,12 +1,35 @@
-from .chargesContainer import *
+"""This file is used to import all the containerclasses in the data/container
+folder."""
+
+from .charges_container import ChargesContainer, ChargesContainers
 from .core import (
     ArrayDataContainer,
     NectarCAMContainer,
     TriggerMapContainer,
     get_array_keys,
     merge_map_ArrayDataContainer,
 )
-from .eventSource import *
-from .gainContainer import *
-from .waveformsContainer import *
-from .pedestalContainer import *
+from .gain_container import GainContainer, SPEfitContainer
+from .pedestal_container import (
+    NectarCAMPedestalContainer,
+    NectarCAMPedestalContainers,
+    PedestalFlagBits,
+)
+from .waveforms_container import WaveformsContainer, WaveformsContainers
+
+__all__ = [
+    "ArrayDataContainer",
+    "NectarCAMContainer",
+    "TriggerMapContainer",
+    "get_array_keys",
+    "merge_map_ArrayDataContainer",
+    "ChargesContainer",
+    "ChargesContainers",
+    "WaveformsContainer",
+    "WaveformsContainers",
+    "GainContainer",
+    "SPEfitContainer",
+    "NectarCAMPedestalContainer",
+    "NectarCAMPedestalContainers",
+    "PedestalFlagBits",
+]

src/nectarchain/data/container/chargesContainer.py renamed to src/nectarchain/data/container/charges_container.py

@@ -1,20 +1,20 @@
 import logging
 
-logging.basicConfig(format="%(asctime)s %(name)s %(levelname)s %(message)s")
-log = logging.getLogger(__name__)
-log.handlers = logging.getLogger("__main__").handlers
-
 import numpy as np
 from ctapipe.containers import Field, Map, partial
 
 from .core import ArrayDataContainer, TriggerMapContainer
 
+logging.basicConfig(format="%(asctime)s %(name)s %(levelname)s %(message)s")
+log = logging.getLogger(__name__)
+log.handlers = logging.getLogger("__main__").handlers
+
+
 __all__ = ["ChargesContainer", "ChargesContainers"]
 
 
 class ChargesContainer(ArrayDataContainer):
-    """
-    A container that holds information about charges from a specific run.
+    """A container that holds information about charges from a specific run.
 
     Fields:
       charges_hg (np.ndarray): An array of high gain charges.
@@ -40,13 +40,13 @@ class ChargesContainer(ArrayDataContainer):
 
 
 class ChargesContainers(TriggerMapContainer):
-    """
-    Class representing a ChargesContainers.
-
-    This class inherits from the `TriggerMapContainer` class and is used to store trigger or slices of data mappings of `ChargesContainer`.
+    """Class representing a ChargesContainers.This class inherits from the
+    `TriggerMapContainer` class and is used to store trigger or slices of data
+    mappings of `ChargesContainer`.
 
     Attributes:
-        containers (Field): A field representing the trigger or slices of data mapping of `ChargesContainer`.
+        containers (Field): A field representing the trigger or slices
+        of data mapping of `ChargesContainer`.
     """
 
     containers = Field(