Support several NectarCAM cameras #218

jlenain · 2025-10-02T09:14:20Z

This PR aims at adding support of multiple NectarCAM cameras within nectarchain.

At the moment, support for the submission of DIRAC DQM jobs, the DQM itself, and parsing the DQM results have been implemented.

Support in core classes is currently missing, as well as in the DQM Bokeh website.

[EDIT] Support in core classes is now included.

codecov · 2025-10-02T09:43:36Z

Codecov Report

❌ Patch coverage is 49.47368% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.83%. Comparing base (09b87e2) to head (ad870b9).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/nectarchain/makers/core.py	50.00%	8 Missing ⚠️
src/nectarchain/dqm/charge_integration.py	14.28%	6 Missing ⚠️
src/nectarchain/dqm/mean_camera_display.py	14.28%	6 Missing ⚠️
src/nectarchain/dqm/pixel_participation.py	20.00%	4 Missing ⚠️
src/nectarchain/data/management.py	40.00%	3 Missing ⚠️
src/nectarchain/dqm/camera_monitoring.py	25.00%	3 Missing ⚠️
src/nectarchain/dqm/mean_waveforms.py	25.00%	3 Missing ⚠️
src/nectarchain/dqm/pixel_timeline.py	25.00%	3 Missing ⚠️
src/nectarchain/dqm/start_dqm.py	0.00%	3 Missing ⚠️
src/nectarchain/makers/component/core.py	85.00%	3 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #218      +/-   ##
==========================================
+ Coverage   51.79%   51.83%   +0.04%     
==========================================
  Files          78       79       +1     
  Lines        6505     6542      +37     
==========================================
+ Hits         3369     3391      +22     
- Misses       3136     3151      +15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

review-notebook-app · 2025-10-03T09:32:54Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…esult FITS parser

jlenain · 2025-10-03T09:55:20Z

Support in core classes has been implemented by @guillaumegrolleron , thanks a lot !

jlenain · 2025-10-03T14:08:39Z

Hi !

Anyone willing to review this PR, please ?

jlenain · 2025-10-06T07:38:43Z

Hi @guillaumegrolleron !

I have a small issue with this PR: in data.management.DataManagement.findrun, I wanted to add the possibility to request for which camera (QM, NectarCAM-2, NectarCAM-3, ...) a run should be fetched. Indeed, the same run number could correspond to different data, since each camera has its proper run number. So far, so good: https://github.com/cta-observatory/nectarchain/pull/218/files#diff-1808a3bbae11ffe2370dacb57610b8b31ad793e8c1e8a8746a0dc64e79df2ddaR34-R71

However, findrun is used in makers.core.load_run, which itself is used e.g. in makers.calibration.gain.photostat_makers._load_eventsource.

What would be the best way to pass the camera we would want to use throughout the code ? By e.g. passing a camera option to our tools ?

Many thanks !

jlenain · 2025-10-06T10:24:49Z

Hi @guillaumegrolleron !

I have a small issue with this PR: in data.management.DataManagement.findrun, I wanted to add the possibility to request for which camera (QM, NectarCAM-2, NectarCAM-3, ...) a run should be fetched. Indeed, the same run number could correspond to different data, since each camera has its proper run number. So far, so good: https://github.com/cta-observatory/nectarchain/pull/218/files#diff-1808a3bbae11ffe2370dacb57610b8b31ad793e8c1e8a8746a0dc64e79df2ddaR34-R71

However, findrun is used in makers.core.load_run, which itself is used e.g. in makers.calibration.gain.photostat_makers._load_eventsource.

What would be the best way to pass the camera we would want to use throughout the code ? By e.g. passing a camera option to our tools ?

Many thanks !

Hi @guillaumegrolleron !

At the moment, I think the easiest would be to add a camera field to the class EventsLoopNectarCAMCalibrationTool. Commit coming.

However, I have an issue currently when doing that with:

Traceback (most recent call last):
  File "/home/jlenain/local/src/python/cta-observatory/nectarchain/src/nectarchain/user_scripts/ggrolleron/load_wfs_compute_charge.py", line 171, in main
    tool = ChargesNectarCAMCalibrationTool(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: nectarchain.makers.charges_makers.ChargesNectarCAMCalibrationTool() got multiple values for keyword argument 'camera'

~~Would you be able to help here please ?~~
EDIT: Fixed.

src/nectarchain/makers/core.py

src/nectarchain/user_scripts/ggrolleron/gain_SPEfit_combined_computation.py

guillaumegrolleron · 2025-10-06T11:33:34Z

Hi @jlenain
I also think the easiest way is to add a camera field, I quickly checked your changes and it looks good! Thanks!

jlenain · 2025-10-06T11:36:24Z

Hi @jlenain I also think the easiest way is to add a camera field, I quickly checked your changes and it looks good! Thanks!

Thanks a lot, @guillaumegrolleron !
Still one thing I am not happy about: the propagation of the camera is not yet done properly. For instance:

python load_wfs_compute_charge.py --camera="NectarCAM2" -r 133 --max_events 100  --method GlobalPeakWindowSum --extractor_kwargs '{"window_width":16,"window_shift":4}' --overwrite

still fetch data from the QM:

[...]
2025-10-06 10:34:39,362 INFO [nectarchain.data.management] (management.findrun): will search files on GRID and fetch them
/vo.cta.in2p3.fr/nectarcam/2016/20160706/NectarCAM.Run0133.fits.fz
/vo.cta.in2p3.fr/nectarcam/2016/20160706/Run0133.fits.fz
2025-10-06 10:34:41 UTC Framework/GFAL2_SRM2Storage/GFAL2_StorageBase._getSingleFile INFO: Trying to download srm://ccsrm02.in2p3.fr:8443/srm/managerv2?SFN=/pnfs/in2p3.fr/data/vo.cta.in2p3.fr/nectarcam/2016/20160706/NectarCAM.Run0133.fits.fz to /data/users/jlenain/tmp/runs/NectarCAM.Run0133.fits.fz
2025-10-06 10:34:42 UTC Framework/GFAL2_SRM2Storage/GFAL2_StorageBase._getSingleFile INFO: Trying to download srm://ccsrm02.in2p3.fr:8443/srm/managerv2?SFN=/pnfs/in2p3.fr/data/vo.cta.in2p3.fr/nectarcam/2016/20160706/Run0133.fits.fz to /data/users/jlenain/tmp/runs/Run0133.fits.fz
[...]

Something is still fishy, I am investigating...

guillaumegrolleron · 2025-10-06T11:50:28Z

Hi @jlenain I also think the easiest way is to add a camera field, I quickly checked your changes and it looks good! Thanks!

Thanks a lot, @guillaumegrolleron ! Still one thing I am not happy about: the propagation of the camera is not yet done properly. For instance:

python load_wfs_compute_charge.py --camera="NectarCAM2" -r 133 --max_events 100  --method GlobalPeakWindowSum --extractor_kwargs '{"window_width":16,"window_shift":4}' --overwrite

still fetch data from the QM:

[...]
2025-10-06 10:34:39,362 INFO [nectarchain.data.management] (management.findrun): will search files on GRID and fetch them
/vo.cta.in2p3.fr/nectarcam/2016/20160706/NectarCAM.Run0133.fits.fz
/vo.cta.in2p3.fr/nectarcam/2016/20160706/Run0133.fits.fz
2025-10-06 10:34:41 UTC Framework/GFAL2_SRM2Storage/GFAL2_StorageBase._getSingleFile INFO: Trying to download srm://ccsrm02.in2p3.fr:8443/srm/managerv2?SFN=/pnfs/in2p3.fr/data/vo.cta.in2p3.fr/nectarcam/2016/20160706/NectarCAM.Run0133.fits.fz to /data/users/jlenain/tmp/runs/NectarCAM.Run0133.fits.fz
2025-10-06 10:34:42 UTC Framework/GFAL2_SRM2Storage/GFAL2_StorageBase._getSingleFile INFO: Trying to download srm://ccsrm02.in2p3.fr:8443/srm/managerv2?SFN=/pnfs/in2p3.fr/data/vo.cta.in2p3.fr/nectarcam/2016/20160706/Run0133.fits.fz to /data/users/jlenain/tmp/runs/Run0133.fits.fz
[...]

Something is still fishy, I am investigating...

I guess the problem comes from the propagation of the self.camera argument here

nectarchain/src/nectarchain/makers/core.py

Line 255 in f6fe8b3

self.event_source = self.enter_context(

jlenain · 2025-10-06T11:55:02Z

I guess the problem comes from the propagation of the self.camera argument here

nectarchain/src/nectarchain/makers/core.py

Line 255 in f6fe8b3

self.event_source = self.enter_context(

Argh, good catch, many thanks ! That does the trick !

jlenain · 2025-10-10T14:01:36Z

It may be easier to me to have the relevant changes for the bokeh app in another PR. Marking this PR as ready for review.

jlenain assigned jlenain, hashkar and guillaumegrolleron Oct 2, 2025

jlenain added the enhancement New feature or request label Oct 2, 2025

This was linked to issues Oct 2, 2025

DQM should be able to handle several NectarCAM cameras #217

Open

Manage data for multiple cameras #190

Open

jlenain marked this pull request as ready for review October 2, 2025 14:03

jlenain removed a link to an issue Oct 2, 2025

Manage data for multiple cameras #190

Open

jlenain and others added 4 commits October 3, 2025 11:48

Support of multiple NectarCAM cameras for DQM job submitter and DQM r…

d2713ea

…esult FITS parser

Adapt the DQM to support multiple NectarCAM cameras

59741a4

handle several nectarCAM cameras

570532f

Update notebook

fea5470

jlenain force-pushed the several_cameras branch from 0b90d3c to fea5470 Compare October 3, 2025 09:51

Change help message

502bae1

jlenain added 4 commits October 3, 2025 12:18

Lost a fundamental change during an unfortunate rebase onto main

b0c0cb7

Update unit tests for DQM

1656aec

Lint

72d2924

Leftovers

e20aeb2

jlenain unassigned hashkar Oct 3, 2025

jlenain marked this pull request as draft October 3, 2025 14:10

Should cover and close cta-observatory#190

e0caa91

jlenain linked an issue Oct 3, 2025 that may be closed by this pull request

Manage data for multiple cameras #190

Open

jlenain added 4 commits October 3, 2025 16:22

Better doc

f948b96

Add option to select camera to DQM

92fb281

Typo

a86d2ea

Propagate camera option to DQM job submitter

e48302d

jlenain added 2 commits October 6, 2025 12:25

Propagate camera option to core code, and gain scripts

2bb3ad7

Properly propagate camera option to load_wfs_compute_charge script

f6fe8b3

guillaumegrolleron reviewed Oct 6, 2025

View reviewed changes

src/nectarchain/makers/core.py Outdated Show resolved Hide resolved

guillaumegrolleron reviewed Oct 6, 2025

View reviewed changes

src/nectarchain/user_scripts/ggrolleron/gain_SPEfit_combined_computation.py Outdated Show resolved Hide resolved

guillaumegrolleron requested a review from hashkar October 6, 2025 11:35

Properly propagate camera option in core code

b0e7fd9

jlenain added 8 commits October 6, 2025 17:43

Remove duplicate code

4477865

Propagate camera option to gain users' scripts

14b2114

Propagate telescope ID to flatfield and waveforms components

1f99a6d

Propagate telescope ID to tool tutorial

ef9315a

More consistent auto-discovery of QM as default

c5305b0

Lint

3f08a48

Include camera number in DQM results archive

cad0773

Fix typo

f485835

jlenain marked this pull request as ready for review October 10, 2025 14:01

jlenain added 3 commits October 10, 2025 16:51

Fix typo

d9b29a0

Forgot to propagate camera to DIRAC parent directory for DQM results

a0a071f

Using a local container image is just for testing this PR !

b8505b7

guillaumegrolleron mentioned this pull request Oct 15, 2025

First working version of pipeline calibration tool #226

Draft

Add support for several cameras in flat-field script

ad870b9

Support several NectarCAM cameras #218

Are you sure you want to change the base?

Support several NectarCAM cameras #218

Uh oh!

Conversation

jlenain commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

review-notebook-app bot commented Oct 3, 2025

Uh oh!

jlenain commented Oct 3, 2025

Uh oh!

jlenain commented Oct 3, 2025

Uh oh!

jlenain commented Oct 6, 2025

Uh oh!

jlenain commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guillaumegrolleron commented Oct 6, 2025

Uh oh!

jlenain commented Oct 6, 2025

Uh oh!

guillaumegrolleron commented Oct 6, 2025

Uh oh!

jlenain commented Oct 6, 2025

Uh oh!

jlenain commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jlenain commented Oct 2, 2025 •

edited

Loading

codecov bot commented Oct 2, 2025 •

edited

Loading

jlenain commented Oct 6, 2025 •

edited

Loading