Service Data Source API

[mexanick]

Please describe the use case that requires this feature.

ctapipe needs I/O capabilities to read DL0/Service data: normally, FITS tables with one table per Array Element per update period (irregular). Examples include:

• Cat-A camera calibration results
• Pointing/Bending model parameters
• ...

Describe the solution you'd like

An abstract API defining the interface similarly to EventSource or MonitoringSource.

[kosack]

In fact the ctapipe.utils.datasets was originally designed to be that interface. You can set CTAPIPE_SVC_PATH to list any paths where service data should be found (colon separated, like the PATH variable), and then use get_dataset_path(filename) or get_table_dataset(basename) to retrieve the data: it always searches the paths in CTAPIPE_SVC_PATH first, in order, and then falls back to the data server if no match is found. Of course, over time it became mostly used for test data, but the intent was to be more general, hence why there is the path you can set.

This still can work well for that case, as long as we have a file naming scheme and maybe just some light wrapper on top to manage the specific ones we need. That wrapper could just take care of matching the right service data to the given observation, since in general service data is not linked to a single obs_id: it changes on a much slower rate, so may even be things we update only once in a long while.

[kosack]

To be more clear, the way it was supposed to be used was as follows: imagine you have some calibration service data that changes every few days or so, and also some atmosphere service data that changes once a year. You would add both of those paths before running ctapipe, and inside you would just rely on the filename being static.

export CTAPIPE_SVC_PATH=/service/calibration/latest:/service/atmosphere/v2
ctapipe-process ... 

Then, e.g. when a class like ZFITSEventSource wants to load the atmosphere profile (which unlike the SimtelEventSource would not be inside each event file itself), it can just assume that in /service/atmosphere/v2 there is a file named "atmosphere_density.fits", and it can just do atmo = get_table_dataset("atmosphere_density"). The provenance system will record the full path to the one that was actually used (even resolving symlinks like latest/), as well as the value of CTAPIPE_SVC_PATH, so that info is not lost.

The other way this could be achieved, is to set all service data as command-line options for a given EventSource (or in a separate ServiceSource), which is also fine, but would then require that all service data is registered in one place, rather than in each tool/script that uses it.