Add loader and det cal functionality to load in bandpass info using bg

[tdalford]

Adds custom functionality to be able to load in bandpass information using the det_cal bias group information. The pull request does a combination of:

1. Modify update_det_cal.py to calculate and save bandpass information inside the ManifestDB from the bias group.
2. Modify the PCARelCal module in processes.py to allow a fieldname for the bandpass as input rather than using the hard-coded wafer.bandpass
3. Add a custom metadata loader which allows input of which fields to load into a ResultSet, ignoring the rest. The loader also allows a user to modify the field names.

With the custom loader we can load up the bandpass field from the new det_cal into aman.det_info using e.g. a field name dets:det_cal.bandpass. The rest of the det cal information can be then loaded into aman.det_cal with a separate context entry using the same database.

Here's an example context file snippet which uses the loader to load det_cal information. This assumes that the det_cal database has the bandpass information stored in the field bandpass.

metadata:
  - db: '{manifestdir}/smurf_detsets/v0/smurf_detset_info_local.sqlite'
    label: smurf
    det_info: true
    on_missing: fail

  - db: 'my_det_cal_database.sqlite'
    label: det_cal
    det_info: True
    loader: ModifyFieldsResultSetHdf
    load_fields:
      - 'dets:readout_id'
      - 'bandpass': 'dets:det_cal.bandpass'
    on_missing: trim

  - db: 'my_det_cal_database.sqlite'
    label: det_cal
    unpack: det_cal
    on_missing: trim

In this example, the custom loader first loads in bandpass information from det_cal into aman.det_info.det_cal.bandpass by specifying only to read in dets:readout_id and bandpass from our det cal database, as well as preprending dets:detcal. to the bandpass field so that it properly can be read into det_info.

Then, the default loader reads all of the det_cal fields into aman.det_cal.

Note that this has the slightly inefficient side effect that bandpass is loaded into det_cal but also inside det_info.det_cal in the same axismanager. This can be fixed with the current functionality if the user uses the custom loader and inputs every field besides bandpass to load, although this would be a fairly verbose context file. Or, if you want I can modify the loader code to add something like the input drop_fields to loader.from_loadspec but I figured simpler and fewer changes to the loader were better.

[msilvafe]

If referencing globals I think this needs to be capitalized variable names here.

[tdalford]

Yeah sorry, missed this earlier

[kmharrington]

I think I like this setup but will leave it to @mhasself to approve the additional loader option.

To deploy this we'll need to update the existing det cal databases correct? I believe these are the instructions for setting up to deploy the update. https://simonsobs.atlassian.net/wiki/spaces/PRO/pages/306249767/Site+Pipeline+Metadata+Logs

[mhasself]

Sorry for the delay. I'll review this soon but one thing to do immediately is add tests for ModifyFieldsResultSetHdfLoader. I think the current place where such things is exercised is tests/test_context.py. Adding a basic test of the loader features in there would be appreciated.

[mhasself]

Is there any reason you can think of (and I'll have a go, too) to not include this functionality directly in ResultSetHdfLoader? Seems backwards compatible.

[tdalford]

Is there any reason you can think of (and I'll have a go, too) to not include this functionality directly in ResultSetHdfLoader? Seems backwards compatible.

My reasons were a combination of:

1. The documentation mentioned that 'load_fields' was only used for the AxisManagerHDF loader
2. If I add the functionality to change the fieldnames using load_fields then I think I would have to modify the AxisManagerHDF load_fields behavior to do this also, which I could, but would be more changes.

But these aren't that good.

[kmharrington]

@mhasself Tommy has added tests, could we get a review here?

[mhasself]

Focusing on the metadata changes -- I think it would be good to just add this functionality directly to the base class. This is because the "load_fields" is already implemented on AxisManager loader, with similar semantics (which we can make more similar, in the future) -- and you've taken advantage of the fact that "load_fields", in metadata definitions, is propagated through to the loader. Which is great.

Once that is done there are a couple of documentation fixes -- i.e. remove the "this only works on AxisManager" warning; and mention instead that the ResultSet loader also lets you rename fields. The syntax should be included somehow in readthedocs -- I should be able to reconstruct it starting from this autodocced class.

Thanks for the tests.

[mhasself]

I think it would be better to call this "bandpass_key"

[tdalford]

Are you saying to use 'self.bandpass_key' as the class variable name or to use 'bandpass_key' as the config parameter name? Or both?

[tdalford]

I'm going to guess you mean the config parameter name, that is much more visible

[mhasself]

"load_fields_resultset" would distinguish this from test_120 more clearly.