Issues with CRDS getreferences for darks

[npadmana]

I'm a little confused about CRDS references for darks and the MA Table Numbers in romanisim.

If I use eg. a spectroscopy MA table number (eg 1036), the call to CRDS getreferences fails for the darks because the MA Table Number does not match anything available for the darks (you can see this also by looking at the CRDS webpage). However, the MA Table numbers for the darks don't seem to correspond to other MA Table numbers, including those here (https://roman-docs.stsci.edu/roman-instruments-home/wfi-imaging-mode-user-guide/observing-with-the-wfi-in-imaging-mode/wfi-multiaccum-tables).

Note that this persists even with the bugfixes in #254 .

Here is the error I get

(roman-isim-expt-2) [2025-09-Sep/roman-isim-expt-2 2:53]% ./romanisim/scripts/romanisim-make-image out.asdf --radec 270 66 --usecrds --ma_table_number 1036
2025-09-12 02:53:50 INFO     Starting simulation...
2025-09-12 02:53:50 WARNING  romanisim is under active development.  Its output has not been formally validated; only limited testing has been performed.  For this reason, use of romanisim for preparation of ROSES proposals is not advised.  Other packages like galsim's roman package or STIPS may better serve such purposes.
Using read pattern: [[1], [2], [3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16, 17, 18], [19, 20, 21, 22, 23, 24], [25, 26, 27, 28, 29, 30], [31, 32, 33, 34], [35, 36], [37]]
CRDS - ERROR -  Error determining best reference for 'dark'  =    parameter='ROMAN.META.EXPOSURE.MA_TABLE_NUMBER' value='1036.0' is not in ['1.0', '10.0', '101.0', '102.0', '103.0', '104.0', '105.0', '106.0', '107.0', '108.0', '109.0', '11.0', '110.0', '12.0', '13.0', '14.0', '15.0', '16.0', '17.0', '3.0', '4.0', '5.0', '510.0', '6.0', '7.0', '8.0', '9.0']
Traceback (most recent call last):
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/./romanisim/scripts/romanisim-make-image", line 126, in <module>
    go(args)
    ~~^^^^^^
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/./romanisim/scripts/romanisim-make-image", line 62, in go
    ris.simulate_image_file(args, metadata, cat, rng, persist)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/romanisim/romanisim/ris_make_utils.py", line 323, in simulate_image_file
    im, extras = image.simulate(
                 ~~~~~~~~~~~~~~^
        metadata, cat, usecrds=args.usecrds,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        stpsf=args.stpsf, level=args.level, persistence=persist,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        rng=rng)
        ^^^^^^^^
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/romanisim/romanisim/image.py", line 787, in simulate
    refdata = gather_reference_data(image_mod, usecrds=usecrds)
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/romanisim/romanisim/image.py", line 626, in gather_reference_data
    reffiles.update(crds.getreferences(
                    ~~~~~~~~~~~~~~~~~~^
        image_mod.get_crds_parameters(), reftypes=refsneeded,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        observatory='roman'))
        ^^^^^^^^^^^^^^^^^^^^
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/.pixi/envs/default/lib/python3.13/site-packages/crds/core/heavy_client.py", line 127, in getreferences
    best_refs_paths = api.cache_references(
        final_context, bestrefs, ignore_cache=ignore_cache)
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/.pixi/envs/default/lib/python3.13/site-packages/crds/client/api.py", line 1000, in cache_references
    wanted = _get_cache_filelist_and_report_errors(bestrefs)
  File "/Users/np274/00-Notebook/2025-09-Sep/roman-isim-expt-2/.pixi/envs/default/lib/python3.13/site-packages/crds/client/api.py", line 1044, in _get_cache_filelist_and_report_errors
    raise last_error
crds.core.exceptions.CrdsLookupError: Error determining best reference for 'dark'  =    parameter='ROMAN.META.EXPOSURE.MA_TABLE_NUMBER' value='1036.0' is not in ['1.0', '10.0', '101.0', '102.0', '103.0', '104.0', '105.0', '106.0', '107.0', '108.0', '109.0', '11.0', '110.0', '12.0', '13.0', '14.0', '15.0', '16.0', '17.0', '3.0', '4.0', '5.0', '510.0', '6.0', '7.0', '8.0', '9.0']

[npadmana]

So, I believe part of this comes from an apparent inconsistency between the darks and the MA Table numbers in the latest context (0088.pmap). Going back to a previous one (setting the CRDS_CONTEXT environment variable, things work better.

So maybe this is a do not fix, until the CRDS files are updated...

[schlafly]

Thanks Nikhil. I'm tagging @rcosenti-stsci , who is leads the Roman reference file pipeline. He'll correct me, but I believe the situation is this:

• There's been a bit of a long-term effort to migrate from the current set of reference files and MA tables to a new set.
• One piece of that work has been moving from per-MA table dark reference files to per-device dark reference files, which dramatically reduces the number of dark reference files we need to make.
• The code to support those new per-detector reference files is now in place, and so soon (next week??) we expect new darks that are MA table number independent, after which this should work.
• In the interim, only a handful of MA tables end up supported in romanisim / romancal / CRDS.

If you are able to fix this just by pointing to an old context, great, that's the right solution for now. In the future there won't be any per-MA table reference files, so there shouldn't be any challenges with keeping the darks & the MA tables in sync.

I'll leave this open for a bit while the discussion is ongoing, but yes, I think ultimately this isn't a romanisim issue.