FPS/TVAC datamodels being auto converted into a ScienceRaw model

[WilliamJamieson]

NOTE: This is both an RAD and RDM issue, more so a RDM issue insofar-as implementation; however, the stakeholders for this mostly follow RAD much more closely than RDM.

TLDR: have rdm.open open Fps and Tvac directly as a ScienceRawModel rather than as their own unique datamodel types.

FPS/TVAC are realistically the main sticking points for implementing #860 this is because internally they have nested tags and tagged scalars. In RCAL, they are immediately converted into ScienceRawModel as soon as they are opened

https://github.com/spacetelescope/romancal/blob/487096953036ac57ab0b82bdddfac395997c4d78/romancal/dq_init/dq_init_step.py#L52-L59

and there is only one small area of special processing for these files here:

https://github.com/spacetelescope/romancal/blob/487096953036ac57ab0b82bdddfac395997c4d78/romancal/dq_init/dq_init_step.py#L83-L90

As such we could basically open those files as ScienceRawModel and apply the necessary special processing so that the data users get out of rdm.open is consistent with how the normal ScienceRawModel data acts. asdf.open would still open things using the structure we see now so users could interact with it directly if they so desire. We have no real need to support the TvacModel and FpsModel directly for the pipeline so we have no real reason to support these as datamodels directly. Note that we could add an optional field such as meta.ground that lists if the ScienceRawModel came from FPS or TVAC (it being missing would indicate that the model is the normal case). The pipeline would have no reason to use this, but it could be useful for end users.

In the end this would allow is to simplify RDM significantly and reduce complexity inside it.

[schlafly]

Repeating your suggestion to make sure I understand:

• We could still open TVAC / FPS data with asdf as usual.
• rdm could still open TVAC / FPS data but what the users would see would not match the asdf file as closely as usual; it would look more like a normal L1 file plus extras
• behavior in romancal would be unchanged.

I think that all sounds good to me. @tddesjardins , do you think that's breaking any 'contract' to continue to support TVAC / FPS files indefinitely? Do you use those files in, e.g., tutorials / notebooks? Thanks!

[tddesjardins]

I think this stays within the spirit of what we said we would support, which is that the data should remain compatible with romancal. Let me get concurrence from some stakeholders...standby.

[tddesjardins]

Okay, no concerns from RTB. But I'm adding @rcosenti-stsci for his visibility.

[PaulHuwe]

My only concern is that it removes the ability to modify any TVAC / FPS file and save it as such within RDM. Is this acceptable?

[schlafly]

My only concern is that it removes the ability to modify any TVAC / FPS file and save it as such within RDM. Is this acceptable?

FWIW, I'm okay with that; they can save it as science raw (more normal anyway) or edit with vanilla asdf.

[braingram]

edit with vanilla asdf

Thanks, to expand on this point, is the expectation that for a tvac/fps file. The user could:

af = asdf.open("some_tvac.asdf")
af["roman"]["meta"]["custom_thing"] = "foo"
af.write_to("modified_tvac.asdf")

If so, (since behind the scenes asdf use rdm) whatever solution we come with here should be compatible with the above example where the produced modified file contains the same tags as the original file?

[schlafly]

edit with vanilla asdf

Thanks, to expand on this point, is the expectation that for a tvac/fps file. The user could:

af = asdf.open("some_tvac.asdf")
af["roman"]["meta"]["custom_thing"] = "foo"
af.write_to("modified_tvac.asdf")
If so, (since behind the scenes asdf use rdm) whatever solution we come with here should be compatible with the above example where the produced modified file contains the same tags as the original file?

That was what I was imagining, but if that's not possible or is problematic, we don't have to do that. I don't expect most people to want to edit TVAC files and re-save them as such, but if that becomes impossible we should know that going in.

[braingram]

From discussion at the romancal tag-up today

1. strun must continue to work
2. open and edit meta and save it is not important, but it is awkward, hard is different than impossible, it should not be impossible to save these files
3. RNN has a tvac notebook that works with tvac data, getting details about this is critical before dropping any support
4. some users may process tvac and flight data together, reading them in the same environment is critical for this with no metadata changes, saving isn't needed

It sounds like from the above that:

• expecting users to use an old version of rdm is not an option
• automatic migration to science raw (with fake metadata) is not an option
• modifications to tags/tag-versions on read/write are acceptable (so adding a tvac/fps schema with no tagged scalars and automatically migrating files could be a solution, or perhaps doing this without a new schema)

[schlafly]

@rizeladiaz , if you could share the TVAC notebook here, that would be appreciated, thanks!