CGI level-2a and level-2b keywords

[ejoliet]

Please, see attached the list of keywords and details updated for CGI metadata of level-2 files.

Thanks!

(SSC Internal Jira ticket ROMAN-3531)

cgi_l2a_metadata_definitions.csv
cgi_l2b_metadata_defintions.csv

[scfleming]

@agreenbaum as we start to finalize the CGI GUI, do you consider the column "MAST UI" indicating which metadata columns should have dedicated components to be complete at this point? If so, I'll work with our designer to get an updated GUI mock-up made and then we can schedule a meeting to go over it before we implement it starting next month.

Also, I haven't done this comparison myself yet, but given that the L1, L2a, and L2b are all going to be grouped into the same fileset, are the MAST UI components that are important enough to add to the UI basically what's in the L2b csv file, or should I cross-compare and make a superset?

[agreenbaum]

@agreenbaum as we start to finalize the CGI GUI, do you consider the column "MAST UI" indicating which metadata columns should have dedicated components to be complete at this point? If so, I'll work with our designer to get an updated GUI mock-up made and then we can schedule a meeting to go over it before we implement it starting next month.

Also, I haven't done this comparison myself yet, but given that the L1, L2a, and L2b are all going to be grouped into the same fileset, are the MAST UI components that are important enough to add to the UI basically what's in the L2b csv file, or should I cross-compare and make a superset?

Hi Scott, I think that's in that column for L1 is a good starting point, and we are meeting soon to discuss that, right? What do you mean by L2a and L2b being grouped into the same fileset, exactly? I am preparing an update to L2a/b as well as the other data products soon (working on that this week) so what you have is pretty preliminary for everything except L1.

[scfleming]

What I mean by that is, when a search is done and you get back a result table, the "fileset" would group similar products together. Since L1, L2a, and L2b are all exposure-based, my understanding was we were going to group them into a single "fileset" (based on the root of their filenames) and then a user could select which product level(s) they wanted to download.

If we do that, then the metadata we expose in your search result table would apply to that set of grouped files. So if the L1 metadata was effectively a subset of metadata from the L2a or L2b, since they would all be grouped together into one search result row (one "fileset") we'd only need to show the metadata from the most advanced product in that fileset, e.g., the L2b, if the L2a and L1 files have a subset of the metadata that's in the L2b file they are grouped with.

[scfleming]

Similar to how we group the SOC WFI "uncal.asdf" files with their corresponding "cal.asdf" files under the same fileset. We don't expose in the search results table any of the L1 (uncal.asdf) metadata, because they are grouped with the L2 (cal.asdf) files, and thus in the search results table the L2 metadata serves as sufficient metadata for the entire fileset (the uncal.asdf, the cal.asdf, the preview png files, etc.) that are all grouped under that one fileset.

[agreenbaum]

What I mean by that is, when a search is done and you get back a result table, the "fileset" would group similar products together. Since L1, L2a, and L2b are all exposure-based, my understanding was we were going to group them into a single "fileset" (based on the root of their filenames) and then a user could select which product level(s) they wanted to download.

If we do that, then the metadata we expose in your search result table would apply to that set of grouped files. So if the L1 metadata was effectively a subset of metadata from the L2a or L2b, since they would all be grouped together into one search result row (one "fileset") we'd only need to show the metadata from the most advanced product in that fileset, e.g., the L2b, if the L2a and L1 files have a subset of the metadata that's in the L2b file they are grouped with.

This makes sense and sounds good to me! What are the implications for us? That we don't need to duplicate the common metadata for levels 2+ that are exposure-based? The only exception is level 2b photon counting that does a combination of frames.

[scfleming]

No major implications for SSC side, it's still good to send the metadata in the L1 or L2a SNS messages. The only thing we (SOC) would want to know is if there's any metadata keywords that are ONLY in the L1 or L2a files AND are important enough to make available in the Search Interface. If not, we are good and can base the UI design 100% on the L2b metadata and the columns you marked as "must have" for the GUI buttons!

[agreenbaum]

@scfleming I know we will talk next week in more detail, but I'm trying to wrap my head around the special case of photon counting, which will lead to combination of frames from level 2b. In this case, level 2b can't inherit metadata one-to-one from L2a or L1 (but we will know which frames produced a given L2b). How should we handle this? I am thinking about a "PCFLAG" keyword for L2b (and the corresponding L3 file would inherit that).

[agreenbaum]

Follow up question: do you need MJD start / MJD end for ALL products, including L4s and cals?