Skip to content

Commit 6271ba5

Browse files
ndenbmorris3WilliamJamieson
authored
RAD-151: L2 and L3 catalog and segmentation map schemas (#393)
* adding source catalog schema * fix changelog * adding segmentation map schema * Update src/rad/resources/schemas/source_catalog-1.0.0.yaml Co-authored-by: William Jamieson <wjamieson@stsci.edu> * add meta to source catalog * Update CHANGES.rst Co-authored-by: Nadia Dencheva <nadia.astropy@gmail.com> * Update src/rad/resources/schemas/segmentation_map-1.0.0.yaml Co-authored-by: Nadia Dencheva <nadia.astropy@gmail.com> * Updates from Nadia's review comments * fixing meta and binary block order * tweaking metadata requirements * this will be L3 schema * add mosaic_source_catalog and mosaic_segmentation_map * L2 catalogs and segmentation maps * add optical_element * add change log entry * address comments * address comments * Add database destination tables for mosaic_source_catalog * add archive destinations for L2 catalogs * add segment destination * remove SegmentationMap table from exposure schema * add photometry to L2 catalogs --------- Co-authored-by: Brett M. Morris <bmmorris@stsci.edu> Co-authored-by: Brett M. Morris <morrisbrettm@gmail.com> Co-authored-by: William Jamieson <wjamieson@stsci.edu>
1 parent d869ddc commit 6271ba5

20 files changed

Lines changed: 296 additions & 61 deletions

CHANGES.rst

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
0.19.1 (unreleased)
1+
0.20.0 (unreleased)
22
-------------------
33

44
- Add new schemas to documentation. [#386]
@@ -26,6 +26,7 @@
2626

2727
- Create the flux step schema. [#395]
2828

29+
- Add source_catalog and segmentation_map schemas for Level 2 and Level 3 files. [#393]
2930

3031
0.19.0 (2024-02-09)
3132
-------------------

docs/creating.rst

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -386,6 +386,15 @@ actually functions; meaning that for ASDF 3.0+ to correctly handle the schema
386386
without issues, the ``allOf`` combiner is necessary, see
387387
`PR 222 <https://github.com/spacetelescope/rad/pull/222>`_ for more details.
388388

389+
Testing Schemas
390+
---------------
391+
392+
Once you created a schema, run the tests in the ``rad`` package before proceeding
393+
to write the model.
394+
395+
.. note::
396+
The schemas need to be committed to the working repository and the ``rad``
397+
package needs to be installed before running the tests.
389398

390399
Creating a Data Model
391400
---------------------

src/rad/resources/manifests/datamodels-1.0.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,26 @@ tags:
268268
title: Calibration reference file names.
269269
description: |-
270270
Calibration reference file names.
271+
- tag_uri: asdf://stsci.edu/datamodels/roman/tags/source_catalog-1.0.0
272+
schema_uri: asdf://stsci.edu/datamodels/roman/schemas/source_catalog-1.0.0
273+
title: Source catalog
274+
description: |-
275+
Photometry and astrometry computed by the Source Catalog Step
276+
- tag_uri: asdf://stsci.edu/datamodels/roman/tags/segmentation_map-1.0.0
277+
schema_uri: asdf://stsci.edu/datamodels/roman/schemas/segmentation_map-1.0.0
278+
title: Segmentation map
279+
description: |-
280+
Segmentation map computed by the Source Catalog Step
281+
- tag_uri: asdf://stsci.edu/datamodels/roman/tags/mosaic_source_catalog-1.0.0
282+
schema_uri: asdf://stsci.edu/datamodels/roman/schemas/mosaic_source_catalog-1.0.0
283+
title: Mosaic source catalog
284+
description: |-
285+
Photometry and astrometry computed by the Source Catalog Step
286+
- tag_uri: asdf://stsci.edu/datamodels/roman/tags/mosaic_segmentation_map-1.0.0
287+
schema_uri: asdf://stsci.edu/datamodels/roman/schemas/mosaic_segmentation_map-1.0.0
288+
title: Mosaic segmentation map
289+
description: |-
290+
Segmentation map computed by the Source Catalog Step
271291
# Tagged Scalars
272292
- tag_uri: asdf://stsci.edu/datamodels/roman/tags/calibration_software_version-1.0.0
273293
schema_uri: asdf://stsci.edu/datamodels/roman/schemas/tagged_scalars/calibration_software_version-1.0.0

src/rad/resources/schemas/exposure-1.0.0.yaml

Lines changed: 34 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ allOf:
2222
origin: TBD
2323
archive_catalog:
2424
datatype: int
25-
destination: [WFIExposure.exposure_id, GuideWindow.exposure_id]
25+
destination: [WFIExposure.exposure_id, GuideWindow.exposure_id,
26+
SourceCatalog.exposure_id]
2627
mid_time:
2728
title: Exposure Mid Time (UTC)
2829
description: |
@@ -34,7 +35,8 @@ allOf:
3435
origin: TBD
3536
archive_catalog:
3637
datatype: datetime2
37-
destination: [WFIExposure.exposure_mid_time, GuideWindow.exposure_mid_time]
38+
destination: [WFIExposure.exposure_mid_time, GuideWindow.exposure_mid_time,
39+
SourceCatalog.exposure_mid_time]
3840
end_time:
3941
title: Exposure End Time (UTC)
4042
description: |
@@ -46,7 +48,8 @@ allOf:
4648
origin: TBD
4749
archive_catalog:
4850
datatype: datetime2
49-
destination: [WFIExposure.exposure_end_time, GuideWindow.exposure_end_time]
51+
destination: [WFIExposure.exposure_end_time, GuideWindow.exposure_end_time,
52+
SourceCatalog.exposure_end_time]
5053
start_time_mjd:
5154
title: MJD Start Time (d)
5255
description: |
@@ -59,7 +62,8 @@ allOf:
5962
origin: TBD
6063
archive_catalog:
6164
datatype: float
62-
destination: [WFIExposure.exposure_start_time_mjd, GuideWindow.exposure_start_time_mjd]
65+
destination: [WFIExposure.exposure_start_time_mjd, GuideWindow.exposure_start_time_mjd,
66+
SourceCatalog.exposure_start_time_mjd]
6367
mid_time_mjd:
6468
title: MJD Mid Time (d)
6569
description: |
@@ -72,7 +76,8 @@ allOf:
7276
origin: TBD
7377
archive_catalog:
7478
datatype: float
75-
destination: [WFIExposure.exposure_mid_time_mjd, GuideWindow.exposure_mid_time_mjd]
79+
destination: [WFIExposure.exposure_mid_time_mjd, GuideWindow.exposure_mid_time_mjd,
80+
SourceCatalog.exposure_mid_time_mjd]
7681
end_time_mjd:
7782
title: MJD End Time (d)
7883
description: |
@@ -85,7 +90,8 @@ allOf:
8590
origin: TBD
8691
archive_catalog:
8792
datatype: float
88-
destination: [WFIExposure.exposure_end_time_mjd, GuideWindow.exposure_end_time_mjd]
93+
destination: [WFIExposure.exposure_end_time_mjd, GuideWindow.exposure_end_time_mjd,
94+
SourceCatalog.exposure_end_time_mjd]
8995
start_time_tdb:
9096
title: TDB Start Time (d)
9197
description: |
@@ -98,7 +104,8 @@ allOf:
98104
origin: TBD
99105
archive_catalog:
100106
datatype: float
101-
destination: [WFIExposure.exposure_start_time_tdb, GuideWindow.exposure_start_time_tdb]
107+
destination: [WFIExposure.exposure_start_time_tdb, GuideWindow.exposure_start_time_tdb,
108+
SourceCatalog.exposure_start_time_tdb]
102109
mid_time_tdb:
103110
title: TDB Mid Time (d)
104111
description: |
@@ -111,7 +118,8 @@ allOf:
111118
origin: TBD
112119
archive_catalog:
113120
datatype: float
114-
destination: [WFIExposure.exposure_mid_time_tdb, GuideWindow.exposure_mid_time_tdb]
121+
destination: [WFIExposure.exposure_mid_time_tdb, GuideWindow.exposure_mid_time_tdb,
122+
SourceCatalog.exposure_mid_time_tdb]
115123
end_time_tdb:
116124
title: TDB End Time (d)
117125
description: |
@@ -124,7 +132,8 @@ allOf:
124132
origin: TBD
125133
archive_catalog:
126134
datatype: float
127-
destination: [WFIExposure.exposure_end_time_tdb, GuideWindow.exposure_end_time_tdb]
135+
destination: [WFIExposure.exposure_end_time_tdb, GuideWindow.exposure_end_time_tdb,
136+
SourceCatalog.exposure_end_time_tdb]
128137
sca_number:
129138
title: SCA Number
130139
description: |
@@ -137,7 +146,8 @@ allOf:
137146
origin: TBD
138147
archive_catalog:
139148
datatype: int
140-
destination: [WFIExposure.exposure_sca_number, GuideWindow.exposure_sca_number]
149+
destination: [WFIExposure.exposure_sca_number, GuideWindow.exposure_sca_number,
150+
SourceCatalog.exposure_sca_number]
141151
gain_factor:
142152
title: Gain Factor
143153
type: number
@@ -147,7 +157,8 @@ allOf:
147157
origin: TBD
148158
archive_catalog:
149159
datatype: float
150-
destination: [WFIExposure.exposure_gain_factor, GuideWindow.exposure_gain_factor]
160+
destination: [WFIExposure.exposure_gain_factor, GuideWindow.exposure_gain_factor,
161+
SourceCatalog.exposure_gain_factor]
151162
integration_time:
152163
title: Effective Integration Time (s)
153164
description:
@@ -159,7 +170,8 @@ allOf:
159170
origin: TBD
160171
archive_catalog:
161172
datatype: float
162-
destination: [WFIExposure.exposure_integration_time, GuideWindow.exposure_integration_time]
173+
destination: [WFIExposure.exposure_integration_time, GuideWindow.exposure_integration_time,
174+
SourceCatalog.exposure_integration_time]
163175
elapsed_exposure_time:
164176
title: Elapsed Exposure Time (s)
165177
description: |
@@ -172,7 +184,8 @@ allOf:
172184
origin: TBD
173185
archive_catalog:
174186
datatype: float
175-
destination: [WFIExposure.elapsed_exposure_time, GuideWindow.elapsed_exposure_time]
187+
destination: [WFIExposure.elapsed_exposure_time, GuideWindow.elapsed_exposure_time,
188+
SourceCatalog.elapsed_exposure_time]
176189
effective_exposure_time:
177190
title: Effective Exposure Time (s)
178191
description: |
@@ -185,7 +198,8 @@ allOf:
185198
origin: TBD
186199
archive_catalog:
187200
datatype: float
188-
destination: [WFIExposure.effective_exposure_time, GuideWindow.effective_exposure_time]
201+
destination: [WFIExposure.effective_exposure_time, GuideWindow.effective_exposure_time,
202+
SourceCatalog.effective_exposure_time]
189203
duration:
190204
title: Exposure Duration (s)
191205
description: |
@@ -198,7 +212,8 @@ allOf:
198212
origin: TBD
199213
archive_catalog:
200214
datatype: float
201-
destination: [WFIExposure.exposure_duration, GuideWindow.exposure_duration]
215+
destination: [WFIExposure.exposure_duration, GuideWindow.exposure_duration,
216+
SourceCatalog.exposure_duration]
202217
level0_compressed:
203218
title: Level 0 Compression
204219
description: |
@@ -211,7 +226,8 @@ allOf:
211226
origin: TBD
212227
archive_catalog:
213228
datatype: nchar(1)
214-
destination: [WFIExposure.exposure_level0_compressed, GuideWindow.exposure_level0_compressed]
229+
destination: [WFIExposure.exposure_level0_compressed, GuideWindow.exposure_level0_compressed,
230+
SourceCatalog.exposure_level0_compressed]
215231
truncated:
216232
title: Truncated MA Table
217233
description: |
@@ -223,7 +239,8 @@ allOf:
223239
origin: TBD
224240
archive_catalog:
225241
datatype: nchar(1)
226-
destination: [WFIExposure.exposure_truncated]
242+
destination: [WFIExposure.exposure_truncated,
243+
SourceCatalog.exposure_truncated]
227244
required: [id,
228245
mid_time, end_time,
229246
start_time_mjd, mid_time_mjd, end_time_mjd,

src/rad/resources/schemas/mosaic_basic-1.0.0.yaml

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ properties:
1616
origin: TBD
1717
archive_catalog:
1818
datatype: float
19-
destination: [WFIMosaic.time_first_mjd]
19+
destination: [WFIMosaic.time_first_mjd, SourceCatalog.time_first_mjd, SegmentationMap.time_first_mjd]
2020
time_last_mjd:
2121
title: Latest component image end time in the mosaic in MJD
2222
type: number
@@ -26,7 +26,7 @@ properties:
2626
origin: TBD
2727
archive_catalog:
2828
datatype: float
29-
destination: [WFIMosaic.time_last_mjd]
29+
destination: [WFIMosaic.time_last_mjd, SourceCatalog.time_last_mjd, SegmentationMap.time_last_mjd]
3030
time_mean_mjd:
3131
title: Mean of mid-times of component images in MJD
3232
type: number
@@ -36,7 +36,7 @@ properties:
3636
origin: TBD
3737
archive_catalog:
3838
datatype: float
39-
destination: [WFIMosaic.time_mean_mjd]
39+
destination: [WFIMosaic.time_mean_mjd, SourceCatalog.time_mean_mjd, SegmentationMap.time_mean_mjd]
4040
max_exposure_time:
4141
title: Maximum component image exposure time in MJD
4242
type: number
@@ -46,7 +46,7 @@ properties:
4646
origin: TBD
4747
archive_catalog:
4848
datatype: float
49-
destination: [WFIMosaic.max_exposure_time]
49+
destination: [WFIMosaic.max_exposure_time, SourceCatalog.max_exposure_time, SegmentationMap.max_exposure_time]
5050
mean_exposure_time:
5151
title: Mean of component image exposure times in MJD
5252
type: number
@@ -56,7 +56,7 @@ properties:
5656
origin: TBD
5757
archive_catalog:
5858
datatype: float
59-
destination: [WFIMosaic.mean_exposure_time]
59+
destination: [WFIMosaic.mean_exposure_time, SourceCatalog.mean_exposure_time, SegmentationMap.mean_exposure_time]
6060
visit:
6161
title: Visit number within the observation, defined range of
6262
values is 1..999; included in obs_id and visit_id as 'VVV'.
@@ -67,7 +67,7 @@ properties:
6767
origin: TBD
6868
archive_catalog:
6969
datatype: smallint
70-
destination: [WFIMosaic.visit]
70+
destination: [WFIMosaic.visit, SourceCatalog.visit, SegmentationMap.visit]
7171
segment:
7272
title: Segment Number within pass, defined range is 1..999;
7373
included in obs_id and visit_id as 'SSS'.
@@ -78,7 +78,7 @@ properties:
7878
origin: TBD
7979
archive_catalog:
8080
datatype: smallint
81-
destination: [WFIMosaic.segment]
81+
destination: [WFIMosaic.segment, SourceCatalog.segment, SegmentationMap.segment]
8282
pass:
8383
title: Pass number within execution plan, defined range is 1..999;
8484
included in obs_id and visit_id as 'AA'.
@@ -89,7 +89,7 @@ properties:
8989
origin: TBD
9090
archive_catalog:
9191
datatype: smallint
92-
destination: [WFIMosaic.pass]
92+
destination: [WFIMosaic.pass, SourceCatalog.pass, SegmentationMap.pass]
9393
program:
9494
title: Program number, defined range is 1..18445;
9595
included in obs_id and visit_id as 'PPPPP'.
@@ -100,13 +100,13 @@ properties:
100100
origin: TBD
101101
archive_catalog:
102102
datatype: int
103-
destination: [WFIMosaic.program]
103+
destination: [WFIMosaic.program, SourceCatalog.program, SegmentationMap.program]
104104
survey:
105105
title: Observation Survey
106106
type: string
107107
archive_catalog:
108108
datatype: nvarchar(15)
109-
destination: [WFIMosaic.survey]
109+
destination: [WFIMosaic.survey, SourceCatalog.survey, SegmentationMap.survey]
110110
optical_element:
111111
$ref: wfi_optical_element-1.0.0
112112
sdf:
@@ -115,7 +115,7 @@ properties:
115115
origin: TBD
116116
archive_catalog:
117117
datatype: nvarchar(20)
118-
destination: [WFIMosaic.optical_element]
118+
destination: [WFIMosaic.optical_element, SourceCatalog.optical_element, SegmentationMap.optical_element]
119119
instrument:
120120
title: Instrument used to acquire the data
121121
type: string
@@ -126,7 +126,7 @@ properties:
126126
origin: TBD
127127
archive_catalog:
128128
datatype: nvarchar(5)
129-
destination: [WFIMosaic.instrument_name]
129+
destination: [WFIMosaic.instrument_name, SourceCatalog.instrument_name, SegmentationMap.instrument_name]
130130
location_name:
131131
title: Name of the skycell containing the mosaic
132132
type: string
@@ -136,7 +136,7 @@ properties:
136136
origin: TBD
137137
archive_catalog:
138138
datatype: nvarchar(25)
139-
destination: [WFIMosaic.location_name]
139+
destination: [WFIMosaic.location_name, SourceCatalog.location_name, SegmentationMap.location_name]
140140
product_type:
141141
title: Association product type
142142
type: string
@@ -146,7 +146,7 @@ properties:
146146
origin: TBD
147147
archive_catalog:
148148
datatype: nvarchar(25)
149-
destination: [WFIMosaic.product_type]
149+
destination: [WFIMosaic.product_type, SourceCatalog.product_type, SegmentationMap.product_type]
150150
propertyOrder: [ time_first_mjd, time_last_mjd, time_mean_mjd, max_exposure_time,
151151
mean_exposure_time, visit, segment, pass, program,
152152
survey, optical_element, instrument, location_name, product_type]
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
%YAML 1.1
2+
---
3+
$schema: asdf://stsci.edu/datamodels/roman/schemas/rad_schema-1.0.0
4+
id: asdf://stsci.edu/datamodels/roman/schemas/mosaic_segmentation_map-1.0.0
5+
6+
title: Segmentation map generated from a Level 3 file by the Source Catalog Step.
7+
8+
datamodel_name: MosaicSegmentationMapModel
9+
10+
archive_meta: None
11+
type: object
12+
properties:
13+
meta:
14+
allOf:
15+
- $ref: basic-1.0.0
16+
- type: object
17+
properties:
18+
basic:
19+
tag: asdf://stsci.edu/datamodels/roman/tags/mosaic_basic-1.0.0
20+
program:
21+
title: Program Information
22+
tag: asdf://stsci.edu/datamodels/roman/tags/program-1.0.0
23+
required: [basic, program]
24+
data:
25+
title: Segmentation map
26+
description: |
27+
Segmentation map of an image model, zeros correspond to background.
28+
tag: tag:stsci.edu:asdf/core/ndarray-1.*
29+
ndim: 2
30+
datatype: uint32
31+
exact_datatype: true
32+
33+
required: [meta, data]
34+
flowStyle: block
35+
propertyOrder: [meta, data]
36+
...

0 commit comments

Comments
 (0)