You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
``DiaSources`` are associated with known solar system objects and stationary objects using an association radius of 1 arcsecond.
13
+
14
+
Solar System objects are stored in the ``SSObject`` catalog and stationary objects are stored in the ``DiaObject`` catalog.
15
+
``DiaSources`` not associated with either will create new ``DiaObjects``.
16
+
17
+
Variability characterization
18
+
----------------------------
19
+
20
+
A small set of light curve characterization parameters (timeseries features) are calculated for all ``DiaObjects``.
21
+
22
+
23
+
Alert Production Database
24
+
=========================
25
+
26
+
The live results of association are held in a Cassandra database known as the Alert Production Database (APDB).
27
+
28
+
Daily exports of the APDB contents will be make available to Data Rights holders for query in the Prompt Products Database (PPDB).
29
+
30
+
Duplication
31
+
-----------
32
+
Because is run in real time, if two consecutive images of the same sky area are processed simultaneously, it is possible for two new ``DiaObjects`` to be created at the same position.
33
+
These duplicates will be cleaned up in morning processing.
34
+
Similarly, alerts may not include the ``DiaSources`` from images taken immediately prior to the current image.
35
+
Such ``DiaSources`` will appear in future alerts to that ``DiaObject``, however.
36
+
This situation is most common for deep drilling fields but can also occur in field overlaps.
Copy file name to clipboardExpand all lines: docs/processing/dia/index.rst
+14-34Lines changed: 14 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,11 +4,6 @@
4
4
Difference image analysis (DIA)
5
5
###############################
6
6
7
-
.. important::
8
-
9
-
Prompt data processing is under construction.
10
-
11
-
12
7
DIA uses the decorrelated `Alard and Lupton (1998) <https://ui.adsabs.harvard.edu/abs/1998ApJ...503..325A/abstract>`_ image differencing algorithm, as described in "Implementation of Image Difference Decorrelation" (`dmtn-021.lsst.io <https://dmtn-021.lsst.io/>`_) and in the LSST Science Pipelines documentation for the `Alard-Lupton implementation <https://pipelines.lsst.io/modules/lsst.ip.diffim/AL_implementation.html>`_.
13
8
14
9
Key terminology:
@@ -32,52 +27,37 @@ DIA source detection refers to the process of finding regions with above-thresho
32
27
33
28
Difference images contain sources of residual flux from the subtraction, both positive and negative.
34
29
Real astrophysical transients, variables, and moving objects appear as sources with a flux that is equal to the difference between their flux in the template and visit images.
30
+
The ``scienceFlux`` column of the ``DiaSource`` catalog provides a forced PSF measurement on the direct image at the ``DiaSource`` position, which may be useful for scientists interested in variable objects with flux in the template.
35
31
36
-
Spurious sources are also present due to unavoidably imperfect template matching.
32
+
Spurious sources are also present due to unavoidably imperfect template matching as well as other image artifacts.
37
33
38
-
Deblending is not run on difference image sources.
34
+
Deblending is not currently run on difference image sources.
39
35
40
36
The results of DIA source detection are stored in the ``DiaSource`` catalog.
41
37
42
-
43
-
Reliability score
44
-
-----------------
45
-
46
-
Both real and spurious DIA sources can be detected with a signal-to-noise ratio of 5.
47
-
An estimate of how likely a source is to be real is provided to enable filtering of DIA sources by users.
48
-
49
38
Dipoles
50
39
-------
51
40
52
41
Sources with footprints containing both positive and negative peaks are fit with a dipole centroid code.
53
-
Dipoles represent sources that have moved only slightly (e.g., less than the PSF) between template and visit image.
54
-
55
-
Pixel flags
56
-
-----------
57
-
58
-
If the footprint contains one or more flagged pixels, e.g., for cosmic rays, detector edge, bad pixels, known artifacts, non-astrophysical trails, the source is also flagged.
59
-
60
-
61
-
DIA object association
62
-
======================
42
+
Dipoles represent sources that are offset slightly (e.g., less than the PSF) between template and visit image.
43
+
Note that for very bright sources, dipoles occur even with minor displacements and don't indicate problems with astrometric calibration.
63
44
64
-
DIA sources are associated into DIA objects using a radius of 1 arcsecond.
45
+
Trailed Sources
46
+
---------------
65
47
66
-
The DIA objects are stored in the ``DiaObject`` catalog.
48
+
Trailed source parameters are estimated from the second moments of each ``DiaSource``.
49
+
Users interested in trailed sources are encouraged to filter on ``trailLength/trailLengthErr`` or other suitable parameters.
67
50
51
+
Pixel flags
52
+
-----------
68
53
69
-
Variability characterization
70
-
----------------------------
71
-
72
-
Light curve characterization parameters (timeseries features) are calculated for all DIA objects.
73
-
In the future this set of parameters will extend to include measurements such as standard deviation, skew slope, percentiles, the ``StetsonJ`` parameter and more.
54
+
If the source detection footprint contains one or more flagged pixels, e.g., for cosmic rays, detector edge, bad pixels, known artifacts, non-astrophysical trails, appropriate boolean flag columns are set.
74
55
75
56
76
57
DIA forced photometry
77
58
=====================
78
59
79
60
In general, "forced" photometry means a measurement made at a fixed coordinate in an image, regardless of whether an above-threshold region was detected in that particular image.
80
61
81
-
Forced PSF photometry measurements are made on all visit and difference images at the locations of all DIA objects.
82
-
The results are stored in the ``ForcedSourceOnDiaObject`` catalog.
83
-
62
+
Forced PSF photometry measurements are made on all visit and difference images at the locations of DIA objects.
63
+
The results are stored in the ``DiaForcedSource`` catalog and included in alerts.
To reduce the proportion of spurious alerts sent to users, several layers of filtering are applied.
8
+
These will continue to be tuned over time as we obtain more on-sky data.
9
+
10
+
11
+
Filtering
12
+
=========
13
+
14
+
We filter artifacts at several stages.
15
+
16
+
At the difference image detection stage, we reject sources withe pixels off image; edge; interpolated bad, edge, no data, saturated, and saturated template center pixels; as well as those occuring in modeled diffraction spikes.
17
+
18
+
We later filter sources with cosmic rays in their center pixels as well as those with high template variance and those with negative direct image fluxes indicative of background subtraction problems.
19
+
20
+
21
+
Artificial Satellites
22
+
=====================
23
+
24
+
We aim to remove artifical satellites from the alert stream.
25
+
We use predicted positions of known satellites to filter ``DiaSources`` prior to alerts.
26
+
Trailed sources with motions faster than 10 deg/day are also filtered.
27
+
28
+
"Glint trails" due to rotating space junks are identified by linear fits to the detected ``DiaSources``.
29
+
Glints are not filtered, but are flagged with a ``glint_trail`` boolean.
30
+
31
+
32
+
Reliability
33
+
===========
34
+
35
+
A Convolutional Neural Network (CNN) was trained with a mixture of simulated data and human-labeled transients, variables, and moving objects.
36
+
It uses cutout science, template, and difference images to estimate whether a ``DiaSource`` is likely astrophysical.
37
+
The resulting ``reliability`` score ranges from 0-1, with larger scores corresponding to more likely real objects.
38
+
To minimize the number of false positives, we presently filter ``DiaSources`` with reliability scores less than 0.1.
39
+
Users may select alerts with higher reliability scores to maximize purity.
40
+
41
+
42
+
.. important::
43
+
44
+
Detailed performance information about the reliability model as well as suggested thresholds are forthcoming.
Papers detailing the design and performance of the Alert Production and Solar System pipelines are in preparation.
8
15
16
+
Pipeline Overview
17
+
=================
9
18
10
-
A high-level overview of the Prompt Processing steps which generated the data products.
19
+
A `pipeline definition <https://github.com/lsst-dm/prompt_processing/blob/main/pipelines/LSSTCam/ApPipe.yaml>`_ is deployed in Prompt Processing.
20
+
It imports pipelines from `ap_pipe <https://pipelines.lsst.io/v/weekly/modules/lsst.ap.pipe/index.html>`_.
21
+
The `pipeline yaml files <https://github.com/lsst/ap_pipe/tree/main/pipelines>`_ may be found in Github.
22
+
23
+
A graph of the pipeline stages and intermediate data products may be found `here <https://tigress-web.princeton.edu/~lkelvin/pipelines/current/ap_pipe/LSSTCam/ApPipe/pipeline_ap_pipe_LSSTCam_ApPipe.pdf>`_.
24
+
The resulting user-facing data products are described in the Data Products section.
11
25
12
-
All processing was done with the `LSST Science Pipelines <https://pipelines.lsst.io/>`_.
13
26
14
27
15
28
Change log
16
29
==========
17
30
18
31
Unlike with the annual data releases which are associated with a given version of the
19
-
LSST Science Pipelines, Prompt processing is constantly ongoing and updates happen
20
-
on-the-fly.
32
+
LSST Science Pipelines, Prompt Processing software payloads are updated frequently.
21
33
22
-
**Need to add link to a human-curated and auto-generated change log.**
34
+
The `pipelines changelog <https://lsst-dm.github.io/lsst_git_changelog/weekly/index.html>`_ shows recent changes to the Science Pipelines.
35
+
Note that recently merged features may not immediately be deployed in Prompt Processing. The deployed pipelines are typically updated at least weekly.
36
+
37
+
Prompt processing
38
+
=================
39
+
40
+
Describes the data processing framework.
41
+
42
+
.. toctree::
43
+
:maxdepth:1
44
+
:glob:
23
45
46
+
prompt-processing/index
24
47
25
48
Single frame
26
49
============
@@ -57,12 +80,32 @@ Runs image subtraction to generate difference images and associated detection ca
57
80
58
81
dia/index
59
82
83
+
Source association
84
+
==================
85
+
86
+
Associates detections with astrophysical objects.
87
+
88
+
.. toctree::
89
+
:maxdepth:1
90
+
:glob:
91
+
92
+
association/index
93
+
94
+
Filtering and Reliability Scoring
95
+
=================================
96
+
97
+
Removing false positives.
98
+
99
+
.. toctree::
100
+
:maxdepth:1
101
+
:glob:
60
102
103
+
filtering/index
61
104
62
105
Alert generation
63
106
================
64
107
65
-
Creating and distributing alert packets to brokers.
Copy file name to clipboardExpand all lines: docs/processing/moving/index.rst
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,12 +4,12 @@
4
4
Solar System processing (SSP)
5
5
#############################
6
6
7
-
.. important::
7
+
Solar System Processing includes both the association of known Solar System Objects (SSOs) during nighttime observations as well as the discovery of new SSOs by linking unassociated ``DiaSources`` during daytime processing.
8
+
All Rubin SSO observations and discoveries are reported to the Minor Planet Center (MPC).
9
+
8
10
9
-
Prompt data processing is under construction.
11
+
.. important::
10
12
13
+
More information about the SSP pipelines is forthcoming.
11
14
12
-
The Solar System processing (SSP) pipeline links together detections of new and known solar system objects,
13
-
and interfaces with the Minor Planet Center (MPC) for physical parameter derivation and orbit fitting.
Prompt Processing is the `pipeline execution framework <https://github.com/lsst-dm/prompt_processing>`_ which enables Rubin to process images and deliver alerts to community brokers with low latency.
8
+
Prompt Processing is implemented using the `KEDA <https://keda.sh/>`_ framework in Kubernetes.
9
+
It runs in the US Data Facility (USDF).
10
+
11
+
Prompt Processing analyzes each detector in the camera focal plane independently.
12
+
Processing begins when the system receives a ``nextVisit`` message from the telescope which specifies the identifier and expected pointing of the forthcoming observation.
13
+
In response, Prompt Processing starts a new pod for each detector it will need to process.
14
+
It also preloads reference catalogs, template images, catalog history, predicted positions of solar system objects, and other data so that processing is ready to begin.
15
+
16
+
Once the pixel data arrive, each Prompt Processing pod executes the subsequent steps of the Alert Production pipeline.
17
+
After the APDB is updated and alerts are transmitted, the pod executes any afterburner processing and transfers the data products to the central data butler.
Copy file name to clipboardExpand all lines: docs/processing/single-frame/index.rst
+2-6Lines changed: 2 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,9 +4,5 @@
4
4
Single-frame processing
5
5
#######################
6
6
7
-
.. important::
8
-
9
-
Prompt data processing is under construction.
10
-
11
-
12
-
This includes instrument signature removal (ISR) and calibration.
7
+
Single-frame processing reduces and calibrates the newly-obtained science (direct) image in preparation for image differencing.
8
+
It includes `instrument signature removal <https://pipelines.lsst.io/v/weekly/modules/lsst.ip.isr/index.html>`_ (ISR) and `astrometric and photomeric calibration <https://pipelines.lsst.io/v/weekly/modules/lsst.pipe.tasks/tasks/lsst.pipe.tasks.calibrateImage.CalibrateImageTask.html>`_.
Copy file name to clipboardExpand all lines: docs/processing/templates/index.rst
+21-13Lines changed: 21 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,9 +4,6 @@
4
4
Template generation
5
5
###################
6
6
7
-
.. important::
8
-
9
-
Prompt data processing is under construction.
10
7
11
8
12
9
Template coadded images are created by combining multiple calibrated images obtained in the same filter and of the same region of the sky.
@@ -15,9 +12,27 @@ This coaddition reduces noise, gaps, and artifacts.
15
12
Incremental Template Generation
16
13
===============================
17
14
15
+
In steady state operations, templates for alert production will be produced by the annual Data Releases.
16
+
17
+
During Early Operations prior to DR1, templates will be built incrementally when sufficient suitable images are available in a given area of the sky.
18
+
Once produced, templates will remain in place as long as possible to avoid repeated shifts of the difference light curves.
19
+
18
20
Template Availability
19
21
=====================
20
22
23
+
As of 2026 February, templates are deployed in the following discrete fields:
24
+
25
+
* COSMOS ``u``, ``g``, ``r``, ``i``, ``z``, ``y``
26
+
* ECDFS ``r``, ``i``, ``z``
27
+
* ELAISS1 ``g``, ``r``, ``i``, ``z``
28
+
* EDFS_A_B ``g``, ``r``, ``i``, ``z``
29
+
* XMM-LSS ``i``, ``z``
30
+
* M49 ``u``, ``g``, ``r``, ``i``
31
+
32
+
33
+
.. important::
34
+
35
+
Tools for visualizing the current template holdings are under development.
21
36
22
37
Coadds by patch
23
38
===============
@@ -35,9 +50,8 @@ Unobserved sections of the deep coadd image have pixels with no data flagged as
35
50
Input image selection
36
51
=====================
37
52
38
-
For template coadds, good seeing (low PSF FWHM) is more important than depth.
39
-
40
-
*More details to be added here.*
53
+
Image selection for coaddition must balance the desire for early templates (which necessarily have fewer images) with the need for good seeing, depth, artifact rejection, and spatial coverage to produce high-quality alerts.
54
+
Details of the image selection for incremental template building are still in development.
41
55
42
56
43
57
Image combination algorithm
@@ -47,10 +61,4 @@ A mean stacking algorithm, weighted by inverse variance, combines selected expos
47
61
48
62
To mitigate transient artifacts before coaddition, an artifact rejection procedure first identifies and masks features such as satellite trails, optical ghosts, and cosmic rays.
49
63
50
-
See "Coaddition Artifact Rejection and CompareWarp" (`dmtn-080.lsst.io <https://dmtn-080.lsst.io/>`_).
51
-
52
-
53
-
Coadd background subtraction
54
-
============================
55
-
56
-
After image coaddition, a constant background residual is fit and subtracted.
64
+
See "Coaddition Artifact Rejection and CompareWarp" (`dmtn-080.lsst.io <https://dmtn-080.lsst.io/>`_).
0 commit comments