Currently the image IDs that are pulled from the database are only subject to a single filter (filterImages; here). We should make it so that the list of images to be processed can be whittled down to fewer images.
The motivation behind this request is to accommodate times when a session has stopped processing for an extended period of time due to a system failure (e.g., image transfer didn't occur, dask-jobqueue misbehaved and futures weren't run). For a session during which data is actively being gathered, the most recently acquired images prior to noticing a processing issue are more important than images that were acquired hours earlier. From an operational perspective it makes the most sense to process images using a window that includes the newest images as they are acquired going forward plus a few older images at a time, with the order that older images are processed going from most recently to least recently acquired.
Currently the image IDs that are pulled from the database are only subject to a single filter (
filterImages; here). We should make it so that the list of images to be processed can be whittled down to fewer images.The motivation behind this request is to accommodate times when a session has stopped processing for an extended period of time due to a system failure (e.g., image transfer didn't occur,
dask-jobqueuemisbehaved and futures weren't run). For a session during which data is actively being gathered, the most recently acquired images prior to noticing a processing issue are more important than images that were acquired hours earlier. From an operational perspective it makes the most sense to process images using a window that includes the newest images as they are acquired going forward plus a few older images at a time, with the order that older images are processed going from most recently to least recently acquired.