emit-main/CHANGELOG.md at develop · emit-sds/emit-main

Run workflow updates by @adamchlus in #74
GHG metrics by @adamchlus in #75
Add frame_hdr_format to support enhanced cloud screening by @winstonolson in #76
Use Graph API for emit-sds mailbox by @winstonolson in #77

v1.6.28

14 August 2025

Sync isofit_3x with develop by @adamchlus in #63
Add Isofit 3x to minforge branch for testing by @winstonolson in #69
Use miniforge with conda-forge and upgrade to isofit 3.x by @winstonolson in #72

v1.6.27

22 July 2025

V1.6.26 hotfix (including calibration update) by @winstonolson in #70

v1.6.26

26 June 2025

Added removed GHG mosaic code by @adamchlus in #67

v1.6.25

24 June 2025

Ghg mosaic by @adamchlus in #65

v1.6.24

23 May 2025

Solar zenith and daynight updates by @adamchlus in #62

v1.6.23

5 May 2025

V1.6.22 patch by @winstonolson in #60

v1.6.22

2 April 2025

Julia 191 rollback by @pgbrodrick in #53
Merge main back into develop to resolve some commit issues by @winstonolson in #55
GHG and config updates by @adamchlus in #57
config change for tetracorder libraries - test only (not ops) by @pgbrodrick in #58

v1.6.21

12 December 2024

Julia 191 rollback #53
Prepare for v1.6.21 release. Add new task runtimes script. 0fe5082
Hotfix runtime bugs 1f288fb

v1.6.20

10 December 2024

Merge develop into main for v1.6.20 #52
small path modifications for new hardware #51
Config updates for migration #50
Use latest compile and plot metrics scripts. 194a5cb
Prep for 1.6.20 release - may need to change l1b config one more time 25e3730
Prepare for v1.6.20 release 8182b5f

v1.6.19

1 October 2024

Merge develop to main for v1.6.19 #49
Use S3 for delivery staging #48
Patch in updates from ops environment 264de52
Switch to S3 delivery 2f8a11f
Prepare for 1.6.19 c0d34a2

v1.6.18

1 September 2023

Merge develop into main for v1.6.18 #47
Prep for 1.6.18 release and clean up PEP8 8d3708f
Don't add ffupdate paths to rdn header. 0c242e0
Don't compress on rsync. Check BAD start/stop time. Use projection on ff_update query. Update status report. 9f5d64d

v1.6.17

27 June 2023

revise changelog #46
L1BATT library issue patch update #45
l2b and l3 software updates #44
workflow updates to harden l1b processing with destriping #43
L1b-Geo patch update (1.6.15) #42
Update for latest l2b patch fix #41
new update for v1.6.13 #40
Merge develop into main for v1.6.12 #38
Merge develop into main for v1.6.11 release #36
Merge develop into main for v1.6.10 #35
Merge develop into main for v1.6.9 #33
Merge develop into main for v1.6.8 #32
Merge develop into main for v1.6.7 #31
Merge develop into main for v1.6.6 #29
Merge develop into main for v1.6.5 #27
Merge develop into main for v1.6.4 #25
Merge develop into main for v1.6.3 #24
Merge develop into main for v1.6.2 #23
Merge develop into main for v1.6.1 #22
Merge develop into main for v1.6.0 #20
increment to version 1.6.17 1834f0e
small l2b updates (tar) and l3 updates (runcall and julia 1.9) a3657d1
update changelog 4278796

v1.6.16

8 June 2023

version number updates 9b23b59
l1b adjustment to ignore unprocessed l1a products during forward dark search ed9a7ff
update auto changelog f237b4f

v1.6.15

6 June 2023

L1b-Geo patch update (1.6.15) #42
updates to incrememnt l1b-geo build 1e5833e
update change log 12ecb79

v1.6.14

1 June 2023

Update for latest l2b patch fix #41
update ops and test configs and build version 511a6b5
update change log e75dbbf
increment to version 1.6.14 889c27c

v1.6.13

19 May 2023

new update for v1.6.13 #40
L2b updates #39
Add build 010613 that uses emit-isofit conda env for l2b. Pass emit_utils to l2b run commands. d79241c
Update abununcert hdr file with emit properties ffa197a
update change log bc38521

v1.6.12

16 May 2023

Merge develop into main for v1.6.12 #38
Daily flat field and many updates to monitors and scripts #37
Call new rdn wrapper script that executes flat field update and median e4c1e8e
Don't use singleton. Don't ingest 1675 in monitor, use new script. Retry L2B in reconciliation. 46a319f
Add l2b and l3 monitors. Also, add mask dependency in l2 monitor. 2ee9fd8

v1.6.11

26 April 2023

Merge develop into main for v1.6.11 release #36
Various patches and updates to reporting scripts. 0beba72
Add status report option and fixes for pp ingest and science ingest. 3def4df
Prep for 1.6.11 release 1d11b56

v1.6.10

17 February 2023

Merge develop into main for v1.6.10 #35
Track build and delivery versions separately in delivered products #34
Add new Python ingest science data script 1a6d1cf
Prep for 1.6.10 release ebb7f13
Track build and delivery versions separately in delivered products. ae95dee

v1.6.9

25 January 2023

Merge develop into main for v1.6.9 #33
Prep for v1.6.9 75c60cd
Require all frames for reassembly. Remove milliseconds on ingest again. 9822e2e
Update change log 581ac8c

v1.6.8

12 January 2023

Merge develop into main for v1.6.8 #32
Prep for 1.6.8 4294d65
Update geo to 1.6.7 and support two PNGs (proj vs not) 6d9dea8
Update change log a5af45a

v1.6.7

6 January 2023

Merge develop into main for v1.6.7 #31
Reconciliation report v2 #30
Add new monitor to process reconciliation responses af17ef1
Add support for L2A deliveries and extend all L1A delivery updates to other collections. 3d0160c
Prep for v1.6.7 d8605aa

v1.6.6

15 December 2022

Merge develop into main for v1.6.6 #29
L1A delivery updates and reconciliation report draft #28
Add ReconciliationReport task. b0f046e
Prep for 1.6.6 9cee802
Remove dependencies on higher level processing for L1ADeliver e1eafb5

v1.6.5

9 December 2022

Merge develop into main for v1.6.5 #27
Ioc delivery #26
Add monitors for assigning DAAC scene nums and L0 and L1A delivery. a85d8a4
Add script to calculate runtimes from slurm log. 6b1cbdf
Prep for 1.6.5 17fc363

v1.6.4

17 November 2022

Merge develop into main for v1.6.4 #25
Prepare for 1.6.4 release 9865eb1
Update change log 62e2d53

v1.6.3

11 November 2022

Merge develop into main for v1.6.3 #24
Prepare for 1.6.3 release. 41765a5
Update change log c379413

v1.6.2

10 November 2022

Merge develop into main for v1.6.2 #23
Update 1675 ingest script and prepare for 1.6.2. e8cd056
Update change log 1acb967
add bandmask argument to calibration destriping e04b722

v1.6.1

7 November 2022

Merge develop into main for v1.6.1 #22
Aoe updates #21
Support forward/backward DAAC queues d3ce6ce
Add build 010601 and update test/ops configs b12a51a
Add FAP script 5cf7d62

v1.6.0

17 October 2022

Merge develop into main for v1.6.0 #20
Rerun updates #19
UMMG Updates #18
Update ioc-ummg branch with latest develop changes #17
IOC Misc Fixes #16
Delete before copy #15
Merge IOC hotfixes into develop #14
Refl updates #13
Daily report #12
Merge develop into main for v1.5.2 #11
Merge develop into main for v1.5.1 #10
Merge develop into main for v1.5.0 #8
Merge develop to main for v1.4.0 #4
Add solar zenith and azimuth and gring in L1B geo. Also, add in new DAAC Helper tasks. 7bc4eea
Move scene generator to L1A 3bc0c49
Add reassembly metrics. d80ed73

v1.5.2

27 July 2022

Prep for v1.5.2 2901c23
handle merge conflict fa5f99c
update configs for tetracorder5.27a and newest unmixing 0cb3cbf

v1.5.1

27 July 2022

Merge develop into main for v1.5.1 #10
Planning product ingest and orbit update #9
Update planning product to delete any previous overlapping orbits or data collections. First, check if any of them have products, and if so, then abort. d9addc1
Update filenames to use orbit.short_oid or acquisition.short_orb, including DAAC delivery files. Use '0000000' for default orbit. f627d80
Prep for v1.5.1 76d39c8

v1.5.0

22 July 2022

Merge develop into main for v1.5.0 #8
Quicklooks #7
Pre ioc #6
Failed monitor tasks #5
Add --date_field and --retry_failed flags. The date field tells the monitor which date field to search (last_modified, start_time, etc.). The retry failed flag forces the monitor to retry a task even if it failed before. ac540c4
Add build 010500 and update config files. 2b68c9e
Update task sizing. Use memory = 18000 as the minimum for all small tasks. Larger tasks are apportioned according to their need. 00ec7f2

v1.4.0

6 June 2022

Merge develop to main for v1.4.0 #4
Stress test updates #3
Check number of valid lines and instrument mode #2
Ummg updates #1
l3 call updates #65
Skip cloudy acquisitions #66
Update change log 5d9170a
Add cron script to rclone backup data products. 41485be
update unmixing call and library reference, also julia settings 0dbe8bf

v1.3.0

4 May 2022

Merge develop to main for v1.3.0 #63
Planning product and bad updates for long duration tests #62
Add code to check for start/stop of BAD file before inserting into DB. ac80098
Add 010300 build and update test and ops to use it. Use tvac4_config.json for l1b config template. 611846b
Update sftp script to only use rpsm check on hosc files, not bad files. 49d7e77

v1.2.0

18 March 2022

Merge develop to main for v1.2.0 #61
L1 edp addl prods #60
Ccsds updates #59
Use /beegfs/store/shared instead of /shared. c655e77
Update L1AReformatEDP to allow for multiple engineering products. Use 1676 stream with corresponding start time as ancillary input file. Call edp script in emit-ios directory instead of using the emit-sds-l1a run script. 9cbdd1e
Update L0 HOSC task to insert stream in DB after getting start/stop time in UTC from CCSDS packets. Rename CCSDS file using this UTC timestamp so that it matches DB start time. f2a7257

v1.1.0

28 February 2022

Merge develop to main for v1.1.0 #58
Add build 010100 and update test/ops configs to reference it. 785a8ae
Update changelog. 282e7e1
Update version to 1.1.0 70cb078

v1.0.0

9 February 2022

Merge develop to main for 1.0.0 #57
Release 3 prep v2 #56
Add build 010000 and update test config to use it. bd76141
Add task dependencies for all deliver tasks. 7a865c3
Add L2BAbundance to l2 monitor and update test and ops configs for deployment. aa79417

v0.6.0

31 January 2022

Merge develop into main for v0.6.0 #55
Release 3 prep (v1) #54
Integrate geo #53
Ummg file creation. #52
Merge develop for release 0.5.0 #51
Merge develop for v0.4 #42
Merge develop for version 0.3 #41
Add L2BDeliver with l2bdaac product choice. Remove umm-g from L2BFormat. e46bf23
Add L1BAttDeliver task and update orbit class to support it. 6cf976f
add license f629e77

v0.5.0

20 January 2022

BAD CSV Convertor, L1B Cal Integration, and DAAC Delivery Updates #50
Orbit monitor #49
TVAC 4 Prep and Misc Updates #48
Daac interface #47
Bad reformatting #46
Ncdf integration #45
Integrate raw line timestamps product into ReassembleRaw task #44
Split acquisitions #43
Add working L0IngestBAD task and not-yet working L1AReformatBAD task. Add 'bad' as new apid in db and check for .sto files when doing inserts and updates. Add support for Orbit class in workflow manager. a0af96e
Begin integration of splitting acquisitions. Not yet complete 72f5b80
Add Orbit class to track orbits. Overhaul L0ProcessPlanningProduct to read in JSON files that also include orbit information in addition to DCIDs. Change dcid symlink on depacketization task. f8e4909

v0.4

26 October 2021

Merge develop for v0.4 #42
Create build_0004 which includes v0.5.6 of emit-ios code 314d809

v0.3

25 October 2021

Merge develop for version 0.3 #41
Dcid updates #40
Tvac3 prep #39
L3a #38
Add DataCollection class. Add data_collection queries to DatabaseManager. Modify change ownership to check that target gid is different from current. a28b815
Import WorkflowManager into other classes inline in order to use helper functions like wm.makedirs and wm.symlink. For data collection class, create two folder structures (by_dcid, and by_date) - the second one is only created when there is a start_time property (which will come from planning product). 29142be
Add data_collection to workflow manager. Update Depacketize task to use data collection objects and paths instead of acquisition. fa0cbbb

v0.2

31 August 2021

Merge develop into main for SDS Release 2 #37
Update l0 calling structure and missing packet thresh #36
Rename CCSDS files #35
Group permissions #34
Failure notification email and DB authentication #33
File monitor update #32
Check Frame Report #31
Rawqa #30
Support L1A EDD Updates #29
Misc fixes, including UTC timestamps #28
Resource management updates #27
Update workflow code to work with latest frame header changes #26
Config updates #25
Integrate l1a #24
Separate build config into build specific config files to be read in by parent config file. Create Config class to read in configs and handle new functionality like date specific ancillary file paths. 859eb17
Add self.config to WorkflowManager, DatabaseManager, Acquisition, Stream, and FileMonitor to hold config fields instead of using self.dict.update in order to manage these properties better. Update environment.yml to use pip install for tensorflow==2.0.1 since conda install was failing locally. a2bb939
Add L0ProcessObservationsProduct to insert DCID and other planned observation parameters into DB. Add/Update L1AReadScienceFrames as a working draft. Include db methods to insert and update observations but this will likely be removed. fddbb81

v0.1

26 January 2021

Merge develop into main for Release 1 #23
Integrate l2b #22
Update l2a #21
Update L1B #20
Additional updates to merge add-pytest #19
Minor tweaks to fix add-pytest merge #18
Add pytest #17
Misc updates #16
Integrate L1B #15
File monitor #14
Integrate l0 and l1a engineering pges #6
Task completion #5
Packaging overhaul #4
L2a draft #3
FileManager update #2
Conda envs #1
Adds basic framework for running workflows including tasks separated by level, a file manager class to handle paths, and ENVITarget class, and some basic unit tests e17f6cd
Add l2amask task 7e66aa0
Move ExampleTask to test_tasks.py because there were issues with loading test_luigi module by slurm_runner. Convert --config_path to abspath in run_workflow and run_file_monitor. Improve references to logging.conf so that run scripts can be called from any directory. 0bbe8af

Uh oh!

FilesExpand file tree

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Changelog

v0.1