Skip to content

Ghgc 632/scheduled worldview nrt collection update #378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
sanzog03 wants to merge 42 commits into
base: dev
Choose a base branch
from
Open

Ghgc 632/scheduled worldview nrt collection update #378

sanzog03 wants to merge 42 commits into from

Conversation

@sanzog03
Copy link
Contributor

@sanzog03 sanzog03 commented Jun 3, 2025 •
edited
Loading

Summary:

Addresses GHGC-632: Automated STAC update for the updated Worldview NRT datasets

Changes

  • Create a DAG capable of indexing WMTS into STAC.
  • Scheduler and gibs url are optional, but when provided, the task for scheduled update (necessary for NRT) will trigger based off gibs metadata. Collection Id should be common in STAC and GIBS.
  • There is validation task added for config json to check if it adheres to the WMTS extension specification.

PR Checklist

  • Ad-hoc testing - Deploy changes and test manually
  • Integration tests

kyle-lesinger
kyle-lesinger reviewed Jun 4, 2025
View reviewed changes
@sanzog03 sanzog03 requested a review from kyle-lesinger June 4, 2025 19:32
@sanzog03 sanzog03 requested a review from slesaad June 5, 2025 16:48
@slesaad
Copy link
Member

slesaad commented Jun 17, 2025

We want to make a generic dag and then put our specific dag config in the s3 bucket. This basically hardcodes the collection within the dag itself. Let's talk

@sanzog03
Copy link
Contributor Author

sanzog03 commented Jun 25, 2025

We want to make a generic dag and then put our specific dag config in the s3 bucket. This basically hardcodes the collection within the dag itself. Let's talk

The hardcoded aspects within the DAG itself are for its default settings. We then pull the unique DAG configurations from the S3 bucket, as you mentioned.

@sanzog03
take schedule from generate dag config. better namings
dd30c6b
@sanzog03
add ability to ingest wmts without gibs
260324f
@sanzog03
remove unwanted extensions from wmts config json
82987e9
@sanzog03
adhere to the return type
7e09e8b
@sanzog03
single task instantiation for whatever is necessary
e5b16e3
@sanzog03
refactor for better task arrangement and readability
c5208a7
@sanzog03
import error
fa67e1b
@sanzog03
1. made gibs url and schedule optional.
94ae24f 
2. fix to the task group call.
@sanzog03
wmts and gibs branching fix
0e16bb1
@sanzog03
rename to make the dag generic for all wmts and not just gibs wmts
d58dc2f
@sanzog03
update doc string
6cf67d9
@sanzog03
1. add validation task which validates if the json config is as per t…
2f0ffd9 
…he stac extension specs.

2. only necessary inputs for dag execution
3. change the config to adhere to the wmts extension specs
@sanzog03
remove comment
087b486
@sanzog03
corrected import for tests
f1fb195
slesaad
slesaad reviewed Dec 1, 2025
View reviewed changes
Copy link
Member

@slesaad slesaad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@anayeaye @smohiudd appreciate it if you guys could have a look too!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kyle-lesinger kyle-lesinger kyle-lesinger left review comments

@slesaad slesaad slesaad approved these changes

@paridhi-parajuli paridhi-parajuli Awaiting requested review from paridhi-parajuli

@amarouane-ABDELHAK amarouane-ABDELHAK Awaiting requested review from amarouane-ABDELHAK

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sanzog03 @slesaad @kyle-lesinger