Code

dev to main BLOCKED by issue #252#245

Code

Open

smohiudd wants to merge 218 commits intomainNASA-IMPACT/veda-data-airflow:mainfrom devNASA-IMPACT/veda-data-airflow:devCopy head branch name to clipboard

Commits

Commits on Jul 19, 2024

• WIP: dynamic ingest mapping for continuous tracing

  ividito

  committed
  c1d67da

  Copy full SHA for c1d67da

Commits on Jul 31, 2024

• feat: use taskflow for concurrent task runs

  ividito

  committed
  55744c6

  Copy full SHA for 55744c6

• clean up debug code

  ividito

  committed
  ee98cf7

  Copy full SHA for ee98cf7

Commits on Aug 1, 2024

• Bugfix vector ingest

  ividito

  committed
  6e151a9

  Copy full SHA for 6e151a9

Commits on Aug 20, 2024

• merge dev

  smohiudd

  committed
  82c724f

  Copy full SHA for 82c724f

• add generic vector pipeline

  smohiudd

  committed
  54b885d

  Copy full SHA for 54b885d

• remove process generic vector pipeline

  smohiudd

  committed
  a7b51e1

  Copy full SHA for a7b51e1

Commits on Aug 27, 2024

• Update dags/veda_data_pipeline/veda_generic_vector_pipeline.py

  ividito

  authored
  318d39a

  Copy full SHA for 318d39a

Commits on Sep 9, 2024

• Restructure dataset get_files output, add retries to some tasks to avoid SIGKILL errors

  ividito

  committed
  0ab867b

  Copy full SHA for 0ab867b

Commits on Oct 4, 2024

• Changes to make vector ingest work in shared VPC environment

  ividito

  committed
  f1411fe

  Copy full SHA for f1411fe

Commits on Oct 16, 2024

• added the statistics calculations in pipeline

  

  vishal

  committed
  b6b4b5e

  Copy full SHA for b6b4b5e

• generating JSON file for the stats and uploading it to S3

  

  vishal

  committed
  42e88d7

  Copy full SHA for 42e88d7

Commits on Oct 17, 2024

• using temp file for JSON

  

  vishal

  committed
  a5dcb51

  Copy full SHA for a5dcb51

Commits on Oct 18, 2024

• feat:port event-driven vector automation to terraform

  ividito

  committed
  4db9b2c

  Copy full SHA for 4db9b2c

• fix: feature flag for vector automation lambda

  ividito

  committed
  ebb9993

  Copy full SHA for ebb9993

• fix: adjust log group count

  ividito

  committed
  4eca40e

  Copy full SHA for 4eca40e

• testing JSON update for multiple datasets

  

  vishal

  committed
  f98add0

  Copy full SHA for f98add0

• fix tf vector automation vars

  smohiudd

  committed
  0005892

  Copy full SHA for 0005892

• update vars in event bridge tf

  smohiudd

  committed
  38bff94

  Copy full SHA for 38bff94

• Merge pull request #239 from NASA-IMPACT/fix/vector-automation

  Show description for 90aed8f
  smohiudd

  authored
  90aed8f

  Copy full SHA for 90aed8f

• add new line

  smohiudd

  committed
  021f4ff

  Copy full SHA for 021f4ff

• Merge pull request #240 from NASA-IMPACT/fix/vector-automation

  Show description for 8ec0cd4
  smohiudd

  authored
  8ec0cd4

  Copy full SHA for 8ec0cd4

• tested the creation of JSON file for different datasets

  

  vishal

  committed
  5678be5

  Copy full SHA for 5678be5

• fix tag filter

  smohiudd

  committed
  abba102

  Copy full SHA for abba102

• Merge pull request #241 from NASA-IMPACT/fix/vector-automation

  Show description for 006e989
  smohiudd

  authored
  006e989

  Copy full SHA for 006e989

• update vector subnet ref

  smohiudd

  committed
  f181ca7

  Copy full SHA for f181ca7

• Merge pull request #242 from NASA-IMPACT/fix/vector-automation

  Show description for 39401a0
  smohiudd

  authored
  39401a0

  Copy full SHA for 39401a0

• fix vector subnet ids

  smohiudd

  committed
  857e2fa

  Copy full SHA for 857e2fa

• Merge pull request #243 from NASA-IMPACT/fix/vector-automation

  Show description for 9b43b89
  smohiudd

  authored
  9b43b89

  Copy full SHA for 9b43b89

Commits on Oct 21, 2024

• conditional vector subnets

  smohiudd

  committed
  e3870e7

  Copy full SHA for e3870e7

• feat: disable default api gateway endpoint for workflows api

  botanical

  committed
  266be89

  Copy full SHA for 266be89

• Merge pull request #246 from NASA-IMPACT/fix/vector-ingest-subnets

  Show description for 65f0a8c
  smohiudd

  authored
  65f0a8c

  Copy full SHA for 65f0a8c

• feat: add variable to configure disabling default apigw endpoint

  botanical

  committed
  d4a57e3

  Copy full SHA for d4a57e3

Commits on Oct 22, 2024

• Merge pull request #197 from NASA-IMPACT/fix/asset-handling

  Show description for b9a9323
  smohiudd

  authored
  b9a9323

  Copy full SHA for b9a9323

Commits on Oct 23, 2024

• Update infrastructure/terraform.tfvars.tmpl

  botanical

  authored
  d77d946

  Copy full SHA for d77d946

• Update infrastructure/terraform.tfvars.tmpl

  botanical

  authored
  7af892a

  Copy full SHA for 7af892a

Commits on Oct 24, 2024

• Using pythonoperators instead of ECS operator

  amarouane-ABDELHAK

  committed
  6d3e29a

  Copy full SHA for 6d3e29a

• Merge pull request #247 from NASA-IMPACT/jt/issue-452-disable-default-endpoint

  Show description for 61a797d
  botanical

  authored
  61a797d

  Copy full SHA for 61a797d

• Switching to pythonOperator

  amarouane-ABDELHAK

  committed
  686c4c8

  Copy full SHA for 686c4c8

Commits on Oct 25, 2024

• Merge pull request #249 from NASA-IMPACT/feature/use-pythonoperators

  Show description for c190564
  amarouane-ABDELHAK

  authored
  c190564

  Copy full SHA for c190564

• test: temporarily use test buckets

  anayeaye

  committed
  305dd43

  Copy full SHA for 305dd43

Commits on Oct 28, 2024

• feat:port event-driven vector automation to terraform

  

  ividito

  authored and

  vishal

  committed
  188f96e

  Copy full SHA for 188f96e

• fix: feature flag for vector automation lambda

  

  ividito

  authored and

  vishal

  committed
  a2e3bad

  Copy full SHA for a2e3bad

• fix: adjust log group count

  

  ividito

  authored and

  vishal

  committed
  12ba68c

  Copy full SHA for 12ba68c

• fix tf vector automation vars

  

  smohiudd

  authored and

  vishal

  committed
  f532a6f

  Copy full SHA for f532a6f

• update vars in event bridge tf

  

  smohiudd

  authored and

  vishal

  committed
  89d5251

  Copy full SHA for 89d5251

• add new line

  

  smohiudd

  authored and

  vishal

  committed
  9c636e0

  Copy full SHA for 9c636e0

• fix tag filter

  

  smohiudd

  authored and

  vishal

  committed
  47b0707

  Copy full SHA for 47b0707

• update vector subnet ref

  

  smohiudd

  authored and

  vishal

  committed
  8169ab6

  Copy full SHA for 8169ab6

• fix vector subnet ids

  

  smohiudd

  authored and

  vishal

  committed
  66cfb79

  Copy full SHA for 66cfb79

• conditional vector subnets

  

  smohiudd

  authored and

  vishal

  committed
  c449cb9

  Copy full SHA for c449cb9

• WIP: dynamic ingest mapping for continuous tracing

  

  ividito

  authored and

  vishal

  committed
  0df795e

  Copy full SHA for 0df795e

• feat: use taskflow for concurrent task runs

  

  ividito

  authored and

  vishal

  committed
  bfce3d5

  Copy full SHA for bfce3d5

• clean up debug code

  

  ividito

  authored and

  vishal

  committed
  31a3e84

  Copy full SHA for 31a3e84

• Bugfix vector ingest

  

  ividito

  authored and

  vishal

  committed
  a811fe2

  Copy full SHA for a811fe2

• add generic vector pipeline

  

  smohiudd

  authored and

  vishal

  committed
  3c034ae

  Copy full SHA for 3c034ae

• remove process generic vector pipeline

  

  smohiudd

  authored and

  vishal

  committed
  4a5dc02

  Copy full SHA for 4a5dc02

• Update dags/veda_data_pipeline/veda_generic_vector_pipeline.py

  

  ividito

  authored and

  vishal

  committed
  de8b69d

  Copy full SHA for de8b69d

• Restructure dataset get_files output, add retries to some tasks to avoid SIGKILL errors

  

  ividito

  authored and

  vishal

  committed
  d28432d

  Copy full SHA for d28432d

• Changes to make vector ingest work in shared VPC environment

  

  ividito

  authored and

  vishal

  committed
  26feccf

  Copy full SHA for 26feccf

• feat: disable default api gateway endpoint for workflows api

  

  botanical

  authored and

  vishal

  committed
  18470d0

  Copy full SHA for 18470d0

• feat: add variable to configure disabling default apigw endpoint

  

  botanical

  authored and

  vishal

  committed
  919e04b

  Copy full SHA for 919e04b

• Update infrastructure/terraform.tfvars.tmpl

  

  botanical

  authored and

  vishal

  committed
  81f94de

  Copy full SHA for 81f94de

• Update infrastructure/terraform.tfvars.tmpl

  

  botanical

  authored and

  vishal

  committed
  7485696

  Copy full SHA for 7485696

• Using pythonoperators instead of ECS operator

  

  amarouane-ABDELHAK

  authored and

  vishal

  committed
  75fef32

  Copy full SHA for 75fef32

• Switching to pythonOperator

  

  amarouane-ABDELHAK

  authored and

  vishal

  committed
  6f080a7

  Copy full SHA for 6f080a7

• tested the creation of JSON file for different datasets

  

  vishal

  committed
  dafb213

  Copy full SHA for dafb213

• optimizing the automation

  

  vishal

  committed
  884c296

  Copy full SHA for 884c296

Commits on Oct 29, 2024

• Merge branch 'dev' into feature/optimize_automated_cog_transformation

  

  vishal

  committed
  05cd27d

  Copy full SHA for 05cd27d

• testing the optimized approach for automation

  

  vishal

  committed
  8d64aaa

  Copy full SHA for 8d64aaa

• successfully tested the automation for gridded population dataset

  

  vishal

  committed
  3ada005

  Copy full SHA for 3ada005

• Add S3 event bridge

  amarouane-ABDELHAK

  committed
  a619098

  Copy full SHA for a619098

• Add S3 event bridge

  amarouane-ABDELHAK

  committed
  4c9753f

  Copy full SHA for 4c9753f

• Add S3 event bridge

  amarouane-ABDELHAK

  committed
  59e1938

  Copy full SHA for 59e1938

Commits on Oct 30, 2024

• Add env deployment example

  amarouane-ABDELHAK

  committed
  685e7db

  Copy full SHA for 685e7db

• Merge pull request #251 from NASA-IMPACT/feat/use-sm2a-in-eventbrige-lambda

  Show description for dd0ea9e
  amarouane-ABDELHAK

  authored
  dd0ea9e

  Copy full SHA for dd0ea9e

Commits on Nov 4, 2024

• updated readme for the folder and resolved the failure bug

  

  vishal

  committed
  a64099a

  Copy full SHA for a64099a

Commits on Nov 6, 2024

• only execute deploy action on push to dev branch

  anayeaye

  committed
  8a3c584

  Copy full SHA for 8a3c584

• Merge pull request #254 from NASA-IMPACT/ci/remove-staging-action

  Show description for 4a277d3
  anayeaye

  authored
  4a277d3

  Copy full SHA for 4a277d3

• ci: do not automatically deploy mwaa to any environment

  anayeaye

  committed
  db73561

  Copy full SHA for db73561

Commits on Nov 8, 2024

• Merge pull request #255 from NASA-IMPACT/ci/remove-all-mwaa-deploy-actions

  Show description for c450be2
  anayeaye

  authored
  c450be2

  Copy full SHA for c450be2

Commits on Nov 13, 2024

• Adding condition on s3 event bridge

  amarouane-ABDELHAK

  committed
  b258d41

  Copy full SHA for b258d41

Commits on Nov 14, 2024

• Adding condition on s3 event bridge

  amarouane-ABDELHAK

  committed
  eb5e32c

  Copy full SHA for eb5e32c

• Fix some missing variables

  amarouane-ABDELHAK

  committed
  4909f43

  Copy full SHA for 4909f43

Commits on Nov 15, 2024

• Remove .env file

  amarouane-ABDELHAK

  committed
  87a08e2

  Copy full SHA for 87a08e2

• Remove .env file

  amarouane-ABDELHAK

  committed
  cf19bd3

  Copy full SHA for cf19bd3

• Remove the airflow config to be created by CICD

  amarouane-ABDELHAK

  committed
  1335a92

  Copy full SHA for 1335a92

Commits on Nov 21, 2024

• print all transfer exceptions

  anayeaye

  committed
  dbcd631

  Copy full SHA for dbcd631

Commits on Nov 22, 2024

• Merge pull request #257 from NASA-IMPACT/feat/use-sm2a-in-eventbrige-lambda

  Show description for 32b2acb
  amarouane-ABDELHAK

  authored
  32b2acb

  Copy full SHA for 32b2acb

• Merge branch 'dev' into fix/reveal-transfer-exception

  anayeaye

  committed
  7872364

  Copy full SHA for 7872364

• print client err on failed transfer

  anayeaye

  committed
  4374f34

  Copy full SHA for 4374f34

• raise transfer exception

  anayeaye

  committed
  091bc80

  Copy full SHA for 091bc80

• update secret variable type to string and add assume role arns to airflow_custom_variables"

  anayeaye

  committed
  4f14c96

  Copy full SHA for 4f14c96

• Merge pull request #260 from NASA-IMPACT/fix/reveal-transfer-exception

  Show description for ef7be91
  anayeaye

  authored
  ef7be91

  Copy full SHA for ef7be91

Commits on Nov 25, 2024

• fix(dockerfile) remove pypgstac install

  anayeaye

  committed
  fe09938

  Copy full SHA for fe09938

• fix(dag-requirements) remove pypgstac install

  anayeaye

  committed
  56738e9

  Copy full SHA for 56738e9

• Merge pull request #262 from NASA-IMPACT/fix/remove-pypgstac

  Show description for 66193a8
  anayeaye

  authored
  66193a8

  Copy full SHA for 66193a8

• let ingest api handle datetime validation and just pass through inputs in collection generation dag

  anayeaye

  committed
  4fbd229

  Copy full SHA for 4fbd229

Commits on Nov 26, 2024

• Merge pull request #263 from NASA-IMPACT/fix/collection-datetime-format

  Show description for d9799cc
  anayeaye

  authored
  d9799cc

  Copy full SHA for d9799cc

• temp print statement

  smohiudd

  committed
  926be29

  Copy full SHA for 926be29

• Merge pull request #264 from NASA-IMPACT/fix/collection-print

  Show description for 7e05d2f
  smohiudd

  authored
  7e05d2f

  Copy full SHA for 7e05d2f

• remove comma

  smohiudd

  committed
  2974701

  Copy full SHA for 2974701

• Merge pull request #265 from NASA-IMPACT/fix/collection-print

  Show description for 5d8757e
  smohiudd

  authored
  5d8757e

  Copy full SHA for 5d8757e

Commits on Dec 3, 2024

• Strip thumbnail assets from payload before discovery

  ividito

  committed
  64683af

  Copy full SHA for 64683af

• rename var

  ividito

  committed
  9d7c611

  Copy full SHA for 9d7c611

• Update dags/veda_data_pipeline/veda_dataset_pipeline.py

  Show description for 3a48975
  

  ividito

  and

  anayeaye

  committed
  3a48975

  Copy full SHA for 3a48975

• Fix callable

  ividito

  committed
  0e5cc7c

  Copy full SHA for 0e5cc7c

Commits on Dec 6, 2024

• Refactor Dataset Pipeline

  amarouane-ABDELHAK

  committed
  53416d0

  Copy full SHA for 53416d0

• Refactor Dataset Pipeline

  amarouane-ABDELHAK

  committed
  5a41003

  Copy full SHA for 5a41003

• Return status of the ingest

  amarouane-ABDELHAK

  committed
  cfb2bf2

  Copy full SHA for cfb2bf2

Commits on Dec 11, 2024

• Update dags/veda_data_pipeline/groups/discover_group.py

  Show description for 4e0416b
  

  amarouane-ABDELHAK

  and

  botanical

  authored
  4e0416b

  Copy full SHA for 4e0416b

• Adding aut ingest

  amarouane-ABDELHAK

  committed
  534e571

  Copy full SHA for 534e571

• Fix the bug of updating a pointer

  amarouane-ABDELHAK

  committed
  000a38c

  Copy full SHA for 000a38c

• Merge branch 'refactor-dataset-pipeline' of https://github.com/NASA-IMPACT/veda-data-airflow into refactor-dataset-pipeline

  amarouane-ABDELHAK

  committed
  64bea83

  Copy full SHA for 64bea83

• Fix updating the copy of config instead of the pointer

  amarouane-ABDELHAK

  committed
  74573b7

  Copy full SHA for 74573b7

• Merge pull request #268 from NASA-IMPACT/refactor-dataset-pipeline

  Show description for e6ca7d4
  amarouane-ABDELHAK

  authored
  e6ca7d4

  Copy full SHA for e6ca7d4

• Fixing automation DAG

  amarouane-ABDELHAK

  committed
  da43490

  Copy full SHA for da43490

• Fixing automation DAG

  amarouane-ABDELHAK

  committed
  ac3d475

  Copy full SHA for ac3d475

• Fixing automation DAG

  amarouane-ABDELHAK

  committed
  b584575

  Copy full SHA for b584575

• Add slack notification libraries

  amarouane-ABDELHAK

  committed
  487f225

  Copy full SHA for 487f225

• Add slack notification libraries

  amarouane-ABDELHAK

  committed
  82f1109

  Copy full SHA for 82f1109

• Merge pull request #236 from NASA-IMPACT/feature/optimize_automated_cog_transformation

  Show description for 4fdf934
  amarouane-ABDELHAK

  authored
  4fdf934

  Copy full SHA for 4fdf934

Commits on Dec 12, 2024

• Update SM2A workers python version to 3.11

  amarouane-ABDELHAK

  committed
  603a713

  Copy full SHA for 603a713

• Update the CMD for the workers

  amarouane-ABDELHAK

  committed
  45175e0

  Copy full SHA for 45175e0

Commits on Dec 16, 2024

• Merge pull request #271 from NASA-IMPACT/fix/update-workers-python-version

  Show description for b5b02be
  amarouane-ABDELHAK

  authored
  b5b02be

  Copy full SHA for b5b02be

Commits on Dec 20, 2024

• Add placeholder transfer to dataset DAG (no-op for now, just testing control flow)

  Show description for 9a8d9d2
  ividito

  committed
  9a8d9d2

  Copy full SHA for 9a8d9d2

• More makefile adjustments, adjust control flow

  ividito

  committed
  95fe45a

  Copy full SHA for 95fe45a

• Reorder transfer <> discovery, change transfer task input

  ividito

  committed
  caad8ba

  Copy full SHA for caad8ba

• Break out promotion to new DAG

  ividito

  committed
  4a5f166

  Copy full SHA for 4a5f166

• Remove transfer flag

  ividito

  committed
  829fa24

  Copy full SHA for 829fa24

• Light refactors to shared tasks, fixes to align promotion to dataset refactors

  ividito

  committed
  dee4542

  Copy full SHA for dee4542

• set transfer dry run default back to false

  ividito

  committed
  46e66e6

  Copy full SHA for 46e66e6

• Fix makefile, fix ti KeyError

  ividito

  committed
  da0650c

  Copy full SHA for da0650c

• regex updated

  siddharth0248

  committed
  2f39ead

  Copy full SHA for 2f39ead

• Merge pull request #274 from NASA-IMPACT/regex/sid

  Show description for c22f94c
  siddharth0248

  authored
  c22f94c

  Copy full SHA for c22f94c

Commits on Jan 15, 2025

• fix(transfer): add non-decorated shared transfer function

  ividito

  committed
  f82f792

  Copy full SHA for f82f792

• Remove defunct comments

  ividito

  committed
  37eb592

  Copy full SHA for 37eb592

Commits on Jan 16, 2025

• Terraform lock update

  ividito

  committed
  098b897

  Copy full SHA for 098b897

• Update terraform actrion version

  ividito

  committed
  771e582

  Copy full SHA for 771e582

Commits on Jan 20, 2025

• fix(submit-stac): modify retry behavior for API submission

  ividito

  committed
  2930826

  Copy full SHA for 2930826

Commits on Jan 22, 2025

• fix(build_stac): Use airflow vars instead of env var for build stac role

  ividito

  committed
  7a4d6fc

  Copy full SHA for 7a4d6fc

• oops

  ividito

  committed
  e3a7f09

  Copy full SHA for e3a7f09

• Missing Variable import

  ividito

  committed
  b31c36b

  Copy full SHA for b31c36b

Commits on Jan 27, 2025

• Add .airflowignore

  ividito

  committed
  4c3a783

  Copy full SHA for 4c3a783

Commits on Jan 28, 2025

• fix: add pystac stac version override

  botanical

  committed
  d2d7a25

  Copy full SHA for d2d7a25

Commits on Jan 29, 2025

• fix: change condition in remove_thumnail_asset to resolve key error for assets

  botanical

  committed
  060488d

  Copy full SHA for 060488d

• fix: update changelog

  botanical

  committed
  2075a0a

  Copy full SHA for 2075a0a

• fix: revert changelog changes

  botanical

  committed
  dd6216d

  Copy full SHA for dd6216d

• Merge pull request #292 from NASA-IMPACT/issue-291-support-stac-v1

  Show description for 80c9437
  botanical

  authored
  80c9437

  Copy full SHA for 80c9437

Commits on Feb 4, 2025

• Automate the transformation

  amarouane-ABDELHAK

  committed
  009ef44

  Copy full SHA for 009ef44

• Fix check function

  amarouane-ABDELHAK

  committed
  31fbf66

  Copy full SHA for 31fbf66

• fix base URL

  amarouane-ABDELHAK

  committed
  b8c220d

  Copy full SHA for b8c220d

• revert base URL

  amarouane-ABDELHAK

  committed
  73ee9ef

  Copy full SHA for 73ee9ef

• Updating a variable name

  amarouane-ABDELHAK

  committed
  35c536f

  Copy full SHA for 35c536f

Commits on Feb 5, 2025

• add raw data filter regex and task

  paridhi-parajuli

  committed
  7035d49

  Copy full SHA for 7035d49

• add config param as str

  paridhi-parajuli

  committed
  8864481

  Copy full SHA for 8864481

• increase runs per dag

  paridhi-parajuli

  committed
  155dba8

  Copy full SHA for 155dba8

Commits on Feb 11, 2025

• 1. Combine filter task to discover files task

  Show description for 98d31aa
  paridhi-parajuli

  committed
  98d31aa

  Copy full SHA for 98d31aa

Commits on Feb 12, 2025

• fix: update to fail safely instead

  botanical

  committed
  1109fc3

  Copy full SHA for 1109fc3

Commits on Feb 13, 2025

• Merge pull request #293 from NASA-IMPACT/issue-287/resolve-key-error-assets

  Show description for c4d1959
  botanical

  authored
  c4d1959

  Copy full SHA for c4d1959

• resolve comment

  paridhi-parajuli

  committed
  9aa1431

  Copy full SHA for 9aa1431

Commits on Feb 17, 2025

• add new line

  paridhi-parajuli

  committed
  a6a57df

  Copy full SHA for a6a57df

• make the max instances per dag configurable

  paridhi-parajuli

  committed
  3840a2e

  Copy full SHA for 3840a2e

• add changes to temp file handling

  paridhi-parajuli

  committed
  f8855e2

  Copy full SHA for f8855e2

• Merge pull request #294 from NASA-IMPACT/automated-transformation/add-raw-data-filter

  Show description for ad0d24c
  amarouane-ABDELHAK

  authored
  ad0d24c

  Copy full SHA for ad0d24c

• Add xcom devision logic

  amarouane-ABDELHAK

  committed
  c1bc38b

  Copy full SHA for c1bc38b

Commits on Feb 18, 2025

• Add check function

  amarouane-ABDELHAK

  committed
  525b097

  Copy full SHA for 525b097

Commits on Feb 19, 2025

• 1. Remove generic vector ingest dag

  Show description for a44532a
  paridhi-parajuli

  committed
  a44532a

  Copy full SHA for a44532a

• remove generic vector ingest

  paridhi-parajuli

  committed
  896d718

  Copy full SHA for 896d718

• add default values

  paridhi-parajuli

  committed
  e9908c0

  Copy full SHA for e9908c0

Commits on Feb 20, 2025

• resolve bug

  paridhi-parajuli

  committed
  f1cbdec

  Copy full SHA for f1cbdec

Commits on Feb 25, 2025

• fix: use configuration dict passed into dag run for transfer_data

  botanical

  committed
  f62308f

  Copy full SHA for f62308f

Commits on Feb 26, 2025

• allow dag to fail if even one file processing fails

  paridhi-parajuli

  committed
  1bd9de2

  Copy full SHA for 1bd9de2

• pass status

  paridhi-parajuli

  committed
  d9ab2c9

  Copy full SHA for d9ab2c9

• fix: update cogify type to be a boolean

  botanical

  committed
  ad8236a

  Copy full SHA for ad8236a

• resolve loop termination bug

  paridhi-parajuli

  committed
  3585daf

  Copy full SHA for 3585daf

Commits on Feb 27, 2025

• handle failure report

  paridhi-parajuli

  committed
  c334158

  Copy full SHA for c334158

• fix: add explicit type for dry_run, set default for cogify

  botanical

  committed
  cd62aa9

  Copy full SHA for cd62aa9

• fix: add default value for dry_run

  botanical

  committed
  bf653d2

  Copy full SHA for bf653d2

• Merge pull request #301 from NASA-IMPACT/jt/issue-295-fix-valid-bucket-error

  Show description for f42f998
  botanical

  authored
  f42f998

  Copy full SHA for f42f998

• update readme with new dag configs

  paridhi-parajuli

  committed
  62776c8

  Copy full SHA for 62776c8

Commits on Mar 3, 2025

• Merge pull request #298 from NASA-IMPACT/update/vector-ingestion

  Show description for 67061a7
  paridhi-parajuli

  authored
  67061a7

  Copy full SHA for 67061a7

Commits on Mar 4, 2025

• feat: update docs and example env files

  botanical

  committed
  5284b33

  Copy full SHA for 5284b33

• fix: clean up env files

  botanical

  committed
  bbecef3

  Copy full SHA for bbecef3

• Merge pull request #304 from NASA-IMPACT/feature/automate-to-cog-transformation

  Show description for 700813d
  paridhi-parajuli

  authored
  700813d

  Copy full SHA for 700813d

• fix: revert changes to sm2a-local-config/env_example

  botanical

  committed
  bd8ea81

  Copy full SHA for bd8ea81

Commits on Mar 6, 2025

• Merge pull request #306 from NASA-IMPACT/issue-303/update-docs

  Show description for f05261d
  botanical

  authored
  f05261d

  Copy full SHA for f05261d

Commits on Mar 7, 2025

• Update readme

  amarouane-ABDELHAK

  committed
  9bda766

  Copy full SHA for 9bda766

• Update readme

  amarouane-ABDELHAK

  committed
  0412c8c

  Copy full SHA for 0412c8c

• Merge pull request #310 from NASA-IMPACT/fix/update-readme

  Show description for 4c0ce93
  amarouane-ABDELHAK

  authored
  4c0ce93

  Copy full SHA for 4c0ce93

Commits on Mar 11, 2025

• feat: generate different types of scheduled dags

  anayeaye

  committed
  eea1cb2

  Copy full SHA for eea1cb2

• comment transfer dag generation until configured

  anayeaye

  committed
  b07f5c9

  Copy full SHA for b07f5c9

• use existing vector naming pattern

  anayeaye

  committed
  566c025

  Copy full SHA for 566c025

Commits on Mar 12, 2025

• log vector ingest object

  anayeaye

  committed
  c00f1a0

  Copy full SHA for c00f1a0

• remove comments

  anayeaye

  committed
  1790c77

  Copy full SHA for 1790c77

• Merge pull request #311 from NASA-IMPACT/feat/schedule-vector-ingest

  Show description for bd04e72
  anayeaye

  authored
  bd04e72

  Copy full SHA for bd04e72

• fix: explicitly add schedule interval instead of trusting template

  anayeaye

  committed
  9032d72

  Copy full SHA for 9032d72

• Merge pull request #312 from NASA-IMPACT/fix/generate-vector-dag-schedule

  Show description for 88f375f
  anayeaye

  authored
  88f375f

  Copy full SHA for 88f375f

• fix: only set vector schedule once on dag generation

  anayeaye

  committed
  dbd68ed

  Copy full SHA for dbd68ed

• committed wrong sm2a/dag instead of dag

  anayeaye

  committed
  797055c

  Copy full SHA for 797055c

• Merge pull request #313 from NASA-IMPACT/fix/vector-dag-schedule-arg

  Show description for f199ef1
  anayeaye

  authored
  f199ef1

  Copy full SHA for f199ef1

• require named parameters for clarity

  anayeaye

  committed
  cb48d97

  Copy full SHA for cb48d97

• Merge branch 'dev' into fix/dynamic-dag-args

  anayeaye

  committed
  21f1542

  Copy full SHA for 21f1542

• revert unintented changes

  anayeaye

  committed
  bc63a96

  Copy full SHA for bc63a96

• revert unintented changes

  anayeaye

  committed
  196a61c

  Copy full SHA for 196a61c

Commits on Mar 13, 2025

• Update dags/veda_data_pipeline/utils/s3_discovery.py

  anayeaye

  authored
  5e513bf

  Copy full SHA for 5e513bf

• Update dags/veda_data_pipeline/groups/discover_group.py

  anayeaye

  authored
  b49c51a

  Copy full SHA for b49c51a

• require prefix in discovery event run config

  anayeaye

  authored
  c851bcd

  Copy full SHA for c851bcd

• Merge pull request #314 from NASA-IMPACT/fix/dynamic-dag-args

  Show description for c53a790
  anayeaye

  authored
  c53a790

  Copy full SHA for c53a790

• fix: adjust signature to accept hidden airflow kwargs

  anayeaye

  committed
  023d4cd

  Copy full SHA for 023d4cd

• Merge pull request #316 from NASA-IMPACT/fix/dynamic-dag-generator-args

  Show description for cf05173
  anayeaye

  authored
  cf05173

  Copy full SHA for cf05173

• fix: adjust s3 discovery group signature to accept hidden airflow kwargs

  anayeaye

  committed
  900c685

  Copy full SHA for 900c685

• Merge pull request #317 from NASA-IMPACT/fix/dag-import-errors-kwargs

  Show description for ce62880
  anayeaye

  authored
  ce62880

  Copy full SHA for ce62880

• fix: vector ingest can use href from object metadata that has already been preprocessed for cog ingest

  anayeaye

  committed
  7cee109

  Copy full SHA for 7cee109

• extract filename in one step

  anayeaye

  committed
  86feb34

  Copy full SHA for 86feb34

• Merge pull request #319 from NASA-IMPACT/fix/vector-ingest-use-custom-keys

  Show description for 55fc01b
  anayeaye

  authored
  55fc01b

  Copy full SHA for 55fc01b

• replace deprecated get_files_to_process with get_files_task

  anayeaye

  committed
  be93362

  Copy full SHA for be93362

• Merge pull request #320 from NASA-IMPACT/fix/vector-get-files-task

  Show description for 2530984
  anayeaye

  authored
  2530984

  Copy full SHA for 2530984