Release/0.8.0 by jackiryan · Pull Request #183 · podaac/bignbit

jackiryan · 2026-06-04T22:18:55Z

Description

This release is intended to address some updates needed for Consolidated Cumulus (CCumulus) deployment, and to fix several pain points of scaling bignbit.

STOP AND PLEASE READ

!!!THIS IS A BREAKING CHANGE FOR EXISTING DEPLOYMENTS!!! If you are deploying this release to a venue with an existing deployment of bignbit, please note that most resource names have changed. You will need to temporarily configure your tfvars to force_delete the ecr repo and staging bucket, then set it back to false after migration:

Name	Type	Description	Default Value
force_delete_ecr	bool	Force delete the ECR repository and all its images on destroy. Set to true when migrating resource names to avoid manual cleanup.	false
force_destroy_staging_bucket	bool	Force destroy the staging S3 bucket and all its contents on destroy. Set to true when migrating resource names to avoid manual cleanup.	false

Migration Procedure:

Empty the contents of the existing bignbit ECR repo in your venue, in other words delete all the images inside it. The old ECR repo will be called sit-bignbit, uat-bignbit or ops-bignbit.
Delete or migrate the contents of the S3 staging bucket used by bignbit. This will have a name like bignbit-<prefix>-staging.
Set the two variables in the table above to true in your tfvars file.
Deploy as normal.
Set the variables back to false after migrating or delete the lines from your tfvars.

Added

issues/150: Added support for mixed NRT & Standard collections with configurable regex.

Changed

issues/173: Changed resource names (e.g. ECR repo name) to include deployment level prefixes. This allows for deploying multiple copies of bignbit to a single AWS account/venue.
Removed MD5 checksum hash computation to optimize performance of handle_big_result lambda.

Fixed

issues/83: Use "_" instead of ":" in CNM filenames for Mac and Linux compatibility
issues/181: Auto-retry logic for Harmony jobs that fail with transient server errors (5xx errors)

Security

issues/151: Added ExpectedBucketOwner parameter when making S3 requests.

Overview of verification done

Deployed to podaac-svc-sit, verified migration works when staging bucket and ecr repo are manually cleared out.
Tested several types of supported data to confirm jobs still execute successfully. Ran up to 3 concurrent jobs
Unit tests passing

Overview of integration done (TBD)

Integration test 1: Stress-testing in ASF's cumulus UAT environment to see if the retry behavior for Harmony status checks works under the load of processing 10 days of NISAR data.
Integration test 2: Consolidated Cumulus deployment and tests with ASDC TEMPO data, likely in Sprint 6.

PR checklist:

Linted
Updated unit tests
Updated changelog
Integration testing (TBD, will be performed by ASF and CCumulus)

See Pull Request Review Checklist for pointers on reviewing this pull request

…ts (#174) * issues/151: Added ExpectedBucketOwner parameter when making S3 requests * issues/151: Added ExpectedBucketOwner to get_object requests

#178) * issues/150: Added support for NRT filtering by granule id, minor fixes

* issues/173: changed default resource names to allow multiple deployments. BREAKING CHANGE, requires migrating using the new module inputs (see readme) * issues/181: Added retry logic for Harmony jobs that fail with 5xx errors

asjohnston-asf

The changes in get_harmony_job_status.py and state_machine_definition.tpl related to #181 look reasonable and are a clear improvement. I left two suggestions to make the retries a little more forgiving in additional error cases, but I don't see those as blocking a release.

asjohnston-asf · 2026-06-05T20:03:02Z

                      },
+                      {
+                        "ErrorEquals":[
+                          "HarmonyTransientError"


I'd consider grouping this into the same clause with HarmonyJobIncompleteError. As written it only allows ~2 minutes for any underlying error to resolve itself. I'd be willing to give the issue the same amount of time to resolve as we're willing to wait for the job to finish.

Yeah you're right, I didn't really think through the retry behavior in that much detail but it's idiosyncratic that it doesn't use the main retry parameters

asjohnston-asf · 2026-06-05T20:07:14Z

-    job_status = harmony_client.status(harmony_job_id)
+    try:
+        job_status = harmony_client.status(harmony_job_id)
+    except requests.exceptions.HTTPError as exc:


consider also catching requests.exceptions.ConnectionTimeout, and possibly requests.exceptions.ConnectionError, which will cover some of the cases where the request couldn't reach the server and doesn't get back any status code, 500 or otherwise.

…188)

Copilot

Pull request overview

This PR prepares the 0.8.0 release with breaking infrastructure renames for Consolidated Cumulus deployments, adds support for mixed NRT/Standard collections, improves robustness of Harmony status polling with retries on transient failures, and tightens S3 security by enforcing ExpectedBucketOwner.

Changes:

Add NRT vs Standard handling via configurable nrtFilenameRegex, and propagate NRT into CNM collection naming and CNM object keys (colon-safe filenames).
Add Harmony transient error classification and Step Functions retry behavior for 5xx/connection/timeouts.
Update Terraform resource naming strategy for multi-deploy-per-account, plus add migration helpers (force_delete_ecr, force_destroy_staging_bucket) and S3 ExpectedBucketOwner enforcement.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`bignbit/handle_big_result.py`	Adds NRT handling, refactors static day-of-year logic, switches checksum strategy to prefer S3 ETag, updates CNM naming.
`bignbit/get_harmony_job_status.py`	Wraps transient Harmony failures in `HarmonyTransientError` for retry.
`bignbit/get_dataset_configuration.py`	Adds `ExpectedBucketOwner` to S3 `get_object`.
`bignbit/send_to_gitc.py`	Adds `ExpectedBucketOwner` to S3 `get_object` when reading CNMs.
`bignbit/utils.py`	Adds cached STS lookup for AWS account ID and applies `ExpectedBucketOwner` to S3 uploads.
`terraform/main.tf`	Changes resource name prefixing strategy and introduces a separate SNS topic name local to preserve the existing topic name.
`terraform/lambda_functions.tf`	Updates ECR repo naming to include deployment prefix and adds `force_delete` option.
`terraform/s3.tf`	Adds `force_destroy` option for the staging bucket to support migration.
`terraform/sqs_sns.tf`	Preserves SNS topic name via `sns_topic_name` local while other resources adopt new naming.
`terraform/state_machine_definition.tpl`	Adds `HarmonyTransientError` to Step Functions retry configuration.
`terraform/variables.tf`	Introduces `force_delete_ecr` and `force_destroy_staging_bucket` variables.
`.github/workflows/cicd-pipeline.yml`	Adds GHCR login step prior to deployment.
`scripts/create_dataset_config.py`	Adds `--nrtFilenameRegex` CLI option for dataset config generation.
`scripts/CONFIGURATION.md`	Documents the new `--nrtFilenameRegex` option and refreshes CLI help formatting.
`README.md`	Documents new Terraform variables and dataset config field `nrtFilenameRegex`.
`tests/test_handle_big_result.py`	Expands tests for ETag-based checksums, multipart ETag fallback, NRT collection naming, and `_resolve_static_data_day`.
`tests/test_get_harmony_job_status.py`	Adds unit tests for transient error wrapping behavior.
`tests/test_send_to_gitc.py`	Updates tests to support STS mocking for account ID resolution.
`tests/test_image_set.py`	Updates `construct_cnm` calls for new `is_nrt` parameter.
`examples/cumulus-tf/tfvars/sit.tfvars`	Adds the new migration variables to the SIT example.
`CHANGELOG.md`	Adds `0.8.0` release notes matching the PR description.
`pyproject.toml`	Bumps package version to `0.8.0rc2`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            get_response = s3_client.get_object(Bucket=bucket, Key=key,
+                                                ExpectedBucketOwner=utils.get_aws_account_id())
+            checksum = hashlib.md5(get_response['Body'].read()).hexdigest()


 [tool.poetry]
 name = "bignbit"
-version = "0.7.2"
+version = "0.8.0rc2"


jackiryan and others added 11 commits May 12, 2026 21:46

/version 0.8.0a0

ab45f22

Merge remote-tracking branch 'origin/main' into develop

0f2424c

/version 0.8.0a6

58d5dd6

issues/151: Added ExpectedBucketOwner parameter when making S3 reques…

a17b71e

…ts (#174) * issues/151: Added ExpectedBucketOwner parameter when making S3 requests * issues/151: Added ExpectedBucketOwner to get_object requests

/version 0.8.0a7

d7da221

issues/150: Added support for NRT filtering by granule id, minor fixes (

2cef61b

#178) * issues/150: Added support for NRT filtering by granule id, minor fixes

/version 0.8.0a8

02f9d68

Issues/173 (#182)

7de117b

* issues/173: changed default resource names to allow multiple deployments. BREAKING CHANGE, requires migrating using the new module inputs (see readme) * issues/181: Added retry logic for Harmony jobs that fail with 5xx errors

/version 0.8.0a9

36175fb

chore: updated changelog

4f29f24

/version 0.8.0rc1

ed83ce2

github-project-automation Bot added this to podaac Jun 4, 2026

tloubrieu-jpl moved this to needs:triage in podaac Jun 4, 2026

jackiryan requested a review from jamesfwood June 4, 2026 22:19

asjohnston-asf reviewed Jun 5, 2026

View reviewed changes

jackiryan and others added 2 commits June 9, 2026 12:54

Updated HarmonyTransientError handling to address reviewer comments (#…

94cf509

…188)

/version 0.8.0rc2

99e93ed

jackiryan had a problem deploying to SIT June 9, 2026 20:16 — with GitHub Actions Failure

jackiryan had a problem deploying to SIT June 9, 2026 20:20 — with GitHub Actions Failure

jackiryan temporarily deployed to SIT June 9, 2026 20:26 — with GitHub Actions Inactive

jackiryan temporarily deployed to SIT June 9, 2026 20:32 — with GitHub Actions Inactive

jamesfwood requested a review from Copilot June 9, 2026 22:19

Copilot started reviewing on behalf of jamesfwood June 9, 2026 22:19 View session

jamesfwood approved these changes Jun 9, 2026

View reviewed changes

jackiryan merged commit 6e420f9 into main Jun 9, 2026
13 of 15 checks passed

jackiryan deleted the release/0.8.0 branch June 9, 2026 22:31

tloubrieu-jpl moved this from needs:triage to closed in podaac Jun 9, 2026

Copilot AI reviewed Jun 9, 2026

View reviewed changes

Navigation Menu

Uh oh!

Release/0.8.0 - #183

Release/0.8.0#183
jackiryan merged 13 commits into
mainfrom
release/0.8.0

jackiryan commented Jun 4, 2026 •

edited

Loading

Uh oh!

asjohnston-asf left a comment

Uh oh!

asjohnston-asf Jun 5, 2026

Uh oh!

jackiryan Jun 8, 2026

Uh oh!

asjohnston-asf Jun 5, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

jackiryan commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

STOP AND PLEASE READ

Added

Changed

Fixed

Security

Overview of verification done

Overview of integration done (TBD)

PR checklist:

Uh oh!

asjohnston-asf left a comment

Choose a reason for hiding this comment

Uh oh!

asjohnston-asf Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

jackiryan Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

asjohnston-asf Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jackiryan commented Jun 4, 2026 •

edited

Loading