issues/150: Added support for NRT filtering by granule id, minor fixes

Author: jackiryan

CHANGELOG.md

@@ -7,10 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 ### Added
+- [issues/150](https://github.com/podaac/bignbit/issues/150): Added support for mixed NRT & Standard collections with configurable regex.
 ### Changed
+- Removed MD5 checksum hash computation to optimize performance of `handle_big_result` lambda.
 ### Deprecated
 ### Removed
 ### Fixed
+- [issues/83](https://github.com/podaac/bignbit/issues/83): Use "_" instead of ":" in CNM filenames for Mac and Linux compatibility
 ### Security
 - [issues/151](https://github.com/podaac/bignbit/issues/151): Added ExpectedBucketOwner parameter when making S3 requests.
 

README.md

@@ -167,6 +167,7 @@ The contents of the configuration file should be a valid json object with the fo
 | sendToHarmony      | boolean      | true/false if this collection should be processed using Harmony to generate browse images                                                                                                                                                                            |
 | operaHLSTreatment  | boolean      | true/false if this collection should have special OPERA_L3_DSWX-HLS processing applied to it (see [apply_opera_hls_treatment](bignbit/apply_opera_hls_treatment.py))                                                                                                 |
 | imageFilenameRegex | string       | Regular expression used to identify which file in a granule should be used as the image file. Uses first if multiple files match                                                                                                                                     |
+| nrtFilenameRegex   | string       | Regular expression used to identify whether a granule should be treated as NRT or standard. All files are assumed standard if not specified.                                                                                                                         |
 | imgVariables       | list(object) | List of JSON objects with at least one attribute called `id` whose value is the name of a variable to generate an image for. `all` can be used in cases where the collection does not have variables or all variables in the collection should have images generated |
 | height             | int          | [OPTIONAL] Controls the height of the output image from Harmony (see https://github.com/nasa/harmony-browse-image-generator?tab=readme-ov-file#dimensions--scale-sizes)                                                                                              |
 | width              | int          | [OPTIONAL] Controls the width of the output image from Harmony (see https://github.com/nasa/harmony-browse-image-generator?tab=readme-ov-file#dimensions--scale-sizes)                                                                                               |

bignbit/handle_big_result.py

@@ -10,6 +10,7 @@
 import xml.etree.ElementTree as ET
 from datetime import datetime, timezone
 from pathlib import Path
+import re
 from typing import Any
 from urllib.parse import urlparse
 
@@ -73,20 +74,7 @@ def process(self) -> dict[str, Any]:
         # Throw KeyError if the state input doesn't contain the dataset config
         dataset_config = self.input['datasetConfigurationForBIG']['config']
 
-        data_day_strat = dataset_config.get('dataDayStrategy')
-        if data_day_strat is not None and data_day_strat == 'single_day_of_year':
-            static_data_day = dataset_config.get('singleDayNumber', 1)
-            # Throw TypeError on bad configuration
-            static_data_day = int(static_data_day)
-        else:
-            static_data_day = None
-
-        if static_data_day is not None and (static_data_day < 1 or static_data_day > 366):
-            CUMULUS_LOGGER.warning(
-                f'Specified data day override {static_data_day} is not logical '
-                'as a day of year. Defaulting to doy 001.'
-            )
-            static_data_day = 1
+        static_data_day = _resolve_static_data_day(dataset_config)
 
         subdaily = dataset_config.get('subdaily', False)
         try:
@@ -102,6 +90,13 @@ def process(self) -> dict[str, Any]:
         bignbit_audit_path = self.config.get('bignbit_audit_path')
         granule_id = self.input['granules'][0]['granuleId']
 
+        nrt_filename_regex = dataset_config.get('nrtFilenameRegex')
+        is_nrt = False
+        if nrt_filename_regex:
+            nrt_match = re.match(nrt_filename_regex, granule_id)
+            if nrt_match:
+                is_nrt = True
+
         try:
             partial_id = utils.extract_mgrs_grid_code(granule_umm_json)
         except KeyError:
@@ -160,6 +155,7 @@ def process(self) -> dict[str, Any]:
                 cmr_provider,
                 collection_name,
                 granule_id,
+                is_nrt,
                 bignbit_audit_bucket,
                 bignbit_audit_path
             )
@@ -179,6 +175,34 @@ def process(self) -> dict[str, Any]:
         return response_payload
 
 
+def _resolve_static_data_day(dataset_config: dict) -> int | None:
+    """
+    Determine the static data day override from dataset config, if configured.
+
+    Parameters
+    ----------
+    dataset_config : dict
+      Dataset configuration dictionary
+
+    Returns
+    -------
+    int | None
+      The static day-of-year override (1–366), or None if not configured.
+      Falls back to 1 if the configured value is outside the valid range.
+    """
+    if dataset_config.get('dataDayStrategy') != 'single_day_of_year':
+        return None
+    # Throw TypeError on bad configuration
+    static_data_day = int(dataset_config.get('singleDayNumber', 1))
+    if static_data_day < 1 or static_data_day > 366:
+        CUMULUS_LOGGER.warning(
+            f'Specified data day override {static_data_day} is not logical '
+            'as a day of year. Defaulting to doy 001.'
+        )
+        return 1
+    return static_data_day
+
+
 def process_harmony_results(harmony_job: dict[str, str], cmr_env: str) -> list[dict[str, Any]]:
     """
     Process the results of a Harmony job
@@ -216,18 +240,17 @@ def process_harmony_results(harmony_job: dict[str, str], cmr_env: str) -> list[d
     for url in result_urls:
         bucket, key = urlparse(url).netloc, urlparse(url).path.lstrip('/')
 
-        response = s3_client.get_object(Bucket=bucket, Key=key,
-                                        ExpectedBucketOwner=utils.get_aws_account_id())
-        md5_hash = hashlib.new('md5')
-        for chunk in response['Body'].iter_chunks(chunk_size=100 * 1024 * 1024):  # 100 MB chunk size
-            md5_hash.update(chunk)
+        response = s3_client.head_object(Bucket=bucket, Key=key,
+                                         ExpectedBucketOwner=utils.get_aws_account_id())
+        # For single-part uploads, the S3 ETag is the MD5 hex digest of the object
+        etag = response['ETag'].strip('"')
 
         filename = key.split('/')[-1]
         file_dict = {
             'fileName': filename,
             'bucket': bucket,
             'key': key,
-            'checksum': md5_hash.hexdigest(),
+            'checksum': etag,
             'checksumType': 'md5'
         }
         # Weird quirk where if we are working with a collection that doesn't define variables, the Harmony request
@@ -392,6 +415,7 @@ def write_cnm_message(
         cmr_provider: str,
         collection_name: str,
         granule_id: str,
+        is_nrt: bool,
         bignbit_audit_bucket: str,
         bignbit_audit_path: str,
 ) -> str:
@@ -408,6 +432,8 @@ def write_cnm_message(
       Collection that this image set belongs to
     granule_id: str
       Granule id (used to determine CNM filename)
+    is_nrt: bool
+      True if the filename of the image_set is considered NRT
     bignbit_audit_bucket: str
       staging bucket where CNM is uploaded (default is *-internal)
     bignbit_audit_path: str
@@ -418,11 +444,11 @@ def write_cnm_message(
     s3_key: str
       s3 key pointing to the uploaded CNM message
     """
-    cnm_message = construct_cnm(image_set, cmr_provider, collection_name)
+    cnm_message = construct_cnm(image_set, cmr_provider, collection_name, is_nrt)
     cnm_bytes = json.dumps(cnm_message).encode()
     submission_time = cnm_message.get('submissionTime')
     collection_fullname = cnm_message.get('collection')
-    cnm_key = f'{bignbit_audit_path}/{collection_fullname}/{granule_id}.{submission_time}.cnm.json'
+    cnm_key = f'{bignbit_audit_path}/{collection_fullname}/{granule_id}.{submission_time}.cnm.json'.replace(':', '_')
     utils.upload_object(
         cnm_bytes,
         bignbit_audit_bucket,
@@ -435,19 +461,22 @@ def write_cnm_message(
 def construct_cnm(
         image_set: ImageSet,
         cmr_provider: str,
-        collection_name: str
+        collection_name: str,
+        is_nrt: bool,
 ) -> dict[str, Any]:
     """
     Construct the CNM message for GITC
 
     Parameters
     ----------
     image_set: ImageSet
-        ImageSet for one image to be sent to gibs
+      ImageSet for one image to be sent to gibs
     cmr_provider: str
       The provider sent in the CNM message
     collection_name: str
       Collection that this image set belongs to
+    is_nrt: bool
+      True if the filename of the image_set is considered NRT
 
     Returns
     ----------
@@ -457,11 +486,12 @@ def construct_cnm(
     product = to_cnm_product_dict(image_set)
     submission_time = datetime.now(timezone.utc).isoformat()[:-9] + 'Z'
     CUMULUS_LOGGER.debug(image_set.image['variable'])
+    nrt_string = '_NRT' if is_nrt else ''
     if 'output_crs' in image_set.image:
         crs_suffix = GIBS_CRS_NAME_TO_SUFFIX.get(image_set.image.get('output_crs', 'EPSG:4326'), 'LL')
-        new_collection = f"{collection_name}_{image_set.image['variable']}_{crs_suffix}".replace('/', '_')
+        new_collection = f"{collection_name}_{image_set.image['variable']}{nrt_string}_{crs_suffix}".replace('/', '_')
     else:
-        new_collection = f"{collection_name}_{image_set.image['variable']}".replace('/', '_')
+        new_collection = f"{collection_name}_{image_set.image['variable']}{nrt_string}".replace('/', '_')
 
     return {
         'version': '1.5.1',

scripts/CONFIGURATION.md

@@ -21,39 +21,51 @@ To configure a collection for bignbit, you need to have the following informatio
   shortName             Short name of the collection
 
   --collectionId COLLECTIONID
-                        CMR collection id, use if the collection name is ambiguous.
-                        WARNING: since collection ids are venue-specific, the output config will need to be re-generated when deploying to a separate venue.
-
+                        CMR collection id, use if the collection name is ambiguous. WARNING: since
+                        collection ids are venue-specific, the output config will need to be re-
+                        generated when deploying to a separate venue.
+  --no-sendToHarmony    Specifies that the collection should not be processed by Harmony API.
+                        Default is `sendToHarmony = True`.
+  --operaHLSTreatment   Specifies whether the collection should receive the "OPERA HLS Treatment".
+                        Default is `operaHLSTreatment = False`
   --imageFilenameRegex IMAGEFILENAMEREGEX
-                        Regular expression used to identify which file in a granule should be used as the image file.
-                        Uses first if multiple files match
-
+                        Regular expression used to identify which file in a granule should be used
+                        as the image file. Uses first if multiple files match
+  --nrtFilenameRegex NRTFILENAMEREGEX
+                        Regular expression used to identify whether a granule should be treated as
+                        NRT or standard. All files are assumed standard if not specified.
   --imgVariables VAR_NAME [VAR_NAME ...]
-                        List of variable names (not UMM-Var concept ids) to use for generating browse image(s).
-                        If none are provided, the default of "all" is used
-
+                        List of variable names (not UMM-Var concept ids) to use for generating
+                        browse image(s). If none are provided, the default of "all" is used
   --dimensions HEIGHT WIDTH
-                        Optional override to specify the height and width that each browse image should have when processed by Harmony.
-
+                        Optional override to specify the height and width that each browse image
+                        should have when processed by Harmony.
   --scaleExtentPolar MINX MINY MAXX MAXY
                         Controls the geographic extent of polar-projected browse image outputs.
-                        This keyword is ignored if `outputCrs` does not contain EPSG:3413 or EPSG:3031
-                        (polar stereographic projections used by GIBS)
-
+                        This keyword is ignored if `outputCrs` does not contain EPSG:3413 or
+                        EPSG:3031 (polar stereographic projections used by GIBS)
   --singleDayNumber JJJ
-                        All granules in this dataset will use the day of year specified in this keyword. ex: "001" for January 1st
-
-  --subdaily            Set to true if granules contain subdaily data. This will send `DataDateTime` metadata to GIBS as described in the GIBS ICD
-
-  --outputCrs list of CRS strings from these options: EPSG:4326,EPSG:3413,EPSG:3031,EPSG:3857
-                        List of output projections or coordinate reference systems for which to produce browse images. Applies to all variables in granule. GIBS-compatible values are EPSG:4326, EPSG:3413, or EPSG:3031
+                        All granules in this dataset will use the day of year specified in this
+                        keyword. ex: "001" for January 1st
+  --subdaily            Set to true if granules contain subdaily data. This will send
+                        `DataDateTime` metadata to GIBS as described in the GIBS ICD
+  --outputCrs {EPSG:4326,EPSG:3413,EPSG:3031,EPSG:3857} [{EPSG:4326,EPSG:3413,EPSG:3031,EPSG:3857} ...]
+                        List of output projections or coordinate reference systems for which to
+                        produce browse images. Applies to all variables in granule. GIBS-
+                        compatible values are EPSG:4326, EPSG:3413, or EPSG:3031
+  --reprojection        Set to true to force reprojection for all granules. This will send the
+                        `outputCrs` parameter in Harmony API calls unconditionally. Warning: if
+                        the Harmony service used to generate browse images does not support
+                        reprojection, jobs will fail.
 ```
 
 Additionally, you can provide the `--s3Destination` keyword if you wish to upload the dataset config directly to s3. You must have valid access credentials to your bucket in the environment you run the script if you wish to use this option.
 
 ```
   --s3Destination BUCKET KEY
-                        Specify an S3 URI (bucket key, ex: --s3Destination podaac-bignbit-sit-svc-internal big-config) to upload the config. Do not specify the filename, it is auto-generated.
+                        Specify an S3 URI (bucket key, ex: --s3Destination podaac-bignbit-sit-svc-
+                        internal big-config) to upload the config. Do not specify the filename, it
+                        is auto-generated.
 ```
 
 ## Setup

scripts/create_dataset_config.py

@@ -303,7 +303,7 @@ def cli() -> argparse.Namespace:
     parser.add_argument(
         'shortName',
         type=str,
-        help='Short name of the collection, '
+        help='Short name of the collection'
     )
     parser.add_argument(
         '--collectionId',
@@ -338,6 +338,14 @@ def cli() -> argparse.Namespace:
             'the image file. Uses first if multiple files match'
         )
     )
+    parser.add_argument(
+        '--nrtFilenameRegex',
+        type=str,
+        help=(
+            'Regular expression used to identify whether a granule should be treated as NRT or '
+            'standard. All files are assumed standard if not specified.'
+        )
+    )
     parser.add_argument(
         '--imgVariables',
         type=str,