Merge pull request #87 from podaac/release/0.3.0

Release/0.3.0

Author: jackiryan

.github/workflows/cicd-pipeline.yml

@@ -279,7 +279,6 @@ jobs:
       TF_VAR_gibs_account_id: ${{ secrets.GIBS_ACCOUNT_ID }}
     if: |
       github.ref == 'refs/heads/develop' ||
-      github.ref == 'refs/heads/main'    ||
       startsWith(github.ref, 'refs/heads/release') ||
       github.event_name == 'workflow_dispatch'
     steps:

CHANGELOG.md

@@ -13,6 +13,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 ### Security
 
+## [0.3.0]
+### Added
+- [issues/59](https://github.com/podaac/bignbit/issues/59): A new pair of keywords (`dataDayStrategy` and `singleDayNumber`) have been added to the DatasetConfiguration for BIG to enable proper image metadata for annual products. These keywords allow a dataset to override the umm-g date info.
+- [issues/84](https://github.com/podaac/bignbit/issues/84): New parameter in dataset config `subdaily` that sends DataDateTime to GIBS instead of DataDay.
+- Added optional `concept_id` keyword to dataset config to provide an override for finding the proper CMR collection concept ID when testing.
+### Changed
+### Deprecated
+### Removed
+### Fixed
+- [issues/82](https://github.com/podaac/bignbit/issues/82): Fixed date parsing bug where ISO-8601 format dates, the default for UMM-G, were not handled properly.
+- Update gibs_response_queue visibility timeout to match aws_lambda_function handle_gitc_response timeout
+### Security
+
 ## [0.2.4]
 ### Added
 - [issues/71](https://github.com/podaac/bignbit/issues/71): New module parameter `cmr_environment` is used to determine which environment to use for CMR requests when processing GIBS responses.

README.md

@@ -99,10 +99,28 @@ should define the bignbit module and the bignbit step function state machine. Se
 
 
 # Configuring a collection
+In order to configure a collection for use with bignbit the following must be done:
 
 1. Add config file to the `config_bucket`. The file should be named "_collection shortname_.cfg" and the contents should be JSON
 2. Associate the UMM-C record to the appropriate Harmony service (HyBIG, net2cog, etc...)
 
+The contents of the configuration file should be a valid json object with the following attributes:
+
+| Name               | Type         | Description                                                                                                                                                                                                                                                          | 
+|--------------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| sendToHarmony      | boolean      | true/false if this collection should be processed using Harmony to generate browse images                                                                                                                                                                            |
+| operaHLSTreatment  | boolean      | true/false if this collection should have special OPERA_L3_DSWX-HLS processing applied to it (see [apply_opera_hls_treatment](bignbit/apply_opera_hls_treatment.py))                                                                                                 |
+| imageFilenameRegex | string       | Regular expression used to identify which file in a granule should be used as the image file. Uses first if multiple files match                                                                                                                                     |
+| imgVariables       | list(object) | List of JSON objects with at least one attribute called `id` whose value is the name of a variable to generate an image for. `all` can be used in cases where the collection does not have variables or all variables in the collection should have images generated |
+| height             | int          | Controls the height of the output image from Harmony (see https://github.com/nasa/harmony-browse-image-generator?tab=readme-ov-file#dimensions--scale-sizes)                                                                                                         |
+| width              | int          | Controls the width of the output image from Harmony (see https://github.com/nasa/harmony-browse-image-generator?tab=readme-ov-file#dimensions--scale-sizes)                                                                                                          |
+| dataDayStrategy    | string       | [OPTIONAL] (Default: "") If this keyword is set to "single_day_of_year", bignbit will override the date from the granule metadata with the one specified in "singleDayNumber"                                                                                        |
+| singleDayNumber    | string       | [OPTIONAL] (Default: "") If using the "dataDayStrategy" keyword, all granules in this dataset will use the day of year specified in this keyword. ex: "001" for January 1st                                                                                          |
+| subdaily           | boolean      | [OPTIONAL] (Default: False) Set to true if granules contain subdaily data. This will send `DataDateTime` metadata to GIBS as described in the GIBS ICD                                                                                                               |
+| concept_id         | string       | [OPTIONAL] (Default: "") Overrides the concept id derived from the granule metadata with this value. ex: "C1996881146-POCLOUD"                                                                                                                                       |
+
+A few example configurations can be found in the [podaac/bignbit-config](https://github.com/podaac/bignbit-config) repository. NOTE: some of the example configurations have other options specified (e.g. `variables`, `latVar`, `lonVar`, etc...) that are no longer supported by this module. The table above are the attributes that are still in use.
+
 ## Harmony requests
 
 > [!IMPORTANT]  
@@ -189,11 +207,12 @@ _Visual representation of the bignbit step function state machine:_
 # Local Development
 ## MacOS
 
-1. Install miniconda (or conda) and [poetry](https://python-poetry.org/)
+1. Install miniconda (or conda) and [poetry](https://python-poetry.org/). On MacOS, you can `brew install miniconda` and `brew install poetry`.
 2. Run `conda env create -f conda-environment.yaml` to install GDAL
-3. Activate the bignbit conda environment `conda activate bignbit`
-4. Install python package and dependencies `poetry install`
-5. Verify tests pass `poetry run pytest tests/`
+3. **(For MacOS users)** If you have a MacOS system and use zsh, you will need to run `conda init zsh`, close your terminal, and re-open before activating the environment in the next step.
+4. Activate the bignbit conda environment `conda activate bignbit`
+5. Install python package and dependencies `poetry install`
+6. Verify tests pass `poetry run pytest tests/`
 
 > [!IMPORTANT] 
 > If developing on a `darwin_arm64` based mac, running terraform locally may result
@@ -203,4 +222,4 @@ _Visual representation of the bignbit step function state machine:_
 > to compile a local arm-based version of the hashicorp/null provider.
 > 
 > This is only necessary as long as cumulus core requires `~>2.1` version of hashicorp/null because
-> v3.x of the provider does have support for `darwin_arm64` platforms
+> v3.x of the provider does have support for `darwin_arm64` platforms

bignbit/generate_image_metadata.py

@@ -6,7 +6,7 @@
 import pathlib
 import uuid
 import xml.etree.ElementTree as ET
-from datetime import datetime
+from datetime import datetime, timedelta
 from typing import Dict, List
 
 from cumulus_logger import CumulusLogger
@@ -53,14 +53,36 @@ def process(self) -> List[Dict]:
             cma_file_list = [item for sublist in cma_file_list for item in sublist]
         granule_umm_json = self.input['granule_umm_json']
 
-        file_metadata_list = generate_metadata(cma_file_list, granule_umm_json, pathlib.Path(f"{self.path}"))
+        # Parse dataset-level overrides
+        dataset_config = self.input['datasetConfigurationForBIG']['config']
+        data_day_strat = dataset_config.get('dataDayStrategy')
+        if data_day_strat is not None and data_day_strat == 'single_day_of_year':
+            static_data_day = dataset_config.get('singleDayNumber', 1)
+            # Will throw TypeError on bad configuration
+            static_data_day = int(static_data_day)
+        else:
+            static_data_day = None
+
+        subdaily = dataset_config.get('subdaily', False)
+
+        file_metadata_list = generate_metadata(
+            cma_file_list, granule_umm_json,
+            pathlib.Path(f"{self.path}"),
+            static_data_day,
+            subdaily
+        )
         del self.input['granule_umm_json']
         del self.input['big']
         self.input['big'] = file_metadata_list
         return self.input
 
 
-def generate_metadata(cma_file_list: List[Dict], granule_umm_json: dict, temp_dir: pathlib.Path) -> List[Dict]:
+def generate_metadata(
+        cma_file_list: List[Dict],
+        granule_umm_json: Dict,
+        temp_dir: pathlib.Path,
+        static_data_day: int | None = None,
+        subdaily: bool = False) -> List[Dict]:
     """
     For each file in the list, create an ImageMetadata-v1.2 xml file and upload it to s3 in the same
     bucket and path as the image file.
@@ -75,6 +97,11 @@ def generate_metadata(cma_file_list: List[Dict], granule_umm_json: dict, temp_di
       umm-json document for the granule being processed
     temp_dir
       Temporary location to write xml file to prior to upload to s3
+    static_data_day
+      Optionally, the DatasetConfiguration can override the date metadata in the
+      granule umm-json
+    subdaily
+      boolean flag if product is subdaily (if True, add DataDateTime to metadata)
 
     Returns
     -------
@@ -86,8 +113,14 @@ def generate_metadata(cma_file_list: List[Dict], granule_umm_json: dict, temp_di
         granule_filename = cma_file_meta['filename'] if 'filename' in cma_file_meta else cma_file_meta['fileName']
         CUMULUS_LOGGER.info(f'Processing file {granule_filename}')
 
-        # Get date information from umm-g
-        begin, mid, end, dataday = extract_granule_dates(granule_umm_json)
+        if static_data_day is not None and (static_data_day < 1 or static_data_day > 366):
+            CUMULUS_LOGGER.warning(
+                f"Specified data day override {static_data_day} is not logical "
+                "as a day of year. Defaulting to doy 001."
+            )
+            static_data_day = 1
+        # Get date information from umm-g, if static_data_day is None it will be ignored
+        begin, mid, end, dataday = extract_granule_dates(granule_umm_json, static_data_day)
 
         # Determine type and subtype for CNM
         granule_extension = pathlib.Path(granule_filename).suffix
@@ -105,15 +138,15 @@ def generate_metadata(cma_file_list: List[Dict], granule_umm_json: dict, temp_di
         except KeyError:
             # No partial id for this granule
             partial_id = None
-            CUMULUS_LOGGER.warn(f"No partial id found in metadata for {granule_filename}. Leaving partial id blank.")
+            CUMULUS_LOGGER.warning(f"No partial id found in metadata for {granule_filename}. Leaving partial id blank.")
 
         # Convert from CMA file dict to CNM file dict
         cnm_file_meta = transform_files_to_cnm_product_files(cma_file_meta, granule_type, granule_subtype, dataday)
         file_metadata_results.append(cnm_file_meta)
 
         # If this is a browse image, generate image metadata for it and upload it to s3
         if granule_type == 'browse':
-            image_metadata_xml = create_metadata_xml(begin, mid, end, dataday, partial_id)
+            image_metadata_xml = create_metadata_xml(begin, mid, end, dataday, subdaily, partial_id)
             temp_xml_path = write_image_metadata_xml(image_metadata_xml, temp_dir)
             image_metadata_xml_metadata = get_file_metadata_for_image_metadta_xml(temp_xml_path, cnm_file_meta)
             s3_uri = upload_image_metadata_xml(image_metadata_xml_metadata, temp_xml_path)
@@ -247,7 +280,7 @@ def transform_files_to_cnm_product_files(cma_file_meta: Dict, file_type: str, su
     return cnm_file_meta
 
 
-def extract_granule_dates(granule_umm_json: dict) -> (str, str, str, str):
+def extract_granule_dates(granule_umm_json: dict, static_data_day: int | None = None) -> tuple[str, str, str, str]:
     """
     Parse the begin, midpoint, end, and dataday for this granule
 
@@ -266,19 +299,28 @@ def extract_granule_dates(granule_umm_json: dict) -> (str, str, str, str):
     dataday
       day this granule applies to as string formatted "%Y%j
     """
-    time_range_dict = granule_umm_json['TemporalExtent']['RangeDateTime']
+    time_range_dict = granule_umm_json["TemporalExtent"]["RangeDateTime"]
 
     beginning_time_dt = parse_datetime(time_range_dict["BeginningDateTime"])
     ending_time_dt = parse_datetime(time_range_dict["EndingDateTime"])
     middle_time_dt = beginning_time_dt + (ending_time_dt - beginning_time_dt) / 2
 
-    begin = beginning_time_dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
-    mid = middle_time_dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
-    end = ending_time_dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
-
-    middle_year = middle_time_dt.strftime("%Y")
-    day_of_year = middle_time_dt.strftime('%j')
-    dataday = middle_year + day_of_year
+    if static_data_day is None:
+        begin = beginning_time_dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+        mid = middle_time_dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+        end = ending_time_dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+
+        middle_year = middle_time_dt.strftime("%Y")
+        day_of_year = middle_time_dt.strftime("%j")
+        dataday = middle_year + day_of_year
+    else:
+        # If the static_data_day override is set, parse the year from the midpoint
+        # of the granule, and set all metadata dates to the doy that was set.
+        data_year = middle_time_dt.year
+        begin = parse_doy(data_year, static_data_day)
+        mid = begin
+        end = begin
+        dataday = middle_time_dt.strftime("%Y") + f"{static_data_day:03d}"
 
     return begin, mid, end, dataday
 
@@ -297,14 +339,46 @@ def parse_datetime(datetime_str: str) -> datetime:
     datetime
       a datetime object
     """
-    try:
-        return datetime.strptime(datetime_str, "%Y-%m-%dT%H:%M:%S.%fZ")
-    except ValueError:
-        return datetime.strptime(datetime_str, "%Y-%m-%dT%H:%M:%SZ")
+    formats = [
+        "%Y-%m-%dT%H:%M:%S.%fZ",        # 2023-01-01T12:30:45.123456Z
+        "%Y-%m-%dT%H:%M:%SZ",           # 2023-01-01T12:30:45Z
+        "%Y-%m-%dT%H:%M:%S.%f%z",       # 2023-01-01T12:30:45.123456+00:00
+        "%Y-%m-%dT%H:%M:%S%z",          # 2023-01-01T12:30:45+00:00
+    ]
+
+    for fmt in formats:
+        try:
+            return datetime.strptime(datetime_str, fmt)
+        except ValueError:
+            continue
+
+    # If none of the formats worked
+    raise ValueError(f"Unable to parse datetime string: {datetime_str}")
+
+
+def parse_doy(year: int, doy: int) -> str:
+    """
+    Parses a year and day of year into a string.
+
+    Parameters
+    ----------
+    year
+      integer year (parsed from midpoint of granule time range)
+    doy
+      static data day provided in DatasetConfiguration
+
+    Returns
+    -------
+    str
+      a static Y-m-d format date string with the time set to midnight UTC
+    """
+    jan_1 = datetime(year, 1, 1)
+    result_dt = jan_1 + timedelta(days=doy - 1)
+    return result_dt.strftime("%Y-%m-%dT00:00:00.000000Z")
 
 
 def create_metadata_xml(beginning_time: str, middle_time: str, ending_time: str, dataday: str,
-                        partial_id: str = None) -> ET.ElementTree:
+                        subdaily: bool = False, partial_id: str | None = None) -> ET.ElementTree:
     """
     Create an ImageMetadata-v1.2 XML Element tree
 
@@ -318,6 +392,8 @@ def create_metadata_xml(beginning_time: str, middle_time: str, ending_time: str,
       formatted datetime string for data end date time
     dataday
       string if format %Y%j for day of year the data represents
+    subdaily
+      boolean flag if product is subdaily (if True, add DataDateTime to metadata)
     partial_id
       partial id associated with data
 
@@ -334,7 +410,10 @@ def create_metadata_xml(beginning_time: str, middle_time: str, ending_time: str,
     ET.SubElement(imagery_metadata, "DataStartDateTime").text = beginning_time
     ET.SubElement(imagery_metadata, "DataMidDateTime").text = middle_time
     ET.SubElement(imagery_metadata, "DataEndDateTime").text = ending_time
-    ET.SubElement(imagery_metadata, "DataDay").text = dataday
+    if subdaily:
+        ET.SubElement(imagery_metadata, "DataDateTime").text = beginning_time
+    else:
+        ET.SubElement(imagery_metadata, "DataDay").text = dataday
     if partial_id:
         ET.SubElement(imagery_metadata, "PartialId").text = partial_id
 

bignbit/get_collection_concept_id.py

@@ -33,7 +33,11 @@ def process(self) -> dict:
         collection_shortname = self.config['collection_shortname']
         cmr_provider = self.config['cmr_provider']
         cmr_environment = self.config['cmr_environment']
-        collection_id = get_collection_concept_id(collection_shortname, cmr_provider, cmr_environment)
+        dataset_config = self.input['datasetConfigurationForBIG']['config']
+        # Use override for collection concept id from dataset config if provided
+        collection_id = dataset_config.get('concept_id')
+        if collection_id is None:
+            collection_id = get_collection_concept_id(collection_shortname, cmr_provider, cmr_environment)
         self.input['collection_concept_id'] = collection_id
         return self.input
 

examples/cumulus-tf/bin/fake_gitc.py

@@ -14,7 +14,7 @@ def handler(event, context):
     logger.info(f"Received event {json.dumps(event)}")
 
     for message in event["Records"]:
-        response_topic_arn = message['messageAttributes']['response_topic_arn']
+        response_topic_arn = message['messageAttributes']['response_topic_arn']['stringValue']
         message_body = loads(message["body"])
         logger.info(f"Processing message {json.dumps(message_body)}")
 

examples/cumulus-tf/browse_image_workflow.tf

@@ -21,7 +21,7 @@ module "bignbit_module" {
   count = 1
 
   stage  = var.bignbit_stage
-  cmr_environment = var.bignbit_stage
+  cmr_environment = var.bignbit_cmr_environment
   prefix = var.prefix
 
   data_buckets = [aws_s3_bucket.protected.id, aws_s3_bucket.public.id, aws_s3_bucket.private.id]

examples/cumulus-tf/tfvars/sit.tfvars

@@ -1,5 +1,6 @@
 # stage: dev, sandbox, sit, uat, prod
 bignbit_stage = "sit"
+bignbit_cmr_environment = "UAT"
 prefix = "podaac-sit-svc"
 
 gibs_region="mocked"

examples/cumulus-tf/tfvars/uat.tfvars

@@ -1,5 +1,6 @@
 # stage: dev, sandbox, sit, uat, prod
 bignbit_stage = "uat"
+bignbit_cmr_environment = "UAT"
 prefix = "podaac-uat-svc"
 
 gibs_region="us-east-1"

examples/cumulus-tf/variable.tf

@@ -23,6 +23,12 @@ variable "bignbit_stage" {
   type = string
   description = "environment used for tagging resources"
 }
+
+variable "bignbit_cmr_environment" {
+  type = string
+  description = "cmr environment used for bignbit"
+}
+
 variable "big_throttled_execution_limit" {
   type    = number
   default = 50