CMA Messages and Testing

As bignbit is a Cumulus Post-Ingestion module, its primary input is a Cumulus Message Adapter (CMA) JSON object. To start, DAACs receive Cumulus Notification Messages (CNM) from mission SDS or SIPS when a new granule is delivered to them. The CNM JSON object indicates the remote location of the granule (such as on a university server or in another S3 bucket) so that it can be uploaded to the Earthdata cloud, but does not contain the required information for running bignbit. CMA messages are passed between cumulus modules to enable further processing on a granule, such as the data transformations executed by bignbit. A CMA message is created after a granule is uploaded, and contains DAAC-specific information.

The basic structure of a CMA is as follows:

• cumulus_meta: Contains metadata about the Cumulus deployment where the job was performed. This information is not required for testing.
• meta: Contains metadata about the job. This includes the S3 buckets used by the cumulus deployment, the CMR service from which to find metadata about the granule, and information about the collection.
• payload: Contains a granules key with a list of all files associated with this job. You might assume this would just be the granule itself, but in general it can include an md5 checksum, a CMR metadata JSON file, and other ancillary files.
• exception: Any exception from the previous step in the Cumulus process, this should always be "None"
• task_config: Optionally supply pointers to relevant information within this message. This is handled internally by the lambdas but you may need to supply it yourself when writing unit tests.

Integration Testing: Sending test CMA messages to a bignbit deployment

1. cumulus_meta is not needed by bignbit so it can be left empty.
2. In meta, the buckets are based on a set naming convention for the deployment, so they can be left empty when testing a deployed bignbit instance (SIT/UAT). The cmr block contains relevant information for the lambdas to find the granule_umm_json and collection_id. clientId should always be POCLOUD, but cmrEnvironment should be "OPS" if testing a dataset that is in production (you can still use production datasets in SIT), and "UAT" otherwise. Any PODAAC dataset will have a provider of POCLOUD, but in general this is end part of a CMR collection Id. You can also check this value by checking the provider-id field in the UMM-C metadata.

    "cmr": {
      "clientId": "POCLOUD",
      "cmrEnvironment": "UAT",
      "provider": "POCLOUD"
    },

3. The collection information in meta only needs a name and optionally a version.
4. For the payload we need to create one object in the granules array, it should have the following format:

  "payload": {
    "granules": [
      {
        "granuleId":"<fileName with no extension",
        "provider":"<provider-id>",
        "cmrLink":"https://cmr.uat.earthdata.nasa.gov/search/concepts/<granule concept-id>.umm_json",
        "cmrConceptId":"<granule concept-id"
      }
    ]
  }

Full example

{
   "cumulus_meta":{
      
   },
   "meta":{
      "buckets":{
         
      },
      "cmr":{
         "clientId":"POCLOUD",
         "cmrEnvironment":"UAT",
         "provider":"LARC_CLOUD"
      },
      "collection":{
         "name":"TEMPO_NO2_L3"
      },
      "stack":"podaac-ops-cumulus",
      "provider":{
         
      }
   },
   "payload":{
      "granules":[
         {
            "granuleId":"TEMPO_NO2_L3_V03_20250422T114702Z_S003",
            "provider":"larc",
            "cmrLink":"https://cmr.uat.earthdata.nasa.gov/search/concepts/G1273455903-LARC_CLOUD.umm_json",
            "cmrConceptId":"G1273455903-LARC_CLOUD"
         }
      ]
   }
}

Unit Testing: Creating CMA messages for unit testing

1. CMA message should be enclosed in a "cma" object at the top level.
2. cumulus_meta can still be left blank. Everything else in the message should be enclosed in an event object. The event object needs a task_config:

"task_config": {
  "cumulus_message": {
    "input": "{$.payload}"
  }
},

3. meta inside the event object should contain a list of buckets. At minimum, you will need internal and private. internal contains dataset configs, and private contains the granules. The cmr metadata follows the same convention as above.
4. For the payload section, you will need to include all the information that bignbit appends to the payload along the way. This varies somewhat depending on the unit test you are designing for, but in general you will need to add granules with a list of files, granule_umm_json which is the UMM-G metadata, big which is a field pointing to the bucket with data (remember to upload the test granules to S3 yourself if they are new), and datasetConfigurationForBIG which contains the dataset config file for your data. See the example for details.

Full example

The granule_umm_json has been truncated.

{
    "cma": {
        "cumulus_meta":{
            
        },
        "event": {
            "meta":{
                "buckets":{
                    "internal": {
                        "name": "podaac-sit-svc-internal",
                        "type": "internal"
                    },
                    "private": {
                        "name": "podaac-sit-svc-private",
                        "type": "private"
                    }
                },
                "cmr":{
                    "clientId":"POCLOUD",
                    "cmrEnvironment":"UAT",
                    "provider":"LARC_CLOUD"
                },
                "collection":{
                    "name":"TEMPO_NO2_L3"
                },
                "stack":"podaac-uat-cumulus",
                "provider":{
                    
                }
            },
            "payload":{
                "granules":{
                    "files": [
                        {
                            "bucket": "podaac-sit-svc-private",
                            "fileName": "TEMPO_NO2_L3_V03_20250422T114702Z_S003.nc",
                            "granuleId":"TEMPO_NO2_L3_V03_20250422T114702Z_S003",
                            "provider":"larc",
                            "cmrLink":"https://cmr.uat.earthdata.nasa.gov/search/concepts/G1273455903-LARC_CLOUD.umm_json",
                            "cmrConceptId":"G1273455903-LARC_CLOUD"
                        }
                    ]
                },
                "granule_umm_json": {
                    "PGEVersionClass": {
                        "PGEVersion": "1.0.0"
                    }
                },
                "big": [
                    {
                        "bucket": "podaac-sit-svc-private",
                        "fileName": "TEMPO_NO2_L3_V03_20250422T114702Z_S003.nc",
                        "type": "data",
                        "key": "TEMPO_NO2_L3/TEMPO_NO2_L3_V03_20250422T114702Z_S003.nc"
                    }
                ],
                "datasetConfigurationForBIG": {
                    "config": {
                        "sendToHarmony":true,
                        "latVar":"latitude",
                        "lonVar":"longiude",
                        "timeVar":"time",
                        "subdaily":true,
                        "imgVariables":[
                            {"id": "product/vertical_column_stratosphere","title":"stratosphere nitrogen dioxide vertical column","units":"molecules/cm^2","min":"0.0","max":"5.0"}
                        ],
                        "height":2950,
                        "width":7750,
                        "variables":["time","latitude","longitude","product/vertical_column_stratosphere"]
                    }
                }
            },
            "task_config": {
                "cumulus_message": {
                    "input": "{$.payload}"
                }
            },
            "exception": "None"
        }
    }
}