Bignbit Project Overview

This wiki is intended to contain documentation about the architecture of the browse image workflow from the point of Cumulus ingestion through to delivery to GIBS. It will also contain procedures on the delivery process for new layers and a roadmap of the intended cloud migration of existing on-prem browse image layers produced by PODAAC.

Data Transformation

The above chart describes the key image transformation steps involved in delivering data to worldview. Although not marked, bignbit handles the two Harmony blocks in this diagram. It makes no attempt to explain metadata sources or the exact functionality of bignbit, but rather to show where data is accessible, in what file formats, and what software packages perform the conversions required to generate browse imagery. The process of creating a granule through to Worldview delivery is as follows:

1. A mission Science Data System (SDS) or Science Investigator-led Process System (SIPS) generates data at various levels of processing from L0 to L4, where L0 refers to raw spacecraft data, and L4 refers to derived data that may contain significant post-analysis. The format of this data is generally netCDF, HDF, or Cloud-Optimized GeoTIFF depending on the mission. Some science users have special access to SIPS or SDS outputs directly, but this is non-standard.
2. Cumulus ingestion is handled by the DAAC responsible for archiving the data. No data transformation is performed during Cumulus ingest, so the data remains in its original format at this stage. After this step, the data is available in Earthdata Search and can be accessed by any Earthdata user, although generally is only downloaded by expert or science users.
3. Bignbit runs as a Cumulus post-ingestion module that performs the data transformations required to deliver data to GIBS/Worldview. In actual implementation, bignbit is an AWS Step Function that nominally calls the Harmony API to produce browse images from granules. At least in terms of the way that bignbit uses it, Harmony takes in a granule and rest query parameters and outputs a browse image along with some metadata. Although the API calls themselves do not call out the Harmony module that gets called in the service chain directly, we know by the requested operation that net2cog is invoked whenever a netCDF granule is input and a COG, PNG, or JPEG is output -- so basically for any browse image.
4. HyBIG is a Harmony module maintained by the GIBS team that handles the conversion of COGs to PNG or JPEG format images. While GIBS can take in COGs directly, there are often palettization, band combination, and quantization steps that are required on a per-collection basis so that GIBS can create image mosaics without having to "care about" the details of the data.
5. Bignbit sends a message to GIBS Ingest and ideally receives a response, at which point GIBS runs a program called MRFgen to create an image mosaic in a format called Meta Raster Format (MRF). Each MRF can contain one or more browse images covering a specific time range. The MRF stores the different zoom levels of the map layer as tiled images that can be served by the GIBS backend. These tiles are accessed by Worldview to serve data to its millions of users.

Key relationships

• NetCDF/HDF Granules to Browse Images: Granules in netCDF or HDF format can have a one-to-many relationship with browse images. One netCDF or HDF can produce one or more browse images due to their internal variables, which are described by UMM-Var metadata. Presently bignbit supports a one-to-one mapping of variables to browse images, but this is not true in general.
• COG Granules to Browse Images: COG-type granules have a one-to-one relationship with browse images. They have no UMM-Var metadata. They may have internal colormaps that indicate how to render the data.
• Browse Images to GIBS Image Mosaics: Browse images have a many-to-one relationship with GIBS MRFs (also called image mosaics). Missions will often tile their data using a grid, but these tiles differ from those required for rendering map data on the web. Therefore, the "tiles" archived at the granule level often must be ingested together on the GIBS side and re-tiled into a size appropriate for web mapping applications.