Scoping: next steps for search and download APIs

[sgreenbury]

This issue describes the current approach, features, and APIs for searching metadata and downloading data and aims to:

1. help identify if the current structure should be refined for the next planned features
2. document the current code structure.

Related issues

• Scoping: Search method for data. #3
• Harmonise data and metrics commands #57
• Ideas: possibly improvements to the output of the metrics command #81

Current structure and design

The diagram describes the relationships between the types and concepts that are currently implemented.

graph TB

    subgraph CLI
        subgraph CombinedParamsArgs
            SearchParamsArgs:::type
            DownloadParamsArgs:::type
        end
        Recipe:::concept
    end
    
    subgraph DataRequest API
        DataRequestSpec:::type
        
        GeometrySpec["GeometrySpec"]:::type
        MetricSpecs:::type
        Years:::type
    end
    
    subgraph Combined API
        Params:::type
        RegionSpecs:::type
    end
    subgraph SearchAPI["Search API"]
        SearchParams:::type    
    end
    
    subgraph DownloadAPI
        DownloadParams:::type
        Config:::type
        SearchResults:::type
    end
    Recipe -- Deserializes to --> DataRequestSpec 
    DataRequestSpec -- 'TryFrom' --> Params 
    
    DataRequestSpec  --- GeometrySpec
    DataRequestSpec  --- RegionSpecs
    DataRequestSpec  --- MetricSpecs
    DataRequestSpec  --- Years
 
    Params --- SearchParams
    Params --- DownloadParams
    
    subgraph Outputs
        Metrics:::concept
        Geometries:::concept
    end
    SearchParamsArgs -- 'From' --> SearchParams
    CombinedParamsArgs -- 'From' --> DownloadParams
    SearchParams -- '.search()' --> SearchResults
    DownloadAPI -- '.download()' --> Outputs
    
    subgraph Key["Key"]
        Concept:::concept
        Type:::type
    end

    classDef api fill:#cbd5e8,stroke:#333,stroke-width:2px
    classDef type fill:##cbd5e8,stroke:#333,stroke-width:2px
    classDef field fill:#fdcdac,stroke:#333,stroke-width:2px
    classDef concept fill:#f4cae4,stroke:#333,stroke-width:2px
    classDef methods fill:#e6f5c9,stroke:#333,stroke-width:2px

Loading

Search API SearchParams

Design goal: aiming to enable flexible, composable and expressive searching of the metadata structure

Current features implemented/planned for search:

• Text
• Year ranges
• Metric IDs
• Geometry Level
• Data publisher
• Country
• Source data release
• Source metric ID
• RegionSpec:
  • Bounding Box
  • Named Area
  • Polygon
• Source Download URLs (Ideas: possibly improvements to the output of the metrics command #81)
• Match type (Ideas: possibly improvements to the output of the metrics command #81)
• Case sensitivity (Ideas: possibly improvements to the output of the metrics command #81)
• Recipe

Download API (DataRequestSpec / SearchParams and DownloadParams)

Design goal: to enable the download of data products in various customizable output structures and formats that various users may require

Current features implemented/planned for download:

• Multiple metrics
• Various geometries
• Multiple years and year ranges
• Region Spec
  • Bounding Box
  • Named Area
  • PolygonBBox
• Recipe
• A list of recipes/data request specs
• Transformations: applying a query, formulae, additional metadata to be returned, column renaming, etc
• Multiple geometries
• Other data types:
  • Categorical data (e.g. spatial signatures)
  • Synthetic populations
  • Origin-destination
• Spatial interpolation

User stories

Some example user stories with use cases to help motivate modifications to the design.

15m neighbourhood tool

See PR10

Other enhancements

• Add bon for implementing a builder for the various params (e.g. for SearchParams)