Create an API to allow any input GI to be mapped to the Bedrock GI Database model

[JoostGevaert]

Split out GI data processing functions that were written for processing AGS (3) data into functions that are applicable to any type of GI data and functions that are only applicable to AGS (3) data.

One of the most important things added to achieve this is the Bedrock GI Mapping. brgi.mapping_models.BedrockGIMapping is an extensible object model that contains which columns from the original data source should be mapped to bedrock-ge specific columns. Expressed as JSON this would look something like this for AGS 3 data:

{
  "Project": {
    "data": {
      "PROJ_ID": "J3573",
      "PROJ_NAME": "Project_Name XYZ ",
      "PROJ_LOC": "Kai Tak, Kowloon City District",
      "PROJ_CLNT": "ARCHITECTURAL SERVICES DEPARTMENT",
      "PROJ_CONT": "GAMMON CONSTRUCTION LIMITED",
      "etc": "..."
    },
    "project_id": "PROJ_ID",
    "horizontal_crs": "EPSG:2326",
    "vertical_crs": "EPSG:5738"
  },
  "Location": {
    "data": {
      "HOLE_ID": ["BH 1", "BH 2", "..."],
      "HOLE_TYPE": ["RCG", "RCG", "..."],
      "HOLE_NATE": [838144.50, 838083.31, "..."],
      "HOLE_NATN": [820697.61, 820670.84, "..."],
      "HOLE_GL": [5.97, 5.52, "..."],
      "HOLE_FDEP": [38.84, 43.55, "..."],
      "etc": ["...", "...", "..."]
    },
    "location_id_column": "HOLE_ID",
    "easting_column": "HOLE_NATE",
    "northing_column": "HOLE_NATN",
    "ground_level_elevation_column": "HOLE_GL",
    "depth_to_base_column": "HOLE_FDEP"
  },
  "InSituTests": [
    {
      "table_name": "GEOL",
      "data": {
        "HOLE_ID": ["BH 1", "BH 1", "..."],
        "GEOL_TOP": [0.00, 0.10, "..."],
        "GEOL_BASE": [0.10, 0.50, "..."],
        "GEOL_DESC": ["CONCRETE slab.", "Dark grey (N3) and light grey (N7), angular fine to coarse GRAVEL...", "..."],
        "etc": ["...", "...", "..."]
      },
      "location_id_column": "HOLE_ID",
      "depth_to_top_column": "GEOL_TOP",
      "depth_to_base_column": "GEOL_BASE"
    },
    {
      "table_name": "ISPT",
      "data": {
        "HOLE_ID": ["BH 1", "BH 1", "..."],
        "ISPT_TOP": [12.00, 15.0, "..."],
        "ISPT_NVAL": [74, 60, "..."],
        "etc": ["...", "...", "..."]
      },
      "location_id_column": "HOLE_ID",
      "depth_to_top_column": "ISPT_TOP"
    },
    "etc..."
  ],
  "Sample": {
    "data": {
      "HOLE_ID": ["BH 1", "BH 1", "..."],
      "SAMP_TOP": [0.45, 12.00, "..."],
      "SAMP_REF": ["A", "1", "..."],
      "SAMP_TYPE": ["D", "SPTLS", "..."],
      "etc": ["...", "...", "..."]
    },
    "sample_id_column": "f'{SAMP_REF}-{SAMP_TYPE}-{SAMP_TOP}-{HOLE_ID}'",
    "location_id_column": "HOLE_ID",
    "depth_to_top_column": "SAMP_TOP"
    },
  "LabTests": [
    "We still have to find some HK AGS 3 Lab test data..."
  ],
  "Other": [
    {
      "table_name": "ABBR",
      "data": {}
    },
    {
      "table_name": "UNIT",
      "data": {}
    },
    {
      "table_name": "DICT",
      "data": {}
    }
  ]
}

The conversion from a brgi.mapping_models.BedrockGIMapping to a brgi.schemas.BedrockGIDatabase is then performed using the brgi.mapper.map_to_brgiu_db function.

In addition to the Bedrock GI mapping, several other issues have been addressed

• Separate AGS 3 and AGS 4 specific logic #8
• Rename transform to map or something like that #9
• Put Longitude, Latitude, EllipsoidalHeight in the LonLatHeight table rather than in the Location table #16
• gis_geometry.py -> geospatial.py. Simple Feature geometry for the In-Situ Tests and Samples is now interpolated along the 3D LINESTRING of the corresponding GI Location. The 3D LINESTRING of a GI location is for now still assumed to be vertically down, which is to be resolved with issue Support inclined boreholes #43.
• Remove duplicate rows / don't add rows with duplicate data to a Bedrock GI database when merging Bedrock GI databases: brgi.db_operations.merge_databases

This refactoring of the bedrock-ge API allows further developments:

• Validation of incoming AGS data against their ags_data_dictionary.json
• Using DuckDB instead of pandas for data processing

[konnerhorton]

I started working on some ags4 to brdb functionality then noticed this. Where you planning to add that to this as well? (what should I be focusing on so we don't duplicate efforts?)