Extending OGRFieldDefn with SetUnitName()/GetUnitName() ?

[rouault]

We have no standardized API to get or set the unit of a numeric field, which may crucial to be able to process it correctly. Users' workarounds may be to add a suffix like "_m" or "_celcius" to the field name, use the field comment to add the unit as free text, add an extra field "foo_unit" whose value is the unit name, or give up.

That said very few formats have built-in ways of communicating that information. I can only think of:

• netCDF: through the units attribute of a variable (https://cfconventions.org/Data/cf-conventions/cf-conventions-1.13/cf-conventions.html#units)

• IHO S-101: the XML feature catalog contains <S100FC:uom> elements for simple attributes that have a unit

• some shapefiles (coming from Esri software) may be accompanied by a .shp.xml side-car file containing FGDC (Federal Geographic Data Committee) metadata. For example https://pubs.usgs.gov/sim/2840/DATA/bathy/bathy.shp.xml with the <attrunit> element:

                        <attr>
                                <attrlabl>CONTOUR</attrlabl>
                                <attrdef>Depth of bathymetric contour. Values are in meters above sea level, so all values are negative. Double of width 13, but values are integer. Another attribute "INTERVAL" contains the same data stored as an integer.</attrdef>
                                <attrdomv>
                                        <rdom>
                                                <rdommin>-270</rdommin>
                                                <rdommax>-5</rdommax>
                                                <attrunit>meters</attrunit>
                                                <attrmres>.000</attrmres>
                                        </rdom>
                                </attrdomv>
                                <attrmfrq>5 m</attrmfrq>
                                <attrtype Sync="TRUE">Number</attrtype>
                                <attwidth Sync="TRUE">9</attwidth>
                                <atnumdec Sync="TRUE">3</atnumdec>
                                <attrdefs>USGS</attrdefs>
                        </attr>

  That said looking a bit at online Esri documentation it doesn't seem their GUI makes it easy to define a unit for a field.

It would seem desirable to me that the GDAL API helps unlocking that restriction, so GIS (like QGIS) could offer such a capability to their end user. We'd particularly need to address GeoPackage and PostgreSQL cases.

For Geopackage, while extending gpkg_ prefixed system tables to add a new column to store the unit name could be done in theory, it is known to cause issues to some implementations that use automatic mapping of table structure to code. This leaves us with 2 candidates: the gpkg_metadata/gpkg_metadata_reference tables used by the metadata extension, or the gpkg_data_columns table used by the schema extension. Both are implemented by GDAL. metadata is mapped to GDAL dataset and layer metadata. schema is used to store field alias and comment, as well as field domains.

For metadata , that could look like:

INSERT INTO gpkg_metadata (id, md_scope, md_standard_uri, mime_type, metadata)
                   VALUES (1, 'attributeType', 'http:/gdal.org', 'text/xml',
                           '<FieldDefinition><UnitName>meter</UnitName></FieldDefinition>');
INSERT INTO gpkg_metadata_reference (reference_scope, table_name, column_name, row_id_value, timestamp, md_file_id, md_parent_id)
    VALUES ('col','my_table_name','my_column_name',-1,strftime('%Y-%m-%dT%H:%M:%fZ','now'),1,NULL);

For schema , we could hijack the description field by appending a " | Unit={value}" suffix (or just "Unit={value}" if there is no user set comment/descripion). The GDAL implementation on reading would recognize it to call SetUnitName({value}), and would remove the suffix before setting the rest of the content to SetComment().

Both ways are forward & backward compatible. The schema way is simpler to implement.

For PostgreSQL, we could actually reuse the same idea than the above hijack. We already have code that translates between PostgreSQL COMMENT ON COLUMN and OGRFieldDefn::SetComment().

And also for GML and FlatGeobuf that both support comments.

Which values to put in SetUnitName() ? We could encourage the use of strings recognized by UDUnits, as done in netCDF CF, but in no way enforce it.

Happy to hear thoughts of readers of this ticket whether adding such capability makes sense and whether the above rather hacky solutions are acceptable.

I was also wondering about a more general OGRFieldDefn::SetMetadata()/GetMetadata() capability, as we have already on dataset, raster bands and layers. But I'm not sure what users would need to put in that beyond the existing information (name, type, default value, alias, comment, field domain, nullability, uniqueness) and unit.

[jratike80]

Which values to put in SetUnitName() ?

I don't know much about The Unified Code for Units of Measure https://ucum.org/ucum, or QUDT https://qudt.org/, mainly that they exist, and that they are referenced in the OGC API Coverages draft https://docs.ogc.org/DRAFTS/19-087.html#_description_of_the_coverage_fields.

I feel that the OGCCov standard is getting very complex and therefore it may be that it never comes very popular, but perhaps this part of the standard draft with unitLang+unit approach makes sense.

[ctoney]

R package sf for Simple Features supports units via the separate units package also maintained by @edzer (providing an interface to UNIDATA udunits). The repo links to a paper that's mostly R specific but has some background information. Edzer might have thoughts on this.

[rouault]

I would prefer avoiding enforcing one unit naming convention over other one. This seems to me to be non realistic. If we can have some form of information accessible to users, that would be already a win. And complicating the API to mention the convention used like in OGC API Coverage seems to me to be also overkill. At that point just use ISO 19115 / ISO 19139 metadata and enjoy the full verbosity...

My fundamental question is more if it makes sense to "abuse" existing formats to stuff that information into them when there is almost no native provision for it. I don't anticipate GDAL hacks to be used by non-GDAL data producers (or it must be something essential missing, like the GDAL TIFF NODATA tag that is also used outside of the GDAL ecosystem), so that would restrict the usefulness of that only to a pure GDAL based ecosystem. But I'm not sure to understand why that omission is a recurring pattern in file formats. Either almost everybody seems to have forgotten that during design, or they rejected it for other reasons ("if we add that, then why not also adding X, Y or Z ?"). But I clearly remember physics teachers threatening pupils of a 0 mark to any numeric result without the unit mentioned :-)

About the API naming, in the raster side, we have GDALRasterBand::GetUnitType(). I've always found that naming a bit awkward because it is fundamentally the unit name that is reported. I assume the "type" here was by opposition to a value / conversion factor. But for me unit type and unit name are different. The unit type correspond to the unit name "meter" is "length".

In the multidimension API, we have GDALMDArray::GetUnit(). Maybe that short version could also be used for OGRFieldDefn?

[edzer]

Thanks for tagging me here, @ctoney! Most of what I learned about units comes from the SI brochure from the BIPM. Pick the english version to avoid you have to skip the 110 pages French. It contains lovely definitions such as

The value of a quantity is generally expressed as the product of a number and a unit. The unit is a particular example of the quantity concerned which is used as a reference, and the number is the ratio of the value of the quantity to the unit.

Following that logic, GetUnit() and SetUnit() are the right expressions for what you do if you get or set a unit of a quantity.

My initial motivation for writing the units package for R is that if we compute measures on geometries (length, area, distance) we know the unit of the result if we know the coordinate reference system of the geometries, and we can pass this on to the user, to avoid confusion (in addition, we can pick the right algorithm depending on whether coordinates are geodetic or Cartesian). Using units may eliminate a whole class of potential scientific programming mistakes.

When looking for useful, working code supporting units I found udunits, which comes from the same place as NetCDF. It comes with a database with units and conversion constants, does units parsing, units conversion, units compatibility checking, computes derived units, basically all you need, + wide support in open source software distributions. I also looked at ucum, but didn't find much useful there. Udunits is pretty much prescribed for CFconventions, and hence used in CDL & Zarr (I assume). Despite that, in the wild one often finds NetCDF (or similar) arrays with units that do not follow this. Back then I didn't find anything similar (beyond POC) around ucum. ucum and udunits are obviously somewhat similar, but not identical / compatible afair.

I think it is a good idea to not prescribe how to describe / encode units. Anyone who is serious about units and wants to make them useful will pick their ecosystem. A reason that most database systems have no built-in support for units of measurement may be that they are dominantly developed for domains mostly not concerned with physical or chemical quantities.

I agree that field domains might be a good place to put them, in the existing OGR API, as they describe properties of feature attributes. Still, if every attribute has its own unit each would call for its own field domain, which feels convoluted, but may just be an implementation detail?

[rouault]

I agree that field domains might be a good place to put them, in the existing OGR API, as they describe properties of feature attributes. Still, if every attribute has its own unit each would call for its own field domain, which feels convoluted, but may just be an implementation detail?

Field domains are implemented only in 2 vector drivers, OpenFileGDB and GeoPackage, because they are actually the only formats supporting that concept. File Geodatabase doesn't support units (despite Shapefile apparently allowing FGDC metadata which had a provision for them). GeoPackage has no direct support for unit in field domains (we could likely "extend" it a bit in a way similar to what I sketched for fields). So seeing that, I don't see field domains being a very practical avenue.

[edzer]

So seeing that, I don't see field domains being a very practical avenue.

I understand what you say, but e.g. assuming that attribute (field) X has an associated field in the same table called X_unit with the unit of X is also ugly, in particular because its value should be a single constant. But I won't die on this hill: doing anything that allows you setting units of something is better than not doing it. I haven't looked at databases distributed by physicists.

[jratike80]

I am trying to review prior ideas, and the draft of the OGC API Features Schema extension https://docs.ogc.org/DRAFTS/23-058r1.html deals with units as well. In a verbose way:

...
      "title" : "Atmospheric Pressure [hPa]",
      "description": "The pressure exerted by the weight of the air above it at any point on the earth's surface. At sea level the atmosphere will support a column of mercury about 760 mm high. This decreases with increasing altitude. The standard value for the atmospheric pressure at sea level is 1.01325 hectopascal.",
      "type" : "number",
      "x-ogc-definition": "https://qudt.org/vocab/quantitykind/AtmosphericPressure",
      "x-ogc-unit": "https://qudt.org/vocab/unit/HectoPA",
      "x-ogc-unitLang": "QUDT",
...

[jratike80]

For Geopackage, while extending gpkg_ prefixed system tables to add a new column to store the unit name could be done in theory, it is known to cause issues to some implementations that use automatic mapping of table structure to code. This leaves us with 2 candidates: the gpkg_metadata/gpkg_metadata_reference tables used by the metadata extension, or the gpkg_data_columns table used by the schema extension.

Imaging myself as a user who reads the GeoPackage with plain SQL tools, with the schema approach it would be handy to find the unit from a single gpkg_data_columns table. But maybe I would like to extract the tables, columns, and units with SQL. AI suggested something like the following, at it seems to work if the unit part is at the end of the description and if "Unit" does not appear earlier in the description text. I am not sure if this is a reliable method.

SELECT table_name, column_name,
substr(description,
              instr(description, 'Unit') + 5
FROM gpkg_data_columns;

The metadata approach feels logical and explicit but a bit harder to query manually.

[edzer]

We discovered, after writing the units R package, that many people have data with mixed units: variables in tables where each value has an associated unit, not constant for the variable as a whole. A field with units would allow this; a GetUnit() on the field level would have to deal with that possibility.

[rouault]

variables in tables where each value has an associated unit, not constant for the variable as a whole.

really... ? sounds like risky dataset design

a GetUnit() on the field level would have to deal with that possibility.

Users wouldn't just fill that field in that situation.

[jratike80]

One thing to remember is to document and warn users that manipulating the values can set the data and the units out of sync. Converting square meters into hectares by x/10000 would not update the units automatically.