Add gdal mdim get-refs algorithm by mdsumner · Pull Request #14677 · OSGeo/gdal

mdsumner · 2026-05-28T00:51:26Z

gdal mdim get-refs command.

Creates a vector layer (no geometry) that lists the path, offset, size and info of chunk byte references from a multidimensional raster array.

The emitted table also includes dim_0, ...dim_n fields that identify the chunk's address (and allows filter queries), and present a boolean for whether the chunk exists.

Metadata is also recorded in the layer for ARRAY_NAME, DTYPE, DIM_0{:N}, and .papszInfo items in the form CODEC_%s eg. CODEC_COMPRESSION, CODEC_FILTER.

AI tool usage

AI Claude assisted

Tasklist

Description

Usage: gdal mdim get-refs [OPTIONS] <INPUT> <OUTPUT>

Return byte references from a multidimensional raster source as vector/table layer.

Positional arguments:
  -i, --dataset, --input <INPUT>                       Input multidimensional raster dataset [required]
  -o, --output <OUTPUT>                                Output vector dataset [required]

Common Options:
  -h, --help                                           Display help message and exit
  --json-usage                                         Display usage as JSON document and exit
  --config <KEY>=<VALUE>                               Configuration option [may be repeated]
  -q, --quiet                                          Quiet mode (no progress bar)

Options:
  -f, --of, --format, --output-format <OUTPUT-FORMAT>  Output format [required]
  --array <ARRAY>                                      Name of the array, used to restrict the output to the specified array. [required]
  --overwrite                                          Whether overwriting existing output is allowed
  --co, --creation-option <KEY>=<VALUE>                Creation option [may be repeated]

Advanced Options:
  --oo, --open-option <KEY>=<VALUE>                    Open options [may be repeated]
  --if, --input-format <INPUT-FORMAT>                  Input formats [may be repeated]

For more details, consult https://gdal.org/programs/gdal_mdim_get_refs.html

Example

gdal mdim get-refs --array /Band1 autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc \
    /tmp/Band1.parquet --of Parquet

gdal vector info /tmp/Band1.parquet --layer Band1 --features
INFO: Open of `/tmp/Band1.parquet'
      using driver `Parquet' successful.

Layer name: Band1
Metadata:
  ARRAY_NAME=/Band1
  DTYPE=Byte
  DIM_0_NAME=y
  DIM_0_SIZE=20
  DIM_0_BLOCK=10
  DIM_0_CHUNKS=2
  DIM_1_NAME=x
  DIM_1_SIZE=20
  DIM_1_BLOCK=10
  DIM_1_CHUNKS=2
  CODEC_COMPRESSION=DEFLATE
  CODEC_FILTER=SHUFFLE
Geometry: None
Feature Count: 4
Layer SRS WKT:
(unknown)
dim_0: Integer64 (0.0)
dim_1: Integer64 (0.0)
present: Integer(Boolean) (0.0)
path: String (0.0)
offset: Integer64 (0.0)
size: Integer64 (0.0)
info: String (0.0)
OGRFeature(Band1):0
  dim_0 (Integer64) = 0
  dim_1 (Integer64) = 0
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 6108
  size (Integer64) = 86
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

OGRFeature(Band1):1
  dim_0 (Integer64) = 0
  dim_1 (Integer64) = 1
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 6194
  size (Integer64) = 74
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

OGRFeature(Band1):2
  dim_0 (Integer64) = 1
  dim_1 (Integer64) = 0
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 5959
  size (Integer64) = 74
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

OGRFeature(Band1):3
  dim_0 (Integer64) = 1
  dim_1 (Integer64) = 1
  present (Integer(Boolean)) = 1
  path (String) = autotest/gdrivers/data/netcdf/byte_chunked_multiple.nc
  offset (Integer64) = 6033
  size (Integer64) = 75
  info (String) = COMPRESSION=DEFLATE; FILTER=SHUFFLE

rouault

not a bad idea, but clearly this doesn't follow our LLM tool policy. The code has clearly not been read carefully. A lot of comments are hyper verbose, with typical "LLM plan garbage" references. I feel some of the tests are also a bit more verbose than what would be strictly needed (although it might just be my aversion for being exposed to prior verbosity)

rouault · 2026-05-28T01:04:44Z

+/******************************************************************************
+ *
+ * Project:  GDAL
+ * Purpose:  gdal "mdim get-refs" subcommand


get-refs is a bit mysterious until you read the doc.
brainstorming: get-chunk-location, chunk-loc

or chunk-info ?

list-chunks or extract-chunks - I don't have a really strong feel here (get-chunk-location is absolutely fine so I'll go with that unless a stronger candidate or position arrives)

rouault · 2026-05-28T01:05:36Z

+}  // namespace
+
+/************************************************************************/
+/*                        GDALMdimGetRefsAlgorithm()                  */


formatting issue. your pre-commit is not setup ?

it is setup, I will check

rouault · 2026-05-28T01:07:35Z

+bool GDALMdimGetRefsAlgorithm::RunImpl(GDALProgressFunc pfnProgress,
+                                       void *pProgressData)
+{
+    // ----------------------------------------------------------------------


all those "stage", "A1", "D6" etc are really LLM noise. Please remove in general.

And you can pretty much remove that particular 5-line comment for a 2 line effective code...

rouault · 2026-05-28T01:07:57Z

+    auto poSrcDS = m_inputDataset.GetDatasetRef();
+    CPLAssert(poSrcDS);
+
+    // A2. GetRootGroup(). Null root => driver lacks mdim support => fail with a


make comment more succinct (remove the ==> fail part .... that's obvious from the code)

rouault · 2026-05-28T01:08:41Z

+    auto poRootGroup = poSrcDS->GetRootGroup();
+    CPLDebug("MDIM-GET-REFS", "input: %s, root group: %s",
+             poSrcDS->GetDescription(), poRootGroup ? "present" : "NULL");
+


Suggested change

rouault · 2026-05-28T01:49:06Z

+
+* ``ARRAY_NAME`` — the fullname of the input array
+* ``DTYPE`` — the array's data type (e.g. ``Int16``, ``Float32``)
+* ``DIM_N_NAME``, ``DIM_N_SIZE``, ``DIM_N_BLOCK``, ``DIM_N_CHUNKS`` — for


make a global pass to replace UTF-8 dash by ASCII dash

rouault · 2026-05-28T01:49:28Z

+* Only a single array can be emitted (``--array`` is required).
+* Arrays without natural block size decline with a ``not chunk-enumerable`` error.
+* The inline data payload is not extracted as a binary field.
+* No geometry column is emitted.


Suggested change

* No geometry column is emitted.

rouault · 2026-05-28T01:50:40Z

+-----------
+
+* Only a single array can be emitted (``--array`` is required).
+* Arrays without natural block size decline with a ``not chunk-enumerable`` error.


not really a limitation IMHO (in the sense there's nothing we can do about that)

Suggested change

* Arrays without natural block size decline with a ``not chunk-enumerable`` error.

rouault · 2026-05-28T01:51:46Z

+
+   .. code-block:: bash
+
+       ogrinfo ocean_chunks.parquet -sql \


Suggested change

ogrinfo ocean_chunks.parquet -sql \

gdal vector info ocean_chunks.parquet --features --sql \

rouault · 2026-05-28T01:57:33Z

+
+
+@pytest.mark.require_driver("Parquet")
+def test_gdalalg_mdim_get_refs_parquet_boolean_subtype(tmp_vsimem):


does it exercice any code path not already exercised ? I don't think so

Add gdal mdim get-refs algorithm

aa69a6e

mdsumner marked this pull request as draft May 28, 2026 00:51

rouault reviewed May 28, 2026

View reviewed changes

apply review changes (some)

a68f158

	ogrinfo ocean_chunks.parquet -sql \
	gdal vector info ocean_chunks.parquet --features --sql \



		@pytest.mark.require_driver("Parquet")
		def test_gdalalg_mdim_get_refs_parquet_boolean_subtype(tmp_vsimem):

Uh oh!

Conversation

mdsumner commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI tool usage

Tasklist

Description

Example

Uh oh!

rouault left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mdsumner commented May 28, 2026 •

edited

Loading