Support for DuckDB 1.4.0 as it now has ST_AsMVT and ST_AsMVTGeom implemented.

[ggaans]

Now that DuckDB 1.4.0 supports ST_AsMVT and ST_AsMVTGeom is would be great to have Martin connect to it like is does for PostGIS. This would provide a pathway for Parquet -> DuckDB -> Martin.

[CommanderStorm]

This should be much simpler to do, given that we now have refactored our interals to the martin-core-Architecture.

We already have an issue for geoparquet, so lets continue tracking this there.

• GeoParquet backend #1693

Would be open to a PR

[AbelVM]

I'd love to see this working regardless of geoparquet, just for the sake of DuckDB. So it can be a first-class Martin citizen like Postgres. Geoparquet support would be just a side effect of this issue, indeed.

So, IMHO, this issue is not a clean duplicate of #1693

For reference, two implementations of this DuckDB feature to serve tiles:

• Flask: https://gist.github.com/Maxxen/37e4a9f8595ea5e6a20c0c8fbbefe955
• R: https://gist.github.com/walkerke/c90ab6b8f403169e615eabeb0339b15b

[CommanderStorm]

which features of duckdb would you like to see explicitely?
All the non-parquet file formats that duckdb supports (https://duckdb.org/docs/stable/guides/performance/file_formats) are not ideal for generating tiles from.

[AbelVM]

I second that, but we might keep it agnostic to future-proofing the efforts. I mean, once the logic for DuckDB is implemented, the actual source of the data is mostly only at the FROM statement, regardless it is a geoparquet, an avro file, or just a db file.

Regarding the features... just mimicking the functionality Martin provides for Postgres would be great

[nyurik]

sure, but please keep in mind - every feature we add must be "useful" rather than just mimicking - otherwise we raise the cost of maintenance without benefits... grumble grumble grumble :)

[AbelVM]

My point is that once DuckDB (via spatial extension) supports MVT-related functions, it might be an in-place replacement in a stack where Martin plays as the tiles endpoint.

As an example, right now I have datasets with sizes in the hundreds of millions of features, and I need them to be available for :

1. Martin, to serve the info as tiles
2. Services that respond to both simple and complex search queries
3. Services that aggregate information on the fly

First requirement is fulfilled by Postgres, but the 2nd and 3rd would be better served by DuckDB.

On the other hand, as the life-cycle of those datasets is in the months range, I'd love to automate data deploys to static files while keeping the performance.

My 2 cents.

[CommanderStorm]

how would you like to store tha data on disk, likely geoparquet or flatgeobuf, right?

There are other things in the spatial exstension, but none for generating tiles resonably fast (unless i am missing some, possible).

[CommanderStorm]

Avro seems to need loading plugins, same as the spatial plugin.
performance would likely not be great due to the indexing and schema evolution, not sure.
Would need research.

[CommanderStorm]

All in all:
I think we can add it if

• it is okay code wise
• documented when the different formats this adds support for should be used
• and well tested.

For the workloads that I have this does not matter, so TUM won't invest the engineering here.
Maybe this is interesting for others, the new architecture should make adding things like this much more straigtforward.
I would love to review PRs into this direction.

[CommanderStorm]

To add this, the first step would be a design pass, to work out how this should be configured in the:

• CLI
• config file

Maybe also the which datasources does this enable and how do they fit into the already supported picture.