martin-cp: better nullness/bbox handling for vastly increased performance

[CommanderStorm]

Currently, when martin-cp sees the following, the green area will be attempted to copy.

This is suboptimal, as the green area cannot ever have tiles.
We should either

• if the query is taking very long (more than 10 minutes, or something) calculate the bboxes for each source and see if we can skip tiles
• use the bboxes of sources to pre-filter sources (This might also apply to martin itsself, I have not checked)

The motivation for this is the following observation:

[todtb]

@CommanderStorm, this is something I've thought a bit about but wasn't sure if there is a universal approach on how to handle this. I think it also depends quite a bit on the extent of the data (global vs local) and how the mbtiles are being used (offline, "cache" of static data, etc).

Global vs local

For small, locally constrained data sets, I've been able to use existing ST_ functions to generate the list of bboxes, so it should be possible to use similar logic to eliminate generating empty tiles.

However, for global data sets (think adm0) bbox unions quickly become very large and are further complicated by geometries spanning the antimeridian (US, RU, KI, FJ, possibly more depending on sources/definitions and their bboxes).

Perhaps the adm examples are more niche than I imagine, but the fundamental challenge remains.

Approaches

The way I'm currently using the tool is simply passing in the bboxes I'm concerned with:

martin-cp  --output-file /tiles/out.mbtiles --mbtiles-type normalized \
"--bbox=..." \
"--bbox=..." \
"--bbox=..." \
...
--config /config/mbtile_config.yaml --source source --on-duplicate override

These bboxes are generated, but manually validated or adjusted as needed.

I imagine you were thinking about something a little less manual, but I'm wondering if you can describe a rough heuristic of what you were thinking?

[CommanderStorm]

Yea, pretty much what I was thinking.
Having to add those boxes for acceptable performance is not what we want, way too error-proawn. 😅

My thinking was a bit more simple/naiive.
taking one bbox per source would in fact already be an improvement and we have this data in the tilejson.

[todtb]

Sure, makes sense. Simple is always a good starting point. I can give this a go and you can assign me.

Re: global vs local - I've been more in the "global" data scenario so all of my sources are a base of [ -180.0, -90.0, 180.0, 90.0 ] in the config.yml thus there's not much savings to be had there. But as mentioned, I could see this being a massive improvement for smaller "local" data sets.

For what it's worth, I'm using mbtiles for static data to ease pressure on the db. That is how I'm getting away with the multiple bbox arg approach, since I really only need to figure those out once for the command to generate the tiles.

For now, I think accepting the vec<bounds> via --bbox with a fallback to the configured bounds in the config.yml sounds reasonable.