Martin. Support for nested directory hierarchies and large numbers of PMTiles sources (100k+)

[pausto]

Hi! I'm evaluating Martin (v1.11.0) for serving weather tiles in a structure similar to Windy, where tiles are organized along several dimensions: forecast model × variable × forecast hour/timestamp. With a separate .pmtiles archive per combination, this naturally produces a deep directory tree and a very large number of files — on the order of 100k+.

A simplified layout looks like this:

data/
├── gfs/
│   ├── temperature/
│   │   ├── 202606300100.pmtiles
│   │   ├── 202606300200.pmtiles
│   │   └── ...
│   └── wind/
│       └── ...
├── ecmwf/
│   └── ...
└── icon/
    └── ...

I have two questions:

1. Recursive directory scanning

From my testing and reading the docs, a directory listed under pmtiles.paths seems to be scanned only at the top level — nested subdirectories are not discovered.

• Is that understanding correct — is tile-source directory scanning intentionally non-recursive?
• Is there interest in (or an existing option for) recursive discovery, so a single root path could pick up archives in nested subdirectories?

Right now my workarounds are to either list every leaf directory explicitly under paths, enumerate files individually under sources, or flatten everything into one directory. For a tree this large, none of these scale well.

2. Serving a very large number of PMTiles sources (100k+)

Is Martin designed and tested to handle on the order of 100k+ PMTiles sources from local disk? I'd love to understand the practical limits and trade-offs:

• Startup source discovery time and memory footprint as the source count grows
• /catalog behavior and response size at this scale
• Directory-cache (directory_cache / size_mb) sizing recommendations
• Hot-reload / filesystem-watch behavior when watching directories containing tens of thousands of files
• Whether fewer-but-larger archives (e.g. one PMTiles per model/variable covering all timestamps) is the recommended pattern over many small files

Any guidance on recommended patterns for this kind of high-cardinality, frequently-updated weather workload would be very welcome. Thanks!

[CommanderStorm]

Is that understanding correct — is tile-source directory scanning intentionally non-recursive?

Yes. Discovery is currently non-recursive.
This is because we never settled on how nested sources should map to source IDs.
There's a TODO/FIXME in the discovery code noting this, and we intentionally left it non-recursive rather than picking a naming scheme without discussion.
Should be easy enough to find and rewarding to fix 😉

I'd be happy to review a PR adding recursive discovery if it comes with a well-defined source ID mapping.

Is Martin designed and tested to handle on the order of 100k+ PMTiles sources from local disk?

We haven't specifically tested or benchmarked Martin at that scale.
The common deployment pattern is to serve a relatively small number of large PMTiles archives rather than hundreds of thousands of small ones.

It may work, but we don't have data on startup time, memory usage, catalog size, or filesystem watching with that many sources.
If you run into scalability issues - or have benchmark results to share - we'd be interested to see them.

Contributions to improve this use case are also welcome.

Any guidance on recommended patterns for this kind of high-cardinality, frequently-updated weather workload would be very welcome. Thanks!

We don't have any established recommendations for workloads like this, as it's not a use case I've seen often or worked with.
I just don't know the system requirements (what is "high" or "frequently", ...).

Martin is generally optimized around serving a relatively small number of large tile archives rather than a very large number of small ones, but whether that trade-off makes sense depends on your update frequency, storage layout, and access patterns.

If you end up evaluating approaches, we'd be interested in hearing about your findings. Real-world benchmark data for this kind of workload would be valuable.

[pausto]

Thanks — let me put some numbers on it from my side.

I just don't know the system requirements (what is "high" or "frequently", ...).

Forecast models publish new runs on an hourly cadence, and when a run lands, many fields get rewritten/replaced in a short window. In practice this can mean up to a few hundred file updates per minute during a refresh. At that rate I don't think filesystem-event watching is a good fit — the event volume (plus churn from temp files and atomic renames) tends to be noisy and easy to overwhelm.

What I'd actually want is periodic re-discovery for local directories: re-list the directory tree on a fixed interval (say once per minute) and diff against the previous snapshot to update the catalog. This is essentially what Martin already does for remote object-storage prefixes via reload_interval — just applied to local paths, as an alternative to (or instead of) the fs-watch mode. Does something like that already exist, or could it be exposed as an option?

I can't simulate 100k+ files yet, so realistically my first milestone is just confirming Martin is happy at 10-20k+ local PMTiles sources — if that works well, it's already a great outcome.

To estimate whether that's feasible, could you share — architecturally — roughly how much memory a single opened/registered PMTiles source consumes? I.e. the per-source footprint of the header + cached root directory + the object_store handle (excluding the shared tile cache, which I understand is bounded separately). With a rough per-source number I can extrapolate total memory for N sources and size the host accordingly.

Thanks again!

[CommanderStorm]

how much memory this consumes is hard to say since I don't know how much cacheing you want to allow, which is the primary driver here..