Optimal Configuration for Running Planetiler on High-Performance Server

[mapgate]

I have a server with the following specifications:

CPU: 2 x Intel Xeon E5-2696 v3 (36 cores, 72 threads total)
Memory: 512 GB
SSD: ADATA LEGEND 960 MAX 2TB

I need to generate the entire planet using Planetiler. How should I configure and run Planetiler for the best performance on this server?

[msbarry]

That machine is plenty big so you've got a few options:

The default recommendation for generating the planet should work fine: -Xmx30g with --storage=mmap --nodemap-type=array.

You could also try storing node locations in ram to see if it speeds things up at all: -Xmx300g with --storage=ram --nodemap-type=array.

Or you could mount ~450gb as ramdisk and try generating the whole thing on that with -Xmx30g and --storage=mmap --nodemap-type=array

Let me know how that works out, it should probably finish in <1 hour or so.

[mapgate]

Thanks for the help!

I ran the command: java -Xmx300g -XX:MaxHeapFreeRatio=40 -jar planetiler-openmaptiles.jar --osm-path=planet-240729.osm.pbf --output=output.pmtiles --building-merge-z13=false --fetch-wikidata --nodemap-type=array --storage=ram --force

It took a long time to complete.

Here’s the execution log:
https://gist.githubusercontent.com/mapgate/e50d8ebd893c4e1b219328b903c306db/raw/15ea4648710e1e12affe356dea9cc1ab62e59557/gistfile1.txt

What am I doing wrong? I’m open to any suggestions to speed up the planet generation process.

[mapgate]

The command ended with an error
java -Xmx300G -XX:MaxHeapFreeRatio=40 -jar planetiler-openmaptiles.jar --osm-path=planet.osm.pbf --output=output.pmtiles --building-merge-z13=false --nodemap-type=array --storage=ram --threads=72 --force

Here is its log:
https://gist.githubusercontent.com/mapgate/85ebef2a57b35b1f54434a686ad5ca8e/raw/7efba75e2d270bdd4c48bb7190f9a3c075d1ee64/gistfile1.txt

Here is the hs_err_pid15560.log file:
https://gist.githubusercontent.com/mapgate/411347cd2f87ac462018389777229701/raw/5184fc77ad22d56149945bb5337ad4b73a486b7f/gistfile1.txt

[msbarry]

Thanks for the info, looks like --threads=72 helped, not sure about that jvm crash though, did it end up succeeding after that or does it always fail at the same spot?

[mapgate]

That's odd... one thing I notice is that it uses 36 threads. You could try overriding that with --threads=72 and see if it's able to utilize them all. Also the archive step only does 13 cpu minutes of work across all cores but it takes 2 hours. It's like the write is blocking. Only time I've seen that before is writing to a very slow disk.

If you're able to attach visualvm to it and use the cpu sampler during the archive phase that would explain exactly what's going on.

I am attaching sampler data
data.zip

[msbarry]

Thanks! Looks like that captured about a minute of activity, do you know what zoom level it was working on? From what I can see there it spent nearly 100% of the time gzipping the tiles. I know this uses native zlib code on each platform, so there may be some performance differences there.

The initial log file you shared shows that threads were 0% utilized starting around z10/z11 - was this capture from that or earlier on when the logs reported CPU was being fully utilized?

[msbarry]

Someone else reported the exact same planetiler workload on the same machine took 3.5 hours running on windows, and 2.5 hours running on linux.

[wipfli]

I was able to run the protomaps basemap in 1h35min on my new machine. Details of the run are here: https://gist.github.com/wipfli/17e735c79461cf12019d14ac4ed32b67#run-3