Can't handle 80km maps, out of memory regardless if xmx/xms has been specified

[zordard]

[MarcusOnGit]

I made a 80km by using really high settings:
java -jar -Xms1G -Xmx16G .\NeroxisGen_1.11.0.jar --map-size=80km --spawn-count=4 --num-teams=2
the spawn and teams settings are just leftover from other attempts. Maybe it has influence on memory usage ... don't know.
I use Win10 and 32GB Mem and it was used up to 21GB in total (just a few browser windows open).
Hope that helps.

java -jar -Xms1G -Xmx16G .\NeroxisGen_1.11.0.jar --map-size=80km --spawn-count=4 --num-teams=2
neroxis_map_generator_1.11.0_375qqfyxcujha_araae
Starting pipeline
Pipeline completed!
Map generation done: 410208 ms
File export done: 2233 ms
Saving map to D:\Data\gme\SupComMapGen\.\neroxis_map_generator_1.11.0_375qqfyxcujha_araae
Seed: -2307241488832720272
Spawns: 4
Map Size: 4096
Num Teams: 2
Biome: WONDER
Land Density: 0.35433072
Plateau Density: 0.9055118
Mountain Density: 0.14173229
Ramp Density: 0.1496063
Reclaim Density: 0.30708662
Mex Density: 0.9448819
Terrain Symmetry: POINT6

Symmetry Settings: SymmetrySettings(terrainSymmetry=POINT6, teamSymmetry=POINT2, spawnSymmetry=POINT2)
Style: BASIC
TerrainGenerator: BasicTerrainGenerator
TextureGenerator: BasicTextureGenerator
ResourceGenerator: BasicResourceGenerator
PropGenerator: BasicPropGenerator
DecalGenerator: BasicDecalGenerator

[Rikitik]

Yes 40KM go fine 80KM crash OutofMemory

[Rikitik]

Has there been any interest in solving the memory footprint for 80km maps?

[Rikitik]

I just tried the new 1.14.0 and the folowing command

java -jar -Xms1G -Xmx16G .\NeroxisGen_1.11.0.jar --map-size=80km --spawn-count=8 --num-teams=2

succesfully created a 80km map on the same pc that previously crashed with "out of memory" it took close to an hour tho. I'll try on a better specced PC as well as few more times on the old one, maybe it got fixed by proxy

[clivepaterson]

Well I've thought about it before.
But I don't think there's much opportunity in the code to reduce the memory footprint.

I'm surprised you managed to generate an 80k, I've only tried 40k before.
And 1 hour wow that took a while, but it's not surprising that it took this long.

In FAF, when players join a MapGen they all have to generate the map, and players might be using a potato PC with small amount of RAM etc. I think the requiremnts are about 2 Gb of RAM for 20K and below it this runs reasonably fast.

So a 20K limit seems reasonable for FAF.

As for the why:
So mapgen makes many layers of the map, and performs many functions/calculations over each of those layers.
And the layers all get added together at the end.

Each layer can have values that are boolean / float or a Vector.
On a 20k map these are the total memory usages:
boolean: 0.16Mb
float: 4Mb (On an 80K map this allocates 64Mb of memory, roughly equiverlent to a 4096 x 4096 png image)
Vector3: 12Mb
Vector4: 16Mb

So anyway, I added a bit of code to see how many of these layers are being allocated during an average mapgen
Key: BooleanMask Value: 205
Key: NormalMask Value: 5
Key: Vector2Mask Value: 23
Key: Vector4Mask Value: 9
Key: IntegerMask Value: 2
Key: FloatMask Value: 198

So just looking at the float masks which seem to be used the most:
20K map:
198 x 4Mb = 792Mb
80k map:
198 x 64Mb = 12,672Mb (12Gigs)

And with performance, the size matters a lot I think.
Modern CPU's have like ~4mb of L2 cache and ~32Mb of L3 cache.. L1 cache is tiny (like 32k) but it's also the fastest)

So with a 20k map, the mapgen is probably using a lot of L2 cache and L3 cache. (Which is faster)
A 5K map might even use some L1 cache (so I'm guessing it's exponentially faster)

But an 80K map might not use any L1/L2/L3 cache at all.
I'm pretty sure it has to perform all it's many CPU operations over 12Gig's memory without much opportunity for CPU caching.

[Rikitik]

Well there seems to be a scaling issue a 2K map(40km) takes 5 minutes or less on a 2014 I7 with 16GB RAM Win 7
a 4K map(80/81km) takes around 1 hour, would expect 20-30 minutes tops, this is pure observational no benches so give or take

The same mapgens on an 2023 I9 16GB win 10 takes 2-3 minutes for 2K(40km) and 20-30 for the 4K(80KM) i can give a bit more than 16GB on the new one to see if it helps, i have not written any benches never worked with Java before)