OsmAndMapCreator IndexOutOfBoundsException for big extracts #687
Description
Hello!
I have been trying to create .obf maps of .osm.bz2 extracts, and OsmAndMapCreator generate-obf works well for the following:
Switzerland.osm.bz2 (osm.bz2 692MB, obf 783MB)
But it crashes while writing route index when the input file is bigger than about 1GB (I am giving it -Xmx 25G of memory to use) on the following:
(germany) baden-wuerttemberg-latest.osm.bz2 (osm.bz2 854MB, unfinished obf 696MB)
(germany) bayern-latest.osm.bz2 (osm.bz2 1.2G, unfinished obf 889MB)
Here is the end of the log for baden-wuerttemberg:
INFO: Done 98.965 %.
Oct 07, 2023 6:15:34 PM net.osmand.impl.ConsoleProgressImplementation print
INFO: Task index_lo_level_ways: work total has changed to -1
Oct 07, 2023 6:15:34 PM net.osmand.impl.ConsoleProgressImplementation print
INFO: pack_rtree_map started after 2071665 ms - -1
Pack.packTree : Size of elmts: 10458811
Pack.packTree : Size of elmts: 3171796
Pack.packTree : Size of elmts: 1758434
Pack.packTree : Size of elmts: 168685
Pack.packTree : Size of elmts: 2345214
Pack.packTree : Size of elmts: 184318
Oct 07, 2023 6:18:17 PM net.osmand.impl.ConsoleProgressImplementation print
INFO: pack_rtree_transp started after 162578 ms - -1
Pack.packTree : Size of elmts: 36359
Oct 07, 2023 6:18:18 PM net.osmand.impl.ConsoleProgressImplementation print
INFO: Writing map index to binary file... started after 666 ms - -1
Oct 07, 2023 6:55:14 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex
INFO: MAP level SIZE : 469256321
Oct 07, 2023 7:14:33 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex
INFO: MAP level SIZE : 157953833
Oct 07, 2023 7:26:33 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex
INFO: MAP level SIZE : 87773842
Oct 07, 2023 7:28:29 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex
INFO: MAP level SIZE : 8657172
Oct 07, 2023 7:28:29 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapIndex
INFO: MAP INDEX SIZE : 723805292
Oct 07, 2023 7:28:29 PM net.osmand.impl.ConsoleProgressImplementation print
INFO: Writing route index to binary file... started after 4211731 ms - -1
Oct 07, 2023 7:28:48 PM net.osmand.obf.preparation.IndexCreator generateIndexes
SEVERE: Log exception
java.lang.IndexOutOfBoundsException: Index -26077 out of bounds for length 53395
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:372)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
at net.osmand.MainUtilities.main(MainUtilities.java:127)
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index -26077 out of bounds for length 53395
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:372)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
at net.osmand.MainUtilities.main(MainUtilities.java:127)
(IndexRouteCreator.java:1594 that line does appear 3 consecutive times)
The two extracts from Germany are geofabrik extracts, because I was suspecting that the Bayern input I had was corrupted (it had crashed, at the same point). I have just split Bayern along 49 latitude and will start converting the two halves separately and report how it went.
These extracts take more than 10h to convert so it was a bit of an adventure.
I'm running the latest version from github:
$ git log |head
commit e5b6027c2485a39adf5aadcf2761d2f0200906a4
Author: ivanPyrohivskyi <[email protected]>
Date: Wed Oct 4 14:24:28 2023 +0300
Do you also want logs for other and/or successful extracts ? Or the complete log ?
Should I run the conversion, or compile MapCreator, again with some specific flags ?