otp-1.4.0-shaded.jar does not save (write) big graph to file after building it

[SRN1973]

I realized that the otp-1.4.0-shaded.jar does not write back a calculated Graph.obj when the input files cover a whole country.
(the otp-1.3.0-shaded.jar as well as the otp-2.0-shaded.jar both write the graph as expected)

Input files:

• germany-latest-osm.pbf (https://download.geofabrik.de/europe/germany-latest.osm.pbf)
  -GTFS files:
  https://download.gtfs.de/germany/fv_free/latest.zip
  https://download.gtfs.de/germany/nv_free/latest.zip
  https://download.gtfs.de/germany/rv_free/latest.zip

  -> no custom build-config.json
  -> all files are in the directory otp/graphs/current_14

Command to build and save the Graph:
/usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -Xmx300G -jar otp-1.4.0-shaded.jar --build graphs/current_14
respectively with openjdk-11-jre-headless (I tried both but the result is the same)
java -Xmx300G -jar otp-1.4.0-shaded.jar --build graphs/current_14

Operating System:
Ubuntu 18.04.4 LTS
RAM: 1 TB
120 cernels

Error message of otp:
3:10:31.235 INFO (DirectTransferGenerator.java:100) Done connecting stops to one another. Created a total of 999488 transfers from 348928 stops.
23:10:31.247 INFO (Graph.java:963) Summary (number of each type of annotation):
23:10:31.434 INFO (Graph.java:969) TripDegenerate - 2540
23:10:31.434 INFO (Graph.java:969) NoFutureDates - 3
23:10:31.434 INFO (Graph.java:969) TurnRestrictionBad - 11646
23:10:31.434 INFO (Graph.java:969) RepeatedStops - 1754
23:10:31.434 INFO (Graph.java:969) HopZeroTime - 1152895
23:10:31.434 INFO (Graph.java:969) HopSpeedSlow - 9256
23:10:31.434 INFO (Graph.java:969) StopNotLinkedForTransfers - 54007
23:10:31.434 INFO (Graph.java:969) LevelAmbiguous - 17896
23:10:31.434 INFO (Graph.java:969) HopSpeedFast - 52846
23:10:31.434 INFO (Graph.java:969) StopLinkedTooFar - 3088
23:10:31.435 INFO (Graph.java:969) TurnRestrictionUnknown - 439
23:10:31.435 INFO (Graph.java:969) GraphConnectivity - 81826
23:10:31.435 INFO (Graph.java:969) StopUnlinked - 2950
23:10:31.435 INFO (Graph.java:969) TurnRestrictionException - 1394
23:10:31.435 INFO (Graph.java:969) Graphwide - 1
23:10:40.580 INFO (Graph.java:814) Main graph size: |V|=24870759 |E|=60642402
23:10:40.580 INFO (Graph.java:815) Writing graph /media/vol2/stefan/b/Open_Trip_Planner/graphs/current_14/Graph.obj ...
Exception in thread "main" java.lang.IllegalStateException: com.esotericsoftware.kryo.KryoException: java.lang.NegativeArraySizeException Serialization trace: compactGeometry (org.opentripplanner.routing.edgetype.StreetEdge)
at org.opentripplanner.graph_builder.GraphBuilder.run(GraphBuilder.java:146)
at org.opentripplanner.standalone.OTPMain.run(OTPMain.java:103)
at org.opentripplanner.standalone.OTPMain.main(OTPMain.java:73)
Caused by: com.esotericsoftware.kryo.KryoException: java.lang.NegativeArraySizeException Serialization trace: compactGeometry (org.opentripplanner.routing.edgetype.StreetEdge)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)

at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at org.opentripplanner.routing.graph.Graph.save(Graph.java:841)
at org.opentripplanner.routing.graph.Graph.save(Graph.java:817)
at org.opentripplanner.graph_builder.GraphBuilder.run(GraphBuilder.java:144)
... 2 more
Caused by: java.lang.NegativeArraySizeException
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:542)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjectIntMap.java:306)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectIntMap.java:300)
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:162)
at com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:41)
at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:681)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:616)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:86)

[pkamenarsky]

I'm hitting the same issue, which is most probably related to EsotericSoftware/kryo#382. Unfortunately, this seems to be fixed in kryo > 4.0.0, which is the version used in OTP 1.4.0.

I've tried the workaround described here hail-is/hail#5564 (by calling setReferences(false) in src/main/java/org/opentripplanner/routing/graph/Graph.java), but then the graph serialisation fails due to a stack overflow, which indicates that the object graph contains cycles and reference tracking can not be disabled.

I'm out of ideas currently, save for maybe patching IdentityObjectIntMap to use something bigger than Integer, but I'm not sure whether this won't mess with the used hashing function.

[ra0e]

I have the same issue.

[abyrd]

It does look like this is due to integer overflow in a table expecting less than 2^16 entries. And it does look like a related problem was resolved in EsotericSoftware/kryo#400 and included in release https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0. But as you say, OTP1 is using Kryo 4.0.2 so shouldn't have this problem.

You are correct that the suggested workaround of turning off entity references will not work in OTP. We will have to dig into this further to understand why EsotericSoftware/kryo#400 does not fix it. I'm looking at the source code of the Kryo version used in OTP1.x, and although it does set the isBigTable boolean field introduced in EsotericSoftware/kryo#400, it looks like this resize method is still using a 32 bit signed int to represent the capacity and new size of the resized table, even when sizes are larger than 2^16. I would really need to work through this to see how it's intended to work.

In the mean time, could you try using OTP2 RC1? OTP2 should have a lot less objects in the graph so may not run up against this MAX_INT limit.

A side note based on code I was just looking at: I was thinking again about calling kryo.reset() to reduce the number of referenced objects, but OTP1's method Graph#save(OutputStream) already calls kryo.writeClassAndObject twice per Graph (once for the edges and vertices, and once for the rest of the Graph fields) with autoReset enabled, so the number of referenced objects is already smaller than it would be with a single call.

[abyrd]

@t2gran my sense is that this has not been a problem in OTP2 due to transit data not being stored as edges and vertices. We may eventually run into this same size constraint though. We should research whether Kryo 5 will eliminate these overflow issues.

[SRN1973]

Today I tried it with the Open Street Map "tile" covering whole Germany (from Geofabrik) as well as the "free" GTFS datasets for Germany as a whole (see gtfs.de) and the OTP2 RC as recommended but encountered the same issue.

Operating System:
Ubuntu 18.04.4 LTS
RAM: 1 TB
120 cernels

The graph saving process stops after 4.5 GB are written to file with following error message:

14:30:21.942 INFO (SerializedGraphObject.java:250) Save graph progress: 4.5 GB done
14:30:22.146 ERROR (OTPMain.java:45) An uncaught error occurred inside OTP: java.lang.NegativeArraySizeException: -2147483645
Serialization trace:
serviceId (org.opentripplanner.model.Trip)
trip (org.opentripplanner.routing.trippattern.TripTimes)
tripTimes (org.opentripplanner.model.Timetable)
scheduledTimetable (org.opentripplanner.model.TripPattern)
tripPatternForId (org.opentripplanner.routing.graph.Graph)
graph (org.opentripplanner.routing.graph.SerializedGraphObject)
com.esotericsoftware.kryo.KryoException: java.lang.NegativeArraySizeException: -2147483645
Serialization trace:
serviceId (org.opentripplanner.model.Trip)
trip (org.opentripplanner.routing.trippattern.TripTimes)
tripTimes (org.opentripplanner.model.Timetable)
scheduledTimetable (org.opentripplanner.model.TripPattern)
tripPatternForId (org.opentripplanner.routing.graph.Graph)
graph (org.opentripplanner.routing.graph.SerializedGraphObject)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:86) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:113) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:39) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at org.opentripplanner.routing.graph.SerializedGraphObject.save(SerializedGraphObject.java:235) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at org.opentripplanner.routing.graph.SerializedGraphObject.save(SerializedGraphObject.java:127) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at org.opentripplanner.standalone.OTPMain.startOTPServer(OTPMain.java:128) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at org.opentripplanner.standalone.OTPMain.main(OTPMain.java:38) ~[otp-2.0.0-rc1-shaded.jar:1.1]
Caused by: java.lang.NegativeArraySizeException: -2147483645
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:542) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjectIntMap.java:306) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectIntMap.java:300) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:162) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:41) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:681) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:616) ~[otp-2.0.0-rc1-shaded.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:86) ~[otp-2.0.0-rc1-shaded.jar:1.1]
... 27 common frames omitted

[pkamenarsky]

Unfortunately, OTP2 RC didn't solve the issue for me either:

17:32:48.797 ERROR (OTPMain.java:45) An uncaught error occurred inside OTP: java.lang.NegativeArraySizeException: -2147483645
Serialization trace:
serviceDatesByServiceId (org.opentripplanner.model.calendar.CalendarServiceData)
services (org.opentripplanner.routing.graph.Graph)
graph (org.opentripplanner.routing.graph.SerializedGraphObject)
com.esotericsoftware.kryo.KryoException: java.lang.NegativeArraySizeException: -2147483645

[tim1e1ing]

Problem still existing in OTP 2.1.0-Snapshot

ERROR (OTPMain.java:46) An uncaught error occurred inside OTP: java.lang.NegativeArraySizeException: -2147483645
Serialization trace:
edges (org.opentripplanner.model.SimpleTransfer)
transfersByStop (org.opentripplanner.routing.graph.Graph)
graph (org.opentripplanner.routing.graph.SerializedGraphObject)
com.esotericsoftware.kryo.KryoException: java.lang.NegativeArraySizeException: -2147483645
Serialization trace:
edges (org.opentripplanner.model.SimpleTransfer)
transfersByStop (org.opentripplanner.routing.graph.Graph)
graph (org.opentripplanner.routing.graph.SerializedGraphObject)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[otp2.1.0.jar:1.1]
at de.javakaffee.kryoserializers.guava.MultimapSerializerBase.writeMultimap(MultimapSerializerBase.java:22) ~[otp2.1.0.jar:1.1]
at de.javakaffee.kryoserializers.guava.HashMultimapSerializer.write(HashMultimapSerializer.java:26) ~[otp2.1.0.jar:1.1]
at de.javakaffee.kryoserializers.guava.HashMultimapSerializer.write(HashMultimapSerializer.java:14) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) ~[otp2.1.0.jar:1.1]
at org.opentripplanner.routing.graph.SerializedGraphObject.save(SerializedGraphObject.java:235) ~[otp2.1.0.jar:1.1]
at org.opentripplanner.routing.graph.SerializedGraphObject.save(SerializedGraphObject.java:127) ~[otp2.1.0.jar:1.1]
at org.opentripplanner.standalone.OTPMain.startOTPServer(OTPMain.java:132) ~[otp2.1.0.jar:1.1]
at org.opentripplanner.standalone.OTPMain.main(OTPMain.java:39) ~[otp2.1.0.jar:1.1]
Caused by: java.lang.NegativeArraySizeException: -2147483645
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:542) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjectIntMap.java:306) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectIntMap.java:300) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:162) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:41) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:681) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:570) ~[otp2.1.0.jar:1.1]
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79) ~[otp2.1.0.jar:1.1]
... 16 common frames omitted

[SRN1973]

Maybe this helps (however i have not tested it with OTP yet)...
I encountered a similar error with the R5 routing engine. From the programmers I got following hint which worked for me as a workaround (and as I am not interested in car routing the described downside of the workaround is not a problem for me):

"This is definitely an upstream bug in R5, and it has to do with turn restrictions. If you don't care about turn restrictions (which really only matter for driving, and to a lesser extent biking, but you can always walk your bike), you can use osmconvert with the --drop-relations option to remove them from your PBF. Then R5 should build the network, albeit without information on turn restrictions."

[tim1e1ing]

Error accures while adding the GTFS layer to the streetgraph. The latter was created before and saved without error.

Following your linked hint osmconvert in.pbf --drop-relations -o=out.pbf is not aworkaround.

17:03:33.966 INFO (SerializedGraphObject.java:205) Reading graph from './streetGraph.obj'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/holthaus/otp2/otp2.1.0.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
17:06:26.776 INFO (Graph.java:643) Graph version: MavenVersion(2, 1, 0, SNAPSHOT, 04af5a1)
17:06:26.777 INFO (Graph.java:644) OTP version: MavenVersion(2, 1, 0, SNAPSHOT, 04af5a1)