Cassandra startup fail in kubernetes

[kimxogus]

I tried to use zipkin-cassandra:2.10.4 using Financial-Times/zipkin-helm in my kubernetes cluster and cassandra never startup successfully because of SocketException.
It seems dockerContainerIp method doesn't work properly. How can I fix this? if not, can you add some explicit env var configuration for host ip? In kubernetes, pod ip can be set in environment variable

        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

• environment
  • aws
  • kubernetes 1.9.9
  • kops 1.9.2
  • calico networking

CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V
CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: couldn't get host ip
org.apache.cassandra.exceptions.ConfigurationException: couldn't get host ip
	at ZipkinConfigurationLoader.loadConfig(ZipkinConfigurationLoader.java:47)
	at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:262)
	at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:141)
	at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
Caused by: java.net.SocketException
	at java.util.Optional.orElseThrow(Optional.java:290)
	at ZipkinConfigurationLoader.dockerContainerIp(ZipkinConfigurationLoader.java:57)
	at ZipkinConfigurationLoader.loadConfig(ZipkinConfigurationLoader.java:37)
	... 5 more
ERROR [main] 2018-08-01 05:55:09,703 CassandraDaemon.java:708 - Exception encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: couldn't get host ip
	at ZipkinConfigurationLoader.loadConfig(ZipkinConfigurationLoader.java:47) ~[main/:na]
	at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:262) ~[apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:141) ~[apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647) [apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582) [apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]
Caused by: java.net.SocketException: null
	at java.util.Optional.orElseThrow(Optional.java:290) ~[na:1.8.0_171]
	at ZipkinConfigurationLoader.dockerContainerIp(ZipkinConfigurationLoader.java:57) ~[main/:na]
	at ZipkinConfigurationLoader.loadConfig(ZipkinConfigurationLoader.java:37) ~[main/:na]
	... 5 common frames omitted

[codefromthecrypt]

This is a demo image, certainly not for production, but should certainly allow demos to work :)

When considering options we should keep that in mind.

Maybe we need to look at whatever commandline overrides are possible in cassandra, so that the link ip still works by default, but can be overridden without adding maintenance.

cc @openzipkin/cassandra