com.netflix.astyanax.connectionpool.exceptions.TimeoutException

[42701618]

Hi team:
When I run the blueflood,sometime the error will happen.
The error is
com.netflix.astyanax.connectionpool.exceptions.TimeoutException: TimeoutException: [host=192.168.1.1(192.168.1.1):9160, latency=10009(10009), attempts=1]org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:188)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:61)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:478)
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:73)
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:116)
at com.rackspacecloud.blueflood.io.AstyanaxWriter.insertRollups(AstyanaxWriter.java:370)
at com.rackspacecloud.blueflood.service.RollupBatchWriteRunnable.run(RollupBatchWriteRunnable.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:122)
at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:119)
at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:56)
... 12 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 25 more

[42701618]

If the error happen,the performance of rollup will descrease.
The blueflood use the Astyanax 1.56.36,however the newest version of Netflix/astyanax is v3.9.0 .

[shintasmith]

@42701618 , this exception can be caused by many reasons. But generally, it's an indication that your Cassandra cluster is not responding in time (either it is overloaded, or not properly tuned/configured).

What is the size of your Cassandra cluster, your Replication Factor, the version of Cassandra you are using, your rollup configurations (MAX_ROLLUP_READ_THREADS, MAX_ROLLUP_WRITE_THREADS, MAX_LOCATOR_FETCH_THREADS) ?

[42701618]

the version of Cassandra is 2.1.2
There are 3 node, one node is used for ingest,two nodes are used for rollup.
MAX_LOCATOR_FETCH_THREADS=32
MAX_ROLLUP_READ_THREADS=32
MAX_ROLLUP_WRITE_THREADS=32

[42701618]

iostat -d -x -k 1 100

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 2.69 0.15 1.58 2.47 17.07 22.63 0.06 31.97 9.92 1.71
xvde 0.02 1722.30 76.99 196.24 2132.42 7674.28 71.78 12.44 45.53 1.77 48.26

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvde 0.00 1527.00 394.00 140.00 12120.00 5828.00 67.22 138.66 206.88 1.87 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 5.00 0.00 5.00 0.00 40.00 16.00 0.21 42.40 42.40 21.20
xvde 0.00 1121.00 368.00 88.00 11080.00 3724.00 64.93 138.46 168.39 2.19 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 17.00 0.00 8.00 0.00 100.00 25.00 0.24 30.50 14.50 11.60
xvde 0.00 2653.00 229.00 270.00 7068.00 11304.00 73.64 116.55 464.49 2.00 99.60

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvde 0.00 2583.00 190.00 280.00 5928.00 11372.00 73.62 103.14 223.32 2.13 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvde 0.00 1561.00 161.00 221.00 4704.00 8748.00 70.43 137.76 283.29 2.62 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvde 0.00 1165.00 405.00 140.00 12636.00 5516.00 66.61 138.91 257.00 1.83 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvde 0.00 1451.00 364.00 165.00 10928.00 6720.00 66.72 137.65 234.08 1.89 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 9.00 0.00 36.00 8.00 0.32 36.00 8.89 8.00
xvde 0.00 1254.00 407.00 122.00 12024.00 5164.00 64.98 141.09 284.13 1.89 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
xvde 0.00 1495.00 488.00 144.00 14532.00 5780.00 64.28 146.16 206.27 1.58 100.00