Skip to content

Failed onupdate ack - app should exit #32

New issue
New issue
Open
Open
Failed onupdate ack - app should exit#32
@solsson

Description

@solsson
solsson
opened on Oct 22, 2019

I'm logging this before we've analyzed the error, because it's the first time this error happened to us in actual use. The pod never restarted, and we're not sure what the desired behavior would be.

2019-10-21 13:28:30,246 DEBUG [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Polled 0 records
2019-10-21 13:28:36,246 DEBUG [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Polled 0 records
2019-10-21 13:28:40,179 DEBUG [org.apa.kaf.cli.Metadata] (kafkaclient) Updated cluster metadata version 4 to MetadataCache{cluster=Cluster(id = EPuifXv-TL6qnIOTWph3bQ, nodes = [kafka-2.broker.kafka.svc.cluster.local:9092 ...
2019-10-21 13:28:42,247 DEBUG [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Polled 0 records
2019-10-21 13:28:48,247 DEBUG [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Polled 0 records
2019-10-21 13:28:52,899 DEBUG [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Polled 1 records
2019-10-21 13:28:52,900 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 250 ms at count 1 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:53,151 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:53,151 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:28:53,151 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 500 ms at count 2 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:53,651 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:53,651 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:28:53,652 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 1000 ms at count 3 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:54,652 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:54,652 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:28:54,653 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 2000 ms at count 4 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:56,653 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:28:56,653 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:28:56,653 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 4000 ms at count 5 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:00,654 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:00,654 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:29:00,654 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 8000 ms at count 6 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:08,654 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:08,655 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:29:08,655 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 16000 ms at count 7 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:24,655 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:24,655 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:29:24,656 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=true with wait 32000 ms at count 8 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:56,656 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://127.0.0.1:80: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:56,656 INFO  [org.apa.htt.imp.exe.RetryExec] (kafkaclient) Retrying request to {}->http://127.0.0.1:80
2019-10-21 13:29:56,657 INFO  [se.yol.kaf.key.onu.hc.RetryConnectionRefusedThreadSleep] (kafkaclient) Retry=false at count 9 for what we assume is connection refused: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
2019-10-21 13:29:56,657 ERROR [se.yol.kaf.key.onu.OnUpdateForwarder] (kafkaclient) Ack failed for UpdatesDispatcherHttp[http://127.0.0.1/kafka-keyvalue/v1/updates] topic ...: se.yolean.kafka.keyvalue.onupdate.TargetAckFailedException: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
	at se.yolean.kafka.keyvalue.onupdate.hc.UpdatesDispatcherHttp.dispatch(UpdatesDispatcherHttp.java:71)
	at se.yolean.kafka.keyvalue.onupdate.OnUpdateForwarder.pollEndBlockingUntilTargetsAck(OnUpdateForwarder.java:74)
	at se.yolean.kafka.keyvalue.ConsumerAtLeastOnce.run(ConsumerAtLeastOnce.java:292)
	at se.yolean.kafka.keyvalue.ConsumerAtLeastOnce.run(ConsumerAtLeastOnce.java:183)
	at java.lang.Thread.run(Thread.java:748)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
	at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:313)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:191)
	at se.yolean.kafka.keyvalue.onupdate.hc.UpdatesDispatcherHttp.dispatch(UpdatesDispatcherHttp.java:67)
	... 6 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:5912)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	... 18 more

2019-10-21 13:29:56,657 WARN  [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Failed onupdate ack - app should exit: se.yolean.kafka.keyvalue.onupdate.UpdateSemanticsSuggestHalt: Will stop fowarding updates upon any error, to not violate consistency
2019-10-21 13:29:56,657 ERROR [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Unrecognized error at stage Polling: se.yolean.kafka.keyvalue.onupdate.UpdateSemanticsSuggestHalt: Will stop fowarding updates upon any error, to not violate consistency
	at se.yolean.kafka.keyvalue.onupdate.OnUpdateForwarder.pollEndBlockingUntilTargetsAck(OnUpdateForwarder.java:77)
	at se.yolean.kafka.keyvalue.ConsumerAtLeastOnce.run(ConsumerAtLeastOnce.java:292)
	at se.yolean.kafka.keyvalue.ConsumerAtLeastOnce.run(ConsumerAtLeastOnce.java:183)
	at java.lang.Thread.run(Thread.java:748)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: se.yolean.kafka.keyvalue.onupdate.TargetAckFailedException: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
	at se.yolean.kafka.keyvalue.onupdate.hc.UpdatesDispatcherHttp.dispatch(UpdatesDispatcherHttp.java:71)
	at se.yolean.kafka.keyvalue.onupdate.OnUpdateForwarder.pollEndBlockingUntilTargetsAck(OnUpdateForwarder.java:74)
	... 5 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
	at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:313)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:191)
	at se.yolean.kafka.keyvalue.onupdate.hc.UpdatesDispatcherHttp.dispatch(UpdatesDispatcherHttp.java:67)
	... 6 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:5912)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	... 18 more

2019-10-21 13:29:56,657 INFO  [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Closing consumer ...
2019-10-21 13:29:56,657 INFO  [se.yol.kaf.key.ConsumerAtLeastOnce] (kafkaclient) Consumer closed at stage Polling; Use liveness probes with /health for app termination
Exception in thread "kafkaclient" se.yolean.kafka.keyvalue.onupdate.UpdateSemanticsSuggestHalt: Will stop fowarding updates upon any error, to not violate consistency
	at se.yolean.kafka.keyvalue.onupdate.OnUpdateForwarder.pollEndBlockingUntilTargetsAck(OnUpdateForwarder.java:77)
	at se.yolean.kafka.keyvalue.ConsumerAtLeastOnce.run(ConsumerAtLeastOnce.java:292)
	at se.yolean.kafka.keyvalue.ConsumerAtLeastOnce.run(ConsumerAtLeastOnce.java:183)
	at java.lang.Thread.run(Thread.java:748)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: se.yolean.kafka.keyvalue.onupdate.TargetAckFailedException: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
	at se.yolean.kafka.keyvalue.onupdate.hc.UpdatesDispatcherHttp.dispatch(UpdatesDispatcherHttp.java:71)
	at se.yolean.kafka.keyvalue.onupdate.OnUpdateForwarder.pollEndBlockingUntilTargetsAck(OnUpdateForwarder.java:74)
	... 5 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
	at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:313)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:191)
	at se.yolean.kafka.keyvalue.onupdate.hc.UpdatesDispatcherHttp.dispatch(UpdatesDispatcherHttp.java:67)
	... 6 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:5912)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	... 18 more

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions