jmx_exporter_javaagent generate Address Already in Use on Wildfly

[poidras]

Hello everyone,

I'm working on Wildfly, and i often have this kind of issue :

Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:461) at sun.nio.ch.Net.bind(Net.java:453) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) at sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:141) at sun.net.httpserver.HttpServerImpl.<init>(HttpServerImpl.java:50) at sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35) at com.sun.net.httpserver.HttpServer.create(HttpServer.java:130) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:396) at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:31)

Sometimes this happens when I properly shut down my Wildfly server and then restart it quickly (in less than 15 seconds). Although the Wildfly process is terminated, I still observe TIME_WAIT or sometimes LISTEN connections from the HTTPServer created by the JavaAgent code in the JMX exporter. These connections can persist from 5 seconds up to 2 minutes at times. As a workaround, I have to use kill -9 to forcibly terminate the process and then wait until running ss -planetu | grep PROM_PORT shows no lingering connections before I can start the server again.

Is it possible to gracefully handle the HTTPServer shutdown? I noticed there’s a try-catch block in place—perhaps adding a finally block to ensure the metrics exposure is properly closed would help.

To Notice, all of the others ports (JBOSS_HTTP_MANAGEMENT_PORT, JBOSS_HTTP_PORT and so on are at this step completly already closed since a while).

If you already notice this kind of issues, or you've already have a workaround, please let me know !

Many Thanks for your feedbacks.

[dhoard]

It sounds similar to #94.

Adding a Java shutdown hook to stop the embedded HttpServer may resolve the issue.

Which version are you using?

[dhoard]

@poidras I can make changes to add shutdown hooks. Can you test with a development jar?

[poidras]

Hello @dhoard, yes i can test it.

[poidras]

@dhoard I've tested 0.17.0, 0.20.0 and 1.2.0 version. It occur on each.

[dhoard]

@poidras please test...

jmx_prometheus_javaagent-1.2.0-post.zip

[poidras]

I'll let you know asap when i can test it. Many Thanks @dhoard !

[dhoard]

Should be resolved by Add cleanup shutdown hooks https://github.com/prometheus/jmx_exporter/commit/af4a275964bb463efa4219cd51ac111528f8aacb.

Local testing shows the port is being correctly released.

[poidras]

Hello,
i've tested with your 1.2.0-post version, but with a kill -9 and a start of the service quickly, looks like the port still already in use ..
I'm running on wildfly 8.2.1 Final on openjdk 8 SLES 15.

[poidras]

It looks like when we kill -9 and not kill -15, the process Java don't let the time to proceed the shutdown hook step.

[dhoard]

kill -9 (SIGKILL)

• Lingering sockets (TIME_WAIT) can't be resolved from an application code perspective.
• On Linux, you can tune the kernel using net.ipv4.tcp_fin_timeout to decrease the timeout.

kill -15 (SIGTERM)

• The recently merged shutdown hook code correctly closes sockets.