feat: add graceful startup error handling to Java agent - #1547
Conversation
4385a32 to
d8e5e7b
Compare
Add a `graceful:` agent argument prefix that keeps the JVM running when the JMX exporter fails to start. The error is still logged and any started resources are closed, but System.exit(1) is skipped. Default behavior remains fail-fast for backward compatibility. This only affects startup errors; scrape-time errors continue to be reported via the jmx_scrape_error metric. Signed-off-by: Jan Sykora <jan@cast.ai>
d8e5e7b to
b7ef577
Compare
|
@jansyk13 Thanks for the PR. Can you elaborate on the use case as to why you feel this is required? |
|
Hi @dhoard, thanks for the question. The main use case is avoiding a hard crash of the application when the exporter fails to start. In environments with auto-instrumentation, the application owner is often not the same team that manages the exporter configuration or deployment. If a config or environment issue prevents the agent from starting, System.exit(1) brings down the entire application. For many teams it is preferable to keep the application running and simply lose metrics until the issue is fixed, rather than fail the whole JVM. The graceful: prefix makes that opt-in behavior possible without changing the default fail-fast semantics. I should have included this use case in the PR description up front - my mistake. |
|
@jansyk13 after thinking about the PR and intention, this feels like a "feature" that is trying to solve a DevOps process issue. |
|
@dhoard , I see your point. Ideally this is fixed by better DevOps processes, but in practice many companies split app and platform ownership. The app team often can’t quickly fix the exporter config or deployment. So while I agree it’s partly a process problem, I think JMX exporter should still offer a way out. The graceful: prefix keeps fail-fast as the default and only adds an opt-in escape hatch for teams that need it. |
|
@jansyk13 here is my concerning scenario...
To enable metrics, the JMX Exporter configuration has to be fixed and the application restarted. Additionally, if the process is always to deploy and verify metrics, then the flag doesn't provide value. I am inclined to reject the PR since it can hide a real error scenario and ultimately is solving a DevOps process problem. I'll discuss with the team on our next call. |
2 participants
Add a
graceful:agent argument prefix that keeps the JVM running when the JMX exporter fails to start. The error is still logged and any started resources are closed, but System.exit(1) is skipped. Default behavior remains fail-fast for backward compatibility.This only affects startup errors; scrape-time errors continue to be reported via the jmx_scrape_error metric.