Add configurable agent logging backend - #1528
Conversation
|
@arnabnandy7 Thanks for the PR. In general, the idea seems sound, but the code changes the default behavior which would cause unexpected behavior. Thoughts on changing the default behavior if the system property is not used? |
@dhoard thanks for the review. I agree that changing the default could surprise existing users, especially if they rely on JUL for log routing, formatting, or level configuration. I kept JUL as the default when neither the system property nor the environment variable is set, and make native logging opt-in using Updated both agents, the tests, and the documentation accordingly. This should provide a workaround for #1127 without changing the existing behavior. |
Signed-off-by: Arnab Nandy <arnab_nandy7@yahoo.com>
0af38ba to
c46f416
Compare
Summary
Adds configurable logging backends for the JMX Exporter Java agents to prevent exporter logging from prematurely initializing or modifying the application’s Java Util Logging (JUL) configuration.
The Java agent and isolator Java agent now use a JUL-independent native backend by default. Users who rely on the existing JUL integration can explicitly select it through a system property or environment variable.
Fixes #1127.
Background
The Java agent previously initialized JUL while its classes were loaded during JVM startup. This could happen before the target application or application server had configured its logging system.
Because JUL’s
LogManageris global to the JVM, early initialization could interfere with applications that use custom logging managers, including WildFly, JBoss EAP, and GlassFish. Reported symptoms included:LogManagerbecause its classes were unavailable to the agent classloader.Related reports include #455, #976, and #1000.
Changes
Configurable logging backends
Introduces a logging backend abstraction behind the existing
LoggerandLoggerFactoryAPI.Two backends are available:
native: writes exporter log messages directly to standard error without obtaining a JUL logger.jul: retains the existing Java Util Logging integration and formatter behavior.Existing exporter classes continue to use the same logging facade:
This keeps backend selection out of collector and exporter business logic.
Safe Java agent defaults
The following artifacts select
nativeas their default before creating their first logger:jmx_prometheus_javaagentjmx_prometheus_isolator_javaagentThis prevents exporter-owned logger initialization from accessing JUL during normal Java agent startup.
The collector retains
julas its general default, preserving existing behavior when it is used outside these agent entry points, including standalone operation.Configuration
The logging backend can be selected with the system property:
or:
It can also be selected with the environment variable:
Supported values are:
nativejulThe system property takes precedence over the environment variable. Explicit configuration overrides the artifact-specific default.
Invalid values fail with a descriptive error instead of silently falling back to another backend.
Lazy JUL initialization
JUL-specific behavior is contained in
JulLoggerBackend. Selecting the native backend does not call:The existing JUL formatter setup is also performed only after JUL has been selected. This retains the existing JUL behavior without initializing the global logging system in native mode.
HTTP server logging
HTTPServerFactorypreviously created a JUL logger directly, bypassing the project’s logging facade.It now uses
LoggerFactory, ensuring that native mode remains effective when HTTP server configuration warnings are logged.The affected warning messages were converted from JUL
MessageFormatplaceholders to the format-string convention already used by the project’s logger facade.Isolator agent
The isolator agent’s embedded logging implementation now supports the same backend configuration and defaults to native logging.
This is necessary because isolating the nested exporter with a separate classloader does not isolate the JVM’s JUL
LogManager.Documentation
The Java agent and isolator agent deployment documentation now covers:
Versioned documentation was not changed because this functionality has not yet been released.
Compatibility
The existing JUL implementation has not been removed.
Users who require integration with application JUL handlers, levels, files, or formatting can select:
Selecting JUL preserves the existing root and named-logger formatter configuration. It also retains the previous application-server risks because JUL and its
LogManagerremain process-wide.Native logging currently enables
INFO,WARN, andERRORmessages and writes them to standard error.TRACEremains disabled unless developer-debug behavior is enabled separately.Limitations
This change prevents JMX Exporter’s own logging facade and
HTTPServerFactoryfrom directly initializing JUL in native mode.It cannot guarantee that every JDK component or third-party dependency loaded by the agent will never access JUL. JUL remains global JVM state and cannot be isolated through shading or a custom classloader.
The change nevertheless removes the known early initialization and formatter mutation performed directly by JMX Exporter.
Testing
The following modules were built and tested:
collectorjmx_prometheus_commonjmx_prometheus_javaagentjmx_prometheus_isolator_javaagentValidation included:
git diff --checkAll selected module tests and documentation builds passed.