Skip to content

Add configurable agent logging backend - #1528

Open
arnabnandy7 wants to merge 1 commit into
prometheus:mainfrom
arnabnandy7:bugfix/agent-logging-issues
Open

Add configurable agent logging backend#1528
arnabnandy7 wants to merge 1 commit into
prometheus:mainfrom
arnabnandy7:bugfix/agent-logging-issues

Conversation

@arnabnandy7

Copy link
Copy Markdown

Summary

Adds configurable logging backends for the JMX Exporter Java agents to prevent exporter logging from prematurely initializing or modifying the application’s Java Util Logging (JUL) configuration.

The Java agent and isolator Java agent now use a JUL-independent native backend by default. Users who rely on the existing JUL integration can explicitly select it through a system property or environment variable.

Fixes #1127.

Background

The Java agent previously initialized JUL while its classes were loaded during JVM startup. This could happen before the target application or application server had configured its logging system.

Because JUL’s LogManager is global to the JVM, early initialization could interfere with applications that use custom logging managers, including WildFly, JBoss EAP, and GlassFish. Reported symptoms included:

  • Failure to load a custom LogManager because its classes were unavailable to the agent classloader.
  • Application log formatting being initialized before the application configured it.
  • Exporter code replacing formatters on shared JUL handlers.
  • Application-server-specific classpath and startup workarounds being required.

Related reports include #455, #976, and #1000.

Changes

Configurable logging backends

Introduces a logging backend abstraction behind the existing Logger and LoggerFactory API.

Two backends are available:

  • native: writes exporter log messages directly to standard error without obtaining a JUL logger.
  • jul: retains the existing Java Util Logging integration and formatter behavior.

Existing exporter classes continue to use the same logging facade:

private static final Logger LOGGER = LoggerFactory.getLogger(Example.class);

This keeps backend selection out of collector and exporter business logic.

Safe Java agent defaults

The following artifacts select native as their default before creating their first logger:

  • jmx_prometheus_javaagent
  • jmx_prometheus_isolator_javaagent

This prevents exporter-owned logger initialization from accessing JUL during normal Java agent startup.

The collector retains jul as its general default, preserving existing behavior when it is used outside these agent entry points, including standalone operation.

Configuration

The logging backend can be selected with the system property:

-Djmx.prometheus.exporter.logging.backend=native

or:

-Djmx.prometheus.exporter.logging.backend=jul

It can also be selected with the environment variable:

JMX_PROMETHEUS_EXPORTER_LOGGING_BACKEND=native

Supported values are:

  • native
  • jul

The system property takes precedence over the environment variable. Explicit configuration overrides the artifact-specific default.

Invalid values fail with a descriptive error instead of silently falling back to another backend.

Lazy JUL initialization

JUL-specific behavior is contained in JulLoggerBackend. Selecting the native backend does not call:

java.util.logging.Logger.getLogger(...)

The existing JUL formatter setup is also performed only after JUL has been selected. This retains the existing JUL behavior without initializing the global logging system in native mode.

HTTP server logging

HTTPServerFactory previously created a JUL logger directly, bypassing the project’s logging facade.

It now uses LoggerFactory, ensuring that native mode remains effective when HTTP server configuration warnings are logged.

The affected warning messages were converted from JUL MessageFormat placeholders to the format-string convention already used by the project’s logger facade.

Isolator agent

The isolator agent’s embedded logging implementation now supports the same backend configuration and defaults to native logging.

This is necessary because isolating the nested exporter with a separate classloader does not isolate the JVM’s JUL LogManager.

Documentation

The Java agent and isolator agent deployment documentation now covers:

  • Default backend behavior
  • Native output destination and enabled levels
  • System property and environment variable configuration
  • Configuration precedence
  • When backend selection occurs
  • Risks associated with explicitly selecting process-wide JUL during agent startup

Versioned documentation was not changed because this functionality has not yet been released.

Compatibility

The existing JUL implementation has not been removed.

Users who require integration with application JUL handlers, levels, files, or formatting can select:

-Djmx.prometheus.exporter.logging.backend=jul

Selecting JUL preserves the existing root and named-logger formatter configuration. It also retains the previous application-server risks because JUL and its LogManager remain process-wide.

Native logging currently enables INFO, WARN, and ERROR messages and writes them to standard error. TRACE remains disabled unless developer-debug behavior is enabled separately.

Limitations

This change prevents JMX Exporter’s own logging facade and HTTPServerFactory from directly initializing JUL in native mode.

It cannot guarantee that every JDK component or third-party dependency loaded by the agent will never access JUL. JUL remains global JVM state and cannot be isolated through shading or a custom classloader.

The change nevertheless removes the known early initialization and formatter mutation performed directly by JMX Exporter.

Testing

The following modules were built and tested:

  • collector
  • jmx_prometheus_common
  • jmx_prometheus_javaagent
  • jmx_prometheus_isolator_javaagent

Validation included:

  • Existing JUL logging tests
  • JUL level mapping tests
  • Native backend output tests
  • Explicit JUL backend selection
  • Invalid backend validation
  • HTTP server tests
  • Java agent argument tests
  • Isolator agent tests
  • Spotless formatting checks
  • git diff --check
  • Docusaurus production documentation build

All selected module tests and documentation builds passed.

@dhoard

dhoard commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

@arnabnandy7 Thanks for the PR.

In general, the idea seems sound, but the code changes the default behavior which would cause unexpected behavior.

Thoughts on changing the default behavior if the system property is not used?

@arnabnandy7

arnabnandy7 commented Aug 13, 2026

Copy link
Copy Markdown
Author

Thoughts on changing the default behavior if the system property is not used?

@dhoard thanks for the review. I agree that changing the default could surprise existing users, especially if they rely on JUL for log routing, formatting, or level configuration.

I kept JUL as the default when neither the system property nor the environment variable is set, and make native logging opt-in using -Djmx.prometheus.exporter.logging.backend=native.

Updated both agents, the tests, and the documentation accordingly. This should provide a workaround for #1127 without changing the existing behavior.

Signed-off-by: Arnab Nandy <arnab_nandy7@yahoo.com>
@arnabnandy7
arnabnandy7 force-pushed the bugfix/agent-logging-issues branch from 0af38ba to c46f416 Compare August 13, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Java agent causes Java logging (JUL) issues on various platforms

2 participants