Java memory calculator does not count Datadog JVM agent classes

[PerfectSlayer]

Hello,

I’m asking for help about a Java application crash when using paketo-buildpack/datadog due to the memory calcultor.
When using BP_DATADOG_ENABLED to TRUE, Datadog JVM agent is well injected as resource inside the OCI image and as parameter in the JAVA_TOOL_OPTIONS at run time.
But I can see when using BP_DEBUG to TRUE that the memory calculator does not take into account the Datadog JVM agent jar file to count its classes:

Memory Calculation: (100% * (21689 + 11705 + 0 + 0)) * 0.35

You can see that agentClassCount is set to 0 from https://github.com/paketo-buildpacks/libjvm/blob/72a7df779c8ddd1ea128a50b45214e0d1be03307/helper/memory_calculator.go#L131

I wonder it could be related to the datadog helper toggle that append the -javaagent: to the JAVA_TOOL_OPTIONS at runtime? Could it reponsible for the memory calculator to not find out that the Datadog agent is/will be added when the application starts? (sorry I’m not to verse into the buildpack thing, its lifecycle, phases, etc...)

But not taking the Datadog JVM agent into account leads to JVM crash when exceeding the MaxMetaspaceSize.
The application I use is a basic (empty) Spring boot application, using Maven as build tool. But that should not have an impact.

Thanks for your help 🙏

[creator]     ===> ANALYZING
[INFO]     [creator]     Restoring data for SBOM from previous image
[INFO]     [creator]     ===> DETECTING
[INFO]     [creator]     target distro name/version labels not found, reading /etc/os-release file
[INFO]     [creator]     7 of 26 buildpacks participating
[INFO]     [creator]     paketo-buildpacks/ca-certificates   3.10.1
[INFO]     [creator]     paketo-buildpacks/bellsoft-liberica 11.2.2
[INFO]     [creator]     paketo-buildpacks/syft              2.15.0
[INFO]     [creator]     paketo-buildpacks/executable-jar    6.13.0
[INFO]     [creator]     paketo-buildpacks/dist-zip          5.10.0
[INFO]     [creator]     paketo-buildpacks/spring-boot       5.33.0
[INFO]     [creator]     paketo-buildpacks/datadog           5.33.0
[INFO]     [creator]     ===> RESTORING
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:helper" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:java-security-properties" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jre" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/syft:syft" from cache
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:helper" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:web-application-type" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/datadog:datadog-agent-java" from app image
[INFO]     [creator]     Restoring metadata for "paketo-buildpacks/datadog:helper" from app image
[INFO]     [creator]     Restoring data for "paketo-buildpacks/syft:syft" from cache
[INFO]     [creator]     Restoring data for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from cache
[INFO]     [creator]     Restoring data for SBOM from cache
[INFO]     [creator]     ===> BUILDING
[INFO]     [creator]     target distro name/version labels not found, reading /etc/os-release file

[dmikusa]

I wonder it could be related to the datadog helper toggle that append the -javaagent: to the JAVA_TOOL_OPTIONS at runtime? Could it reponsible for the memory calculator to not find out that the Datadog agent is/will be added when the application starts? (sorry I’m not to verse into the buildpack thing, its lifecycle, phases, etc...)

Yes, that's likely it. The runtime helpers execute in the same order as the buildpacks, so the Datadog helper won't run till after the memory calculator, which means the memory calculator won't see that the agent is present.

The Datadog buildpack is different in this regard compared to some of the other APM buildpacks. It is toggling that at runtime, but others will include it in the JVM options at build time, which is why this memory calculator inspection usually works.

I would try setting BPL_JVM_CLASS_ADJUSTMENT at runtime with so you can adjust for this and get a correctly configured metaspace.

Absolute or percentage based adjustment of the memory calculator's class count, which influences various memory settings of the JVM. This is useful when the number of classes cannot be reliably determined during build-time and workloads run into OOM situations. Defaults to 100%.

when adjusted, that should show up in the output here:

Memory Calculation: (100% * (21689 + 11705 + 0 + )) * 0.35

[PerfectSlayer]

That should work, no doubt here.

But I am looking for a way to make the Datadog buildpack to work out of the box as before, and it feels like introducing the toogle helper - ie the capability of enabling / disabling at runtime - is breaking some Datadog users.
For example, there is no unique BPL_JVM_CLASS_ADJUSTMENT value we can recommend to make it work as the value will depend of the user app (how may classes its app has) and the version of Datadog tracer too.

Can you think of a design where the Datadog buildpack would work with the memory calculator, or would you prefer keeping it as is and don’t intend to fix this case? To be clear, we can help for implementation / testing but I would first know if it’s something you would like to support and have a solution about it or not.

[dmikusa]

So my first goal when folks post questions is to try and get them to a situation that's at least working for them. When they're stable, then we can examine possible changes to the buildpacks.

I agree that this isn't ideal. It would be best if the Datadog buildpack could just handle this, so I think it's worthwhile talking through what a good solution would be & at a minimum documenting it as an issue on the buildpack. If you're willing to contribute a solution, that'd be fantastic too.

The first solution that comes to mind would be to make a build-time switch to enable/disable the agent. If you toggle the agent on (i.e. add the -javaagent flag to JAVA_TOOL_OPTIONS at build time, then the existing solution would work. It would see that the agent is enabled, and it would count the classes correctly. That compromises some of the flexibility of the present solution, though, being able to toggle things on/off at runtime. We could modify the buildpack to support both build time and runtime toggle, though, which would give users the choice.

Aside from that, the challenge is that how to let the memory calculator know that there is a JAR that exists outside of the classpath but will still be loaded by the JVM so that the memory calculator can count the classes in that JAR. That would likely involve creating some way for the buildpack at build time to signal to the memory calculator which runs at runtime not using JAVA_TOOL_OPTIONS. We have a pattern for this, which is to use an env variable that starts with BPI_. I'm not opposed to this, but I think it would be a little more work, because we'd need to document a mini-protocol around how the two communicate.

I'm open to other solutions as well, if there's an approach you were considering. Just let me know.

[PerfectSlayer]

The first solution that comes to mind would be to make a build-time switch to enable/disable the agent.

Here is the PR that introduce the runtime toggle: paketo-buildpacks/datadog#160. It can give a bit of context about why it was introduce and how (moving from build time injection to runtime).

We could modify the buildpack to support both build time and runtime toggle, though, which would give users the choice.

How would you do that? You would always inject at build time but have the runtime helper to remove it from the JAVA_TOOL_OPTIONS if disabled?

The challenge is that how to let the memory calculator know that there is a JAR that exists outside of the classpath

It looks like the Datadog build pack is exposing BPI_DATADOG_AGENT_PATH at build time for the runtime helper to find it and inject it in the JAVA_TOOL_OPTIONS environment variable. But I guess we would need to define a more generic mechanism instead of making the memory calculator depend of the Datadog buildpack specific variable...

I would be in favor of fixing the Datadog buildpack rather than trying to solve the memory calculator in a more generic approach, especially since you mention it's a problem specific to the Datadog buildpack as the other APM use build time injection.

[dmikusa]

How would you do that? You would always inject at build time but have the runtime helper to remove it from the JAVA_TOOL_OPTIONS if disabled?

My thought was that if you enabled it at buildtime then it would be permanently enabled for that image. If you don't enable it at buildtime, then you can toggle it on/off at runtime, but you'd have to accept the downside that the memory calculator wouldn't see it and you may need to manually adjust your metaspace limits.

The approach you mentioned is interesting. I think it would work, but it would have the downside of over provisioning memory when you disable Datadog at runtime. For example, if we have the buildpack enable the agent at buildtime. The runtime toggle could then remove the agent when you disable it at runtime, however, the memory calculator would have already run at that point so it would have included the agent in it's calculations.

I like your approach a little better in that it is more cautious. Slightly over provisioning metaspace doesn't result in a crash, whereas under provisioning it can.

The slight adjustment that I would suggest is we make a buildtime default option. So we already have BP_DATADOG_ENABLED which when true causes the buildpack to run and contribute the agent code to the image. We can add BP_DATADOG_AGENT_ENABLED, which controls the default behavior of the agent at runtime. It would default to false, which is the current behavior. If this is set to true by the user, then at buildtime, the buildpack would add the agent to JAVA_TOOL_OPTIONS. If false, it does not add the agent and lets the runtime toggle add it. We'd also need to update the runtime toggle to take BP_DATADOG_AGENT_ENABLED into account.

We can then add to the docs, the caveat that the BP_DATADOG_AGENT_ENABLED flag when false causes the memory calculator to underestimate the required metaspace, which may cause app crashes.

How does that sound?

I would be in favor of fixing the Datadog buildpack rather than trying to solve the memory calculator in a more generic approach, especially since you mention it's a problem specific to the Datadog buildpack as the other APM use build time injection.

Yes, I'm fine with that. I think even if we change things in the memory calculator, we'd still suffer from the buildpack ordering problem. The memory calculator runs before the Datadog agent toggle, so it would not know if you turned that agent on/off. It would only know about what was set at buildtime.