Serve v0.31.0 SNAPSHOT Crashes with Heap Space Errors in Docker

[Null1515]

Hey
The Problem I'm Facing:

I’m using Serve v0.31.0 SNAPSHOT in Docker with:

• 4GB shared memory
• GPU enabled

I’m trying to use Serve as a dynamic batcher for requests from my Java client to avoid performance issues. However, I’m encountering Serve crashes with heap space errors, causing Docker container to stop.

Logs:
2025-03-31 152801 INFO ModelServer.txt

I’ve Tried to Adjusted memory allocation in Docker and Experimented with WLM.
Error on WLM:
java.util.concurrent.ExecutionException: ai.djl.serving.wlm.util.WlmException: Receiver class ai.djl.pytorch.engine.PtNDArray does not define or inherit an implementation of the resolved method 'abstract java.nio.ByteBuffer toByteBuffer(boolean)' of interface ai.djl.ndarray.NDArray.

so, regarding these issues I have 2 questions to ask:

1. Are there known memory optimizations for Serve in Docker?
2. Is the WLM error related to a PyTorch engine compatibility issue?

I’d greatly appreciate any guidance! Sorry for any confusions and thank you for your time.

[frankfliu]

Are you using djl-serving?

The java heap issue is not related to model or engine, you need to adjust -Xmx flags. By default , DJLServing only use 1G of heap size, you can set env var -e JAVA_OPTS ="-Xms2g -Xmx1g -XX:+ExitOnOutOfMemoryError" to override the default options. See: https://docs.djl.ai/master/docs/serving/serving/docs/configurations_global.html#environment-variables

Do you have the full stacktrace for the PtNDArray exception?
The error looks like version mismatched for DJL api jar and pytorch-engine jar. Are you using our docker container?

[Null1515]

I'm using DJL-serving 0.31.0-SNAPSHOT, while WLM is version 0.28.0.

I've successfully used my model with both DJL and DJL-serving without issues, but when I try to use it with WLM, I encounter problems.

For WLM, I'm running it within my application (not Docker), and it automatically downloads the PyTorch-engine as part of DJL.

Here are the WLM logs showing the issue:

log.txt

[frankfliu]

@Null1515

If you use wlm 0.28.0, you must use DJL 0.28.0. We stopped releasing new wlm version due to low adoption. wlm now is an internal component for djl-serving. Can you tell us more about your use case?

[github-actions]

This issue is stale because it has been open for 30 days with no activity.