Experiencing Slow processing when large number of items are added in Subscription request

[jchirantan]

Hi @kevinherron,
I am using Milo SDK version 1.0.8 and I am trying to carry out a load test on Milo SDK.

I am trying to add 30 different subscription requests each containing 50 items using 30 parallel threads. I am using a queue to store the PublishResponse till earlier responses are processed.
After some time, my queue size goes on increasing and it takes up the whole heap space which results in Java Out of memory exception. I suspect this behaviour is because the Milo SDK is getting slower in processing requests.

I was using OPC UA Stack Legacy earlier and there I had tested for this scenario. There this scenario works properly.

I also tried for less items(Like 1 or 2 subscription requests with 50 items each) on milo SDK. There it is working properly, but as the number of client items increase, it gets slowed resulting in Java out of memory exception and in turn stopping the whole process.

Have you tested the Milo SDK for this kind of load?
Also, can you share with me a full fledged code example for this subscription from Client side?

Thanks,
Chirantan

[jchirantan]

Here is the screenshot from Visual VM capturing CPU and Memory Usage:

[kevinherron]

You'll have to share the details (code) of your test so that I can better understand it.

I'm not sure what you're talking about with storing PublishRepsonses, those aren't delivered to any of the normal user callbacks.

[jchirantan]

When we use OpcUaClient.publishAsync(), we receive PublishResponse in return.
In this PublishResponse, we get the NotificationMessage containing acknowledged Sequence numbers and respective data. So, I was talking about these responses.

[kevinherron]

Maybe you can get a heap dump once the memory starts increasing. You are only talking about 1500 items, which is nothing, and it's not clear to me from your test description where objects/memory might be accumulating.

[jchirantan]

Ok, Thanks for the help. Will take the dump and see if I can find something

[jchirantan]

For now I can say there are 30 threads running in parallel, subscribing for 50 items each. Have you tested scenario like this?

[kevinherron]

No, because it doesn't make any sense to use 30 threads and a thread per Subscription. PublishingManager handles all of the publishing needed to keep Subscription data flowing without needing a thread per Subscription for some reason.

[jchirantan]

Okay!!
By the way for how many maximum items have you tried the subscription? Because I am thinking of running it for around 5000 items in single subscription request..

[kevinherron]

The Milo client and server do not enforce a limit, but many servers you find in the wild will. You can have tens or hundreds of thousands per Subscription. With this many, depending on the volume of data changes you might want to increase the value of OpcUaClientConfig::getMaxPendingPublishRequests a little bit (not for your 5000 item case), but that's usually more helpful when there are many Subscriptions, not many items in a single Subscription.

[jchirantan]

Hi @kevinherron,

I just checked the logs and found 1 exception which is coming continuously due to which I am not able to move ahead with subscriptions

io.netty.handler.codec.EncoderException: java.lang.NullPointerException: Cannot invoke "org.eclipse.milo.opcua.stack.core.types.structured.SubscriptionAcknowledgement.getSubscriptionId()" because "value" is null: .
2025-12-19 03:03:33.033524: BIP4362E: Java node error: [IICMsgsBundle_en:3013]IIC3013E: Publish request #781 published at 2025-12-19 11:03:33.022Z (2 milliseconds ago) finished at 2025-12-19 11:03:33.024Z (after 2 milliseconds) with error Uncertain (0x40000000) "The operation was uncertain.":
java.util.concurrent.CompletionException: io.netty.handler.codec.EncoderException: java.lang.NullPointerException: Cannot invoke "org.eclipse.milo.opcua.stack.core.types.structured.SubscriptionAcknowledgement.getSubscriptionId()" because "value" is null
java.util.concurrent.CompletionException: io.netty.handler.codec.EncoderException: java.lang.NullPointerException: Cannot invoke "org.eclipse.milo.opcua.stack.core.types.structured.SubscriptionAcknowledgement.getSubscriptionId()" because "value" is null
at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1097)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.eclipse.milo.opcua.stack.transport.client.AbstractUascClientTransport.lambda$sendRequestMessage$1(AbstractUascClientTransport.java:78)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:603)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:505)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:649)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:642)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:131)
at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
at io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:875)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:832)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1136)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:535)
at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:201)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:853)
Caused by: io.netty.handler.codec.EncoderException: java.lang.NullPointerException: Cannot invoke "org.eclipse.milo.opcua.stack.core.types.structured.SubscriptionAcknowledgement.getSubscriptionId()" because "value" is null
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:125)
at io.netty.handler.codec.ByteToMessageCodec.write(ByteToMessageCodec.java:108)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825)
... 8 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.eclipse.milo.opcua.stack.core.types.structured.SubscriptionAcknowledgement.getSubscriptionId()" because "value" is null
at org.eclipse.milo.opcua.stack.core.types.structured.SubscriptionAcknowledgement$Codec.encodeType(SubscriptionAcknowledgement.java:152)
at org.eclipse.milo.opcua.stack.core.types.structured.SubscriptionAcknowledgement$Codec.encodeType(SubscriptionAcknowledgement.java:134)
at org.eclipse.milo.opcua.stack.core.encoding.GenericDataTypeCodec.encode(GenericDataTypeCodec.java:41)
at org.eclipse.milo.opcua.stack.core.encoding.binary.OpcUaBinaryEncoder.encodeStruct(OpcUaBinaryEncoder.java:985)
at org.eclipse.milo.opcua.stack.core.encoding.binary.OpcUaBinaryEncoder.lambda$encodeStructArray$3(OpcUaBinaryEncoder.java:1142)
at org.eclipse.milo.opcua.stack.core.encoding.binary.OpcUaBinaryEncoder.encodeArray(OpcUaBinaryEncoder.java:97)
at org.eclipse.milo.opcua.stack.core.encoding.binary.OpcUaBinaryEncoder.encodeStructArray(OpcUaBinaryEncoder.java:1142)
at org.eclipse.milo.opcua.stack.core.encoding.binary.OpcUaBinaryEncoder.encodeStructArray(OpcUaBinaryEncoder.java:1157)
at org.eclipse.milo.opcua.stack.core.types.structured.PublishRequest$Codec.encodeType(PublishRequest.java:159)
at org.eclipse.milo.opcua.stack.core.types.structured.PublishRequest$Codec.encodeType(PublishRequest.java:138)
at org.eclipse.milo.opcua.stack.core.encoding.GenericDataTypeCodec.encode(GenericDataTypeCodec.java:41)
at org.eclipse.milo.opcua.stack.core.encoding.binary.OpcUaBinaryEncoder.encodeMessage(OpcUaBinaryEncoder.java:949)
at org.eclipse.milo.opcua.stack.transport.client.uasc.UascClientMessageHandler.encode(UascClientMessageHandler.java:215)
at org.eclipse.milo.opcua.stack.transport.client.uasc.UascClientMessageHandler.encode(UascClientMessageHandler.java:74)
at io.netty.handler.codec.ByteToMessageCodec$Encoder.encode(ByteToMessageCodec.java:176)
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107)
... 10 more

I just debugged at my end and found that there is no null check in code (SubscriptionAcknowledgement.java)

I just fixed it at my end

and then it started working properly.
But with these changes, after some time I get Bad_SubscriptionIdInvalid exception and everything stops.

Can you suggest something here?

[jchirantan]

ok, will check and get back to you. Thanks!!

[jchirantan]

By the way, is this implementation right?
#1674 (reply in thread)

[jchirantan]

Hi @kevinherron,
I just tried with your provided changes but was not able to get this IllegalArgumentException.

I just checked the existing stack trace which I have pasted above.
I can see the call has gone from PublishRequest class encodeType Method

and from this method you have directly passed the PublishRequest.getSubscriptionAcknowledgement() to encodeStructArray() method

As SubscriptionAcknowledgement can be null in PublishRequest object, it is possible that this method is sending null in final method which is throwing the NPE.

[kevinherron]

If you follow the encoding methods, if SubscriptionAcknowlegement[] array was null, its inner components don't get encoded. It must be non-null but contain a null element for the NPE you saw to happen.

Make sure you are running the changed code. Try setting a breakpoint on the line that throws the new IllegalArgumentException in case you swallowing / not logging that error somehow.

[jchirantan]

Thanks @kevinherron,
With your help I was able to find the location from where null was getting inserted.
I was creating an array of "SubscriptionAcknowledgement" without adding any value to it (so it was containing default null values in it) and passing it to publishAsync() method.

I have rectified it, and now I am not getting NPE anymore.

[jchirantan]

Hi @kevinherron,
I am facing some strange behavior and want some information about the same.

To subscribe multiple items at same time, I am creting Subscription and over that subscription I am creating MonitoredItems.

Now when I am trying to create MonitoredItems using the "createMonitoredItems()" method, I am getting "Bad_SubscriptionIdInvalid" from server (Sometimes it gets created correctly without any error but sometimes it just throws "Bad_SubscriptionIdInvalid").

Here the subscription is created

and when I am trying to create MonitoredItems then it is giving statuscode "Bad_SubscriptionIdInvalid"

Is this behavior server specific or something can be there from Eclipse Milo SDK end or Something from my code?

Can you suggest something here?

[kevinherron]

There is 20 seconds between creating the Subscription and creating the Monitored Items, and you did not send a PublishRequest to keep the Subscription from timing out - it seems likely the Subscription is just timing out on the Server side.

[jchirantan]

Hi @kevinherron,
One more observation I found that Eclipse Milo SDK client continuously keeps reading the server state automatically.
Can this be a reason of slow processing?

Is there any way to stop this?

[kevinherron]

It reads it every 5 seconds as a keep alive mechanism and to detect “unclean” network connection losses. You can change the interval but I’m not sure you can disable it entirely. It’s almost certainly not your issue.

[jchirantan]

Hi @kevinherron,
Thanks to your detailed support, I was able to resolve issue regarding slowness in performance.

I was able to restore the performance that I was getting with OPC UA Stack that I was using earlier.
I have resolved all the errors that I was getting and all my subscriptions are working properly.

Thank you for the detailed support and guidance. I truly appreciate the time and effort you put into helping me. It made a real difference.