[Bug] oom due to skywalking java agent too many span log

[hxd123456]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

In my program, there is a xxlJob task that takes a long time to execute. In this task, there is a loop that queries the database and asynchronously sends MQ messages. Due to an MQ exception, the thread pool that asynchronously sends MQ messages experiences message backlog, triggering the rejection strategy of the thread pool. As a result, a large number of tasks that send messages are rejected. At this time, OOM is sent. Upon checking the dump log, I found a large number of Skywalking Log DataEntity and KeyValuePair in it

What you expected to happen

In the above situation, the problem will not occur

How to reproduce

1. create an xxlJob task and an asynchronous thread pool (requiring a CountingRejectHandler rejection policy), and start the skywalk thread pool plugin
2. start a long loop in the task
3. submit sleep tasks (sleep (1000)) in an internal loop to trigger rejection policies in the thread pool

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

I don't think this is a proper case for tracing endless task.
Remove xxl plugin, or do manual tracing, make sure trace always ends in a reasonable duration.

[hxd123456]

Log plugin is not mandatory, you added that. We added limits for number of spans, but not for tags or logs. It is hard to tell. If we will add in the future, it could only be 1k or something.

I did not add log plugin, this is just a logging behavior that is recorded when an exception occurs in the thread pool
for example AbstractThreadingPoolInterceptor
public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes, Throwable t) { ContextManager.activeSpan().log(t); }

[wu-sheng]

That is still due to span is not finished, because that error is from your app codes.
If the thing is processed as expected, the span with logs should be sent out.

As I said, even we added that, it still could be a log of logs, we can't limit and say only 10-20 logs.
The use cases are various, and we can't limit too narrow.

[hxd123456]

That is still due to span is not finished, because that error is from your app codes. If the thing is processed as expected, the span with logs should be sent out.

As I said, even we added that, it still could be a log of logs, we can't limit and say only 10-20 logs. The use cases are various, and we can't limit too narrow.

But for users, this is just a long-term task of updating data, and it is impossible to predict such problems during use, or even to inform them that it is due to skywalking, because there is no indication anywhere that the problem may be triggered. For Java agents with low overhead and non-invasive goals, I believe such behavior should be avoided. We can limit the number of logs（500）, just like limiting the number of spans generated, to avoid this problem.

[wu-sheng]

If someone adds this through a pull request, I can review it. There could be many edge cases to cause memory or CPU issues.
I am not arguing whether this is right or not, instead, I was just explaining why this is not there and why others didn't complain.

[hxd123456]

如果有人通过拉取请求添加了这个功能，我可以审核。可能存在许多导致内存或 CPU 问题的边缘情况。 我不是在争论这是否正确，而是在解释为什么没有这个功能，以及为什么其他人没有抱怨。

I want to fix and solve this problem
apache/skywalking-java#773