Skip to content

[BUG]: use GrpcDtpAdapter occur OOM problem, DtpRunnable cannot be GC, stopWatchMap is always referenced #584

@maplewang-sky

Description

@maplewang-sky

平台

Linux

Jdk版本

JDK21

SpringBoot版本

3.4.1

DynamicTp版本

1.1.9.1-3.x

配置中心类型

No response

配置中心版本

No response

问题描述

  1. We initially used 1.1.9.1-3.x. After referencing dynamic-tp-spring-boot-starter-adapter-grpc, the grpc client reported an error when calling the grpc interface of other services.:rpc call result is null:INTERNAL: Panic! This is a bug! The reason for the error should be here: Fix the grpc client channel executor is shutdown #520
Image Image
  1. Then we upgraded to version 1.2.x and looked at the source code of DTP. The above problem had been fixed when adapting to grpc, that is, the original thread pool would not be closed.
Image
  1. However, because 1.2.x is a major version, the formats of some packages and DTP configuration files have also changed. In order to reduce the impact on the business layer, we still maintain the 1.1.9.1-3.x version, but the GrpcDtpAdapter of the 1.1.9.1-3.x version is changed to use the method after the 1.2.x fix, which is not to close the original thread pool.
Image
  1. We have discovered another problem. The memory usage of the application will gradually increase.
Image Let's go into the container and execute:jmap -histo:live 1 > live_histogram.txt,I saw that there was an object that was not dropped by GC and kept rising.

org.dromara.dynamictp.core.support.task.runnable.DtpRunnable

Image

We dumped the file and analyzed it with MAT, jmap -dump:live,format=b,file=heap.bin 1
It was found that this DtpRunnable has been referenced and cannot be recycled

Image
  1. But we have seen here that the grpc thread will go to afterExecute when it completes the task, and the AwareManager.afterExecute inside will be removed, so I don’t know why. At present, as long as we are not suitable for grpc, there will be no problem if we delete GrpcDtpAdapter.
Image Image

复现步骤

n version 1.1.9.1-3.x, using the GrpcDtpAdapter processing method in 1.2.x will cause OOM problems (we have not tried to use the 1.2.x version completely)

其他信息

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions