内存缓慢增长问题

[brucelin-cn]

我的一个服务，跑了3个月，内存缓慢增长占用了9G了, 使用debug控制台，使用命令"gc"，没有效果

以下是排查问题步骤及问题，请帮帮指导下
使用debug控制台，使用命令"mem"，
输出例如：

`
:00000004 53.97 Kb (snlua cdummy)
:00000006 45.83 Kb (snlua datacenterd)
:00000007 51.26 Kb (snlua service_mgr)
:00000009 57.22 Kb (snlua console)
:0000000a 175.89 Kb (snlua debug_console 9540)
:0000000b 119.66 Kb (snlua resourceque)
:0000000c 299.00 Kb (snlua mq 1)
:0000000d 63779.40 Kb (snlua db 1)
:0000000e 149.90 Kb (snlua roomidcreator 1)
:0000000f 16467.78 Kb (snlua db 2)
:00000010 299.03 Kb (snlua mq 2)
:00000011 88668.24 Kb (snlua db 3)
:00000012 5988.48 Kb (snlua operations 1)
:00000013 59.68 Kb (snlua clusterd)
:00000014 63143.85 Kb (snlua db 4)
:00000015 117.23 Kb (snlua gate)
:00000016 341.32 Kb (snlua confguard 1)
:00000017 20200.88 Kb (snlua db)
:00000018 285.24 Kb (snlua attendance 1)
:00000019 63279.10 Kb (snlua db 5)
:0000001a 350.95 Kb (snlua notifypushservice 1)
:0000001b 64072.93 Kb (snlua db 6)
:0000001c 81471.72 Kb (snlua db 7)
:0000001d 620.88 Kb (snlua statistics 1)
:0000001e 62970.38 Kb (snlua db 8)
:0000001f 699.87 Kb (snlua storagesync 1)
:00000020 63590.47 Kb (snlua db 9)
:00000021 63229.77 Kb (snlua db 10)
:00000022 8593.73 Kb (snlua im 1)
:00000023 299.74 Kb (snlua mq 3)
:00000024 289.66 Kb (snlua inviter 1)
:00000025 984.52 Kb (snlua roomcenter 1)
:00000026 1539.67 Kb (snlua roomcenter 2)
:00000027 778.72 Kb (snlua roomcenter 3)
:00000028 1078.64 Kb (snlua roomcenter 4)
:00000029 925.09 Kb (snlua roomcenter 5)
:0000002a 1353.56 Kb (snlua roomcenter 6)
:0000002b 1271.99 Kb (snlua roomcenter 7)
:0000002c 1100.00 Kb (snlua roomcenter 8)
:0000002d 1163.23 Kb (snlua roomcenter 9)
:0000002e 1855.96 Kb (snlua roomcenter 10)
:0000002f 971.65 Kb (snlua roomcenter 11)
:00000030 935.88 Kb (snlua roomcenter 12)
:00000031 1796.92 Kb (snlua roomcenter 13)
:00000032 1167.01 Kb (snlua roomcenter 14)
:00000033 1078.80 Kb (snlua roomcenter 15)
:00000034 1873.87 Kb (snlua roomcenter 16)
:00000035 1147.85 Kb (snlua roomcenter 17)
:00000036 1744.08 Kb (snlua roomcenter 18)
:00000037 2085.39 Kb (snlua roomcenter 19)
:00000038 1158.02 Kb (snlua roomcenter 20)
:00000039 276.61 Kb (snlua roomface 1)
:0000003a 299.74 Kb (snlua mq 4)
:0000003b 299.74 Kb (snlua mq 5)
:0000003d 299.03 Kb (snlua mq 6)
:0000003e 299.03 Kb (snlua mq 7)
:0000003f 299.00 Kb (snlua mq 8)
:00000040 299.00 Kb (snlua mq 9)
:00000041 299.03 Kb (snlua mq 10)
:00000042 330.10 Kb (snlua vroom 1)
:00000043 1701.80 Kb (snlua requesthandler 1)
:00000044 2095.44 Kb (snlua roomscreen 1)
:00000045 1552.57 Kb (snlua roommediacontrol 1)
:00000046 1369.51 Kb (snlua requesthandler 2)
:00000047 839.02 Kb (snlua roomscreen 2)
:00000048 826.56 Kb (snlua roommediacontrol 2)
:00000049 1283.50 Kb (snlua requesthandler 3)
:0000004a 1376.58 Kb (snlua roomscreen 3)
:0000004b 888.35 Kb (snlua roommediacontrol 3)
:0000004c 989.02 Kb (snlua requesthandler 4)
:0000004d 1332.56 Kb (snlua roomscreen 4)
:0000004e 1634.53 Kb (snlua roommediacontrol 4)
:0000004f 1601.75 Kb (snlua requesthandler 5)
:00000050 1075.06 Kb (snlua roomscreen 5)
:00000051 1822.67 Kb (snlua roommediacontrol 5)
:00000052 922.04 Kb (snlua requesthandler 6)
:00000053 941.86 Kb (snlua roomscreen 6)
:00000054 952.54 Kb (snlua roommediacontrol 6)
:00000055 1319.99 Kb (snlua requesthandler 7)
:00000056 2322.42 Kb (snlua roomscreen 7)
:00000057 1518.39 Kb (snlua roommediacontrol 7)
:00000058 1755.25 Kb (snlua requesthandler 8)
:00000059 1591.00 Kb (snlua roomscreen 8)
:0000005a 1426.69 Kb (snlua roommediacontrol 8)
:0000005b 1720.13 Kb (snlua requesthandler 9)
:0000005c 965.91 Kb (snlua roomscreen 9)
:0000005d 1786.37 Kb (snlua roommediacontrol 9)
:0000005e 953.84 Kb (snlua requesthandler 10)
:0000005f 1181.59 Kb (snlua roomscreen 10)
:00000060 1354.16 Kb (snlua roommediacontrol 10)
:00000061 1442.38 Kb (snlua requesthandler 11)
:00000062 987.52 Kb (snlua roomscreen 11)
:00000063 1144.25 Kb (snlua roommediacontrol 11)
:00000064 1434.75 Kb (snlua requesthandler 12)
:00000065 986.75 Kb (snlua roomscreen 12)
:00000066 1009.49 Kb (snlua roommediacontrol 12)
:00000067 729.55 Kb (snlua requesthandler 13)
:00000068 1650.59 Kb (snlua roomscreen 13)
:00000069 862.03 Kb (snlua roommediacontrol 13)
:0000006a 1858.99 Kb (snlua requesthandler 14)
:0000006b 1566.72 Kb (snlua roomscreen 14)
:0000006c 1619.66 Kb (snlua roommediacontrol 14)
:0000006d 850.82 Kb (snlua requesthandler 15)
:0000006e 1012.92 Kb (snlua roomscreen 15)
:0000006f 1557.74 Kb (snlua roommediacontrol 15)
:00000070 1715.00 Kb (snlua requesthandler 16)
:00000071 1499.61 Kb (snlua roomscreen 16)
:00000072 2073.26 Kb (snlua roommediacontrol 16)
:00000073 1362.33 Kb (snlua requesthandler 17)
:00000074 2017.03 Kb (snlua roomscreen 17)
:00000075 1887.46 Kb (snlua roommediacontrol 17)
:00000076 1049.10 Kb (snlua requesthandler 18)
:00000077 1557.73 Kb (snlua roomscreen 18)
:00000078 1917.71 Kb (snlua roommediacontrol 18)
:00000079 1876.15 Kb (snlua requesthandler 19)
:0000007a 1872.51 Kb (snlua roomscreen 19)
:0000007b 1807.41 Kb (snlua roommediacontrol 19)
:0000007c 2374.71 Kb (snlua requesthandler 20)
:0000007d 396.30 Kb (snlua roomscreen 20)
:0000007e 359.24 Kb (snlua roommediacontrol 20)
:0000007f 136.20 Kb (snlua roomidcreator)
:00000080 467.62 Kb (snlua httpclient)
:00000081 414.57 Kb (snlua conflog)
:00000082 303.17 Kb (snlua confguard)
:00000083 16416.42 Kb (snlua db)
:00000084 289.66 Kb (snlua inviter)
:00000085 46.33 Kb (snlua RemoteProxy)
:00000086 179.58 Kb (snlua kms)
:00000087 16328.04 Kb (snlua db)
:00000088 593958.83 Kb (snlua mq)
:00000089 57.18 Kb (snlua cfg)
:0000008a 349.12 Kb (snlua s2s)
:0000008b 438.28 Kb (snlua cidproducer)
:0000008c 354.17 Kb (snlua licenseSvr)
:0000008d 285.24 Kb (snlua attendance)
:0000008e 340.91 Kb (snlua im)
:0000008f 1025.97 Kb (snlua call)
:00000090 106.84 Kb (snlua gate)
:00000091 282.48 Kb (snlua servicemonitor)
:00000092 281.67 Kb (snlua webservice)
:00000093 1446.26 Kb (snlua webservice agent)
:00000094 102.24 Kb (snlua clustersender media confsvr-3117 mediasvr 55551)
:00000095 1447.17 Kb (snlua webservice agent)
:00000096 1451.75 Kb (snlua webservice agent)
:00000097 1460.50 Kb (snlua webservice agent)
:00000098 1468.73 Kb (snlua webservice agent)
:00000099 1452.90 Kb (snlua webservice agent)
:0000009a 1440.60 Kb (snlua webservice agent)
:0000009b 1436.29 Kb (snlua webservice agent)
:0000009c 1448.28 Kb (snlua webservice agent)
:0000009d 1439.80 Kb (snlua webservice agent)
:0000009e 146.36 Kb (snlua clustersender record confsvr-3117 confrecordsvr 9701)
:0000009f 1445.03 Kb (snlua webservice agent)
:000000a0 1435.86 Kb (snlua webservice agent)
:000000a1 1448.61 Kb (snlua webservice agent)
:000000a2 1334.44 Kb (snlua webservice agent)
:000000a4 1469.26 Kb (snlua webservice agent)
:000000a5 1472.16 Kb (snlua webservice agent)
:000000a6 1452.70 Kb (snlua webservice agent)
:000000a7 1449.64 Kb (snlua webservice agent)
:000000a8 1460.24 Kb (snlua webservice agent)
:000000a9 1452.99 Kb (snlua webservice agent)
:000000aa 75.50 Kb (snlua safeguard)
:000000b3 71.62 Kb (snlua clusteragent 19 144 104)
:000000c3 84.85 Kb (snlua clusteragent 19 21 120)
:000000c4 75.71 Kb (snlua clusteragent 19 21 121)
:000000c5 66.56 Kb (snlua clusteragent 19 21 122)
:000000c6 79.65 Kb (snlua clusteragent 19 21 123)
:000000c7 75.26 Kb (snlua clusteragent 19 21 124)
:000000c8 76.65 Kb (snlua clusteragent 19 21 125)
:000000c9 76.57 Kb (snlua clusteragent 19 21 126)
:000000ca 76.69 Kb (snlua clusteragent 19 21 127)
:000000cb 66.64 Kb (snlua clusteragent 19 21 128)
:000000cc 66.56 Kb (snlua clusteragent 19 21 129)
:000000cd 66.64 Kb (snlua clusteragent 19 21 130)
:000000ce 72.26 Kb (snlua clusteragent 19 21 131)
:000000cf 67.38 Kb (snlua clusteragent 19 21 132)
:000000d0 80.59 Kb (snlua clusteragent 19 21 133)
:000000d1 66.64 Kb (snlua clusteragent 19 21 134)
:000000d2 77.76 Kb (snlua clusteragent 19 21 135)
:000000d3 99.74 Kb (snlua clusteragent 19 21 136)
:000000d4 116.31 Kb (snlua clusteragent 19 21 137)
:000000d5 76.65 Kb (snlua clusteragent 19 21 138)
:000000d6 71.65 Kb (snlua clusteragent 19 21 139)
:000000d7 108.60 Kb (snlua clusteragent 19 21 140)
:000000d8 71.19 Kb (snlua clusteragent 19 21 141)
:000000d9 76.57 Kb (snlua clusteragent 19 21 142)
:000000da 91.74 Kb (snlua clusteragent 19 21 143)
:000000de 57.44 Kb (snlua clusteragent 19 144 275)
:000000df 62.06 Kb (snlua clustersender callconfsvr-3.confsvr.default.svc.cluster.local:9541 confsvr-3117 confsvr-3.confsvr.default.svc.cluster.local 9541)
:000000e0 57.44 Kb (snlua clusteragent 19 144 278)
:000000e2 57.44 Kb (snlua clusteragent 19 144 284)
:000000e3 57.44 Kb (snlua clusteragent 19 144 297)
:000000e4 62.41 Kb (snlua clustersender callconfsvr-0.confsvr.default.svc.cluster.local:9541 confsvr-3117 confsvr-0.confsvr.default.svc.cluster.local 9541)
:00002ab9 62.06 Kb (snlua clustersender callconfsvr-1.confsvr.default.svc.cluster.local:9541 confsvr-3117 confsvr-1.confsvr.default.svc.cluster.local 9541)
:00002bb1 62.47 Kb (snlua clustersender callconfsvr-2.confsvr.default.svc.cluster.local:9541 confsvr-3117 confsvr-2.confsvr.default.svc.cluster.local 9541)
:00008bff 128.35 Kb (snlua clusteragent 19 144 2717252)
:00008c00 70.11 Kb (snlua clusteragent 19 144 2717271)
:00008c02 130.56 Kb (snlua clusteragent 19 144 2717336)
:00008ca3 173.22 Kb (snlua clusteragent 19 144 2717488)
:000167c0 71.62 Kb (snlua clusteragent 19 144 6975223)
:000167c1 71.62 Kb (snlua clusteragent 19 144 6975234)
:00063ab8 78.65 Kb (snlua clusteragent 19 21 31223688)

`

这样看的话，内存占用最多的是 db 服务

我的问题有3个

1. 看上是是db 服务占用多，但是按 “20200.88 Kb (snlua db)”，内存应该占用2MB，也不多的样子，怎么占用9G了

2. db 服务其实都是 请求redisdb 数据，而且一个db 就一个redisdb 长链接，没有特殊变量存储呀。

3. 唯一一个 全局的roomques={} ,它 存储变量的就是 根据cid 存储了skynet.queue(),但是会通过roomques[cid]=nil 来释放，
   可能是roomques[cid]=nil 这个操作无法释放 skynet.queue(),？

[cloudwu]

还需要看 cmem 和 jmem

[brucelin-cn]

控制台 jmem 命令还不支持
以下是cmem的内容：
cmem.txt
skynet 版本是1.3.0.

[cloudwu]

从 cmem 看，几个大头是：

:000000c3	69967112
:000000c4	57696592
:000000c5	39786040
:000000c6	32773584
:000000c7	24093776
:000000c8	63150480
:000000c9	49144432
:000000ca	49158512
:000000cb	35795728
:000000cc	40321704
:000000cd	33594632
:000000ce	48372672
:000000cf	39777568
:000000d0	69735480
:000000d1	16387072
:000000d2	50165592
:000000d3	140960056
:000000d4	159358368
:000000d5	90108240
:000000d6	32762576
:000000d7	140833200
:000000d8	35539680
:000000d9	85267456
:000000da	98307840
:000000db	15190288
:000000dc	14443552
:000000dd	12919744

对照 mem 看，这些是 clusteragent 。

另外，-2 是网络线程、-1 是主线程。

:fffffffe	671707072
:ffffffff	10528944

#define THREAD_MAIN 1
#define THREAD_SOCKET 2

为什么 clusteragent 会统计在 cmem 而不是 mem 里？

因为它是一个消息转发器，通过 skynet_malloc 分配内存，然后（从网络）转发走。cmem 统计内存是以分配者记录。如果这块内存最终没有释放，就会放在这个统计里。

看起来绝大多数内存消耗在网络上。你可以检查是否有网络包积压（debug console 的 netstat），并检查是否有和网络处理的对应模块产生泄露（包括 skynet 本身）。

[brucelin-cn]

netstat.txt

从netstat看，我没看出来异常信息。

我这边cluster，传输的都是json数据，正常应该不会有泄露吧。

clusteragent 的生命周期应该是listen 端创建的。
我的服务端启动代码：

`
skynet.start(
function()
thisSvrname = f_init()

    skynet.dispatch(
        "lua",
        function(session, address, cmd, ...)
            skynet.info(SERVICE_NAME, session, cmd, ...)
            local f = CMD[cmd]
            if f then
                if session == 0 then
                    f(...)
                else
                    skynet.ret(skynet.pack(f(...)))
                end
            else
                error(string.format("Unknown command %s", tostring(cmd)))
            end
        end
    )
    skynet.register ".call"

    cluster.open(thisSvrname)
end

)

`

难道 其中的 “error(string.format("Unknown command %s", tostring(cmd)))” 这个会引起内存泄露？

[brucelin-cn]

感谢 cloudwu 定位到 clusteragent ，并且给出排查方法，后续有内存的问题，我也可以按这个思路去排查。

[brucelin-cn]

MALLOC_CONF: background_thread:true,dirty_decay_ms:0,muzzy_decay_ms:0

我加上这个可能有效果？
@cloudwu

[huahua132]

可以看看是不是这个问题，建议先翻看1.3版本之后cluster相关的bug修复。
#1150

[brucelin-cn]

应该就是它了。感谢感谢。