Double Take memory usage experiences

[leccelecce]

Does anyone have any stats around memory usage for their Double Take instance?

I've been running the jakowenko build set to 512MB hard limit for some time. More recently I moved to this build and have had a number of OOM killer instances at 512MB. I'm familiar with debugging Java memory usage but not Node so before trying to look into this and see where the memory is going, I was interested in seeing if others are tending to allocate more resources? I didn't see any guidance or previous discussion on memory usage.

For the time being I've doubled memory limit to 1GB and will see what happens - for now usage seems to have settled on around 600MB.

Linux OOM killer interestingly seems to report in the final line that total-vm:11712604kB - 11GB - I'm wondering if there's something funny with the /dev/shm side of things...

[Sun Oct  1 10:28:18 2023] node invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[Sun Oct  1 10:28:18 2023] CPU: 2 PID: 1587066 Comm: node Not tainted 5.10.0-25-amd64 #1 Debian 5.10.191-1
[Sun Oct  1 10:28:18 2023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
[Sun Oct  1 10:28:18 2023] Call Trace:
...
[Sun Oct  1 10:28:18 2023] memory: usage 524288kB, limit 524288kB, failcnt 7590
[Sun Oct  1 10:28:18 2023] swap: usage 104284kB, limit 524288kB, failcnt 0
[Sun Oct  1 10:28:18 2023] Memory cgroup stats for /system.slice/docker-d7e5798fdcd705bb8bbdf1a51e5e1ba6c862b01c96d58037408fc5f66b216b94.scope:
[Sun Oct  1 10:28:18 2023] anon 451620864
                           file 2162688
                           kernel_stack 442368
                           percpu 0
                           sock 3239936
                           shmem 2162688
                           file_mapped 0
                           file_dirty 0
                           file_writeback 270336
                           anon_thp 276824064
                           inactive_anon 399921152
                           active_anon 123875328
                           inactive_file 0
                           active_file 0
                           unevictable 0
                           slab_reclaimable 2756584
                           slab_unreclaimable 509312
                           slab 3265896
                           workingset_refault_anon 528
                           workingset_refault_file 594
                           workingset_activate_anon 33
                           workingset_activate_file 594
                           workingset_restore_anon 0
                           workingset_restore_file 66
                           workingset_nodereclaim 0
                           pgfault 1336731
                           pgmajfault 33
                           pgrefill 6924
                           pgscan 197190
                           pgsteal 22241
                           pgactivate 129261
                           pgdeactivate 6924
                           pglazyfree 0
                           pglazyfreed 0
                           thp_fault_alloc 2409
                           thp_collapse_alloc 33
[Sun Oct  1 10:28:18 2023] Tasks state (memory values in pages):
[Sun Oct  1 10:28:18 2023] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Sun Oct  1 10:28:18 2023] [1586979]     0 1586979   188370     8015   696320     3048             0 node
[Sun Oct  1 10:28:18 2023] [1587066]     0 1587066  2928151   122921  4616192    23023             0 node
[Sun Oct  1 10:28:18 2023] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=docker-d7e5798fdcd705bb8bbdf1a51e5e1ba6c862b01c96d58037408fc5f66b216b94.scope,mems_allowed=0,oom_memcg=/system.slice/docker-d7e5798fdcd705bb8bbdf1a51e5e1ba6c862b01c96d58037408fc5f66b216b94.scope,task_memcg=/system.slice/docker-d7e5798fdcd705bb8bbdf1a51e5e1ba6c862b01c96d58037408fc5f66b216b94.scope,task=node,pid=1587066,uid=0
[Sun Oct  1 10:28:18 2023] Memory cgroup out of memory: Killed process 1587066 (node) total-vm:11712604kB, anon-rss:440664kB, file-rss:51020kB, shmem-rss:0kB, UID:0 pgtables:4508kB oom_score_adj:0

[skrashevich]

Interesting. can you run api/server with "trace-gc" node argument (node --trace-gc), wait for problem and collect output?

[leccelecce]

Sorry, not all that familiar with nodemon, should I just shell into the running container, kill the node and nodemon processes and manually call node --trace-gc api/server.js?

As a side point, I have noted that even though the Linux OOM monitor has killed the node process, the container keeps running, I presume because it's monitoring for nodemon to terminate, which it doesn't. This shows to me that this approach is a bit brittle.

Would you object to adding wget here: https://github.com/skrashevich/double-take/blob/beta/.build/Dockerfile#L167C14-L167C14 (which I believe is where the final image is prepared) so that those of us who want to specify a Docker healthcheck can do so? This would enable something like the below in Docker:

wget --spider --no-verbose --tries=1 http://localhost:3000 || exit 1

...which would mean DT actually gets restarted if there's an issue in the container. I'd mor e normally suggest curl but that seems to come with quite a few more dependencies.

[skrashevich]

Would you object to adding wget here: https://github.com/skrashevich/double-take/blob/beta/.build/Dockerfile#L167C14-L167C14 (which I believe is where the final image is prepared) so that those of us who want to specify a Docker healthcheck can do so? This would enable something like the below in Docker:

wget --spider --no-verbose --tries=1 http://localhost:3000 || exit 1

...which would mean DT actually gets restarted if there's an issue in the container. I'd mor e normally suggest curl but that seems to come with quite a few more dependencies.

You can use curl for this, which is already installed in the container and is a more suitable utility for this task

[leccelecce]

If I attach to the container and run curl, I get command not found. If I run apt update && apt install curl, it goes ahead and installs it.