Git memory issue with armv7 fanned containers

[targos]

See https://ci.nodejs.org/job/node-test-binary-armv7l/RUN_SUBSET=native,nodes=ubuntu2004-armv7l/9651/console

08:51:12 + ssh-agent sh -c 'ssh-add **** && git fetch --no-tags binary_tmp@147.75.72.255:binary_tmp.git +refs/heads/jenkins-node-test-commit-arm-fanned-c7f8b1629123065c20362ca5c2be0552405fbc4a-binary-pi1p/cc-armv7:refs/remotes/jenkins_tmp'
08:51:12 Identity added: **** (****)
08:51:49 fatal: packfile .git/objects/pack/pack-dd8f2842296180e0ffb6376cb0696c662e80f52a.pack cannot be mapped: Cannot allocate memory
08:51:49 fatal: index-pack failed

The .git folder takes quite a bit of space:

$ du -sh .git
324G	.git

I'm not sure how to resolve this other than by wiping the workspace. How can we prevent the repository from growing in the future?

[targos]

I tried to run git gc but it also fails with a similar error.

[richardlau]

I'm not sure how to resolve this other than by wiping the workspace. How can we prevent the repository from growing in the future?

Maybe we need to be running git gc more often? I don't know the history but there's stuff commented out wrt gc in https://ci.nodejs.org/job/node-test-binary-armv7l/configure:

FWIW On jenkins-workspace-7 (where the actual git repo that is pulled from lives), we run a cleanup script on a five minute timer in addition to running git-delete-branches if node-test-commit-arm-fanned succeeds (but neither of those affect the local git checkout on the test machines).

build/ansible/roles/jenkins-workspace/tasks/main.yml

Lines 99 to 111 in 172558b

	- name: Create clean-up script
	copy:
	src: "{{ role_path }}/files/clean_binary_tmp.sh"
	dest: "~binary_tmp/clean_binary_tmp.sh"
	owner: "binary_tmp"
	group: "binary_tmp"
	mode: 0755
	- name: Schedule clean-up script to run daily
	lineinfile:
	line: "0 5 * * * binary_tmp ~binary_tmp/clean_binary_tmp.sh"
	dest: "/etc/crontab"
	regexp: "clean_binary_tmp"

[aduh95]

It might be related: some steps of the ARM jobs are taking a VERY long time, e.g. https://ci.nodejs.org/job/node-test-binary-armv7l/9848/RUN_SUBSET=native,nodes=debian11-armv7l/console took 3 hours:

19:04:14 + ssh-agent sh -c 'ssh-add **** && git fetch --no-tags binary_tmp@147.75.72.255:binary_tmp.git +refs/heads/jenkins-node-test-commit-arm-fanned-201ac057dc753c50d1f6fabc2c14ecc4a2bb9dd3-binary-pi1p/cc-armv7:refs/remotes/jenkins_tmp'
19:04:14 Identity added: **** (****)
22:18:59 From 147.75.72.255:binary_tmp
22:18:59  + eee6517716a8...89af234f8ab4 jenkins-node-test-commit-arm-fanned-201ac057dc753c50d1f6fabc2c14ecc4a2bb9dd3-binary-pi1p/cc-armv7 -> jenkins_tmp  (forced update)