Don't divide CODEQL_RAM in half

[mat-sylvia-mark43]

This code in the autobuild.sh script halves memory between the JVM and TS:

codeql/javascript/resources/tools/autobuild.sh

Lines 7 to 13 in 59a77a8

	# If CODEQL_RAM is set, use half for Java and half for TS.
	if [ -n "${CODEQL_RAM:-}" ] ; then
	half_ram="$(( CODEQL_RAM / 2 ))"
	LGTM_TYPESCRIPT_RAM="$half_ram"
	export LGTM_TYPESCRIPT_RAM
	jvm_args="$jvm_args -Xmx${half_ram}m"
	fi

Since CODEQL_RAM appears to be a default variable, consuming most of the Github runner's memory, this script shouldn't be halving the memory allocation as large TS projects hit a FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory error. How can we override this halving?

[mbg]

Hi @mat-sylvia-mark43 👋

Thanks for raising this. I can see that we recently open-sourced this file, but that it has implemented this behaviour since we added support for JS/TS years ago.

I don't think there is a good way to override this behaviour and it may be a sign that the runner you are using does not have enough memory for the project you are trying to analyse. I.e. it is possible that allocating more memory for TS and less for the JVM will just result in an out-of-memory error elsewhere/in the JVM instead.

However, I am checking this with the relevant engineering team to see if this halving is still appropriate for our needs today. I or one of them will get back to you here once we have looked into this.

[mat-sylvia-mark43]

@mbg Thanks for the details! I can see that codeql is setting: CODEQL_RAM: 29537 and the runner for this has 32GB of memory.

I don't have the option of trying a bigger size runner at this time. Also, this project succeeds in GHA workflows where CodeQL isn't present/conducing a scan...so I really think I need to be able to control/override that dividing logic.

[mbg]

@mat-sylvia-mark43 Thanks for the update! The team discussed this issue and is considering the addition of new environment variable(s) that would allow overriding the default memory halving behaviour. I will post an update here once we have something more concrete.

Also, this project succeeds in GHA workflows where CodeQL isn't present/conducing a scan

CodeQL generally requires more memory than an ordinary build, because of the extra work needed to produce the database. So whether or not a build succeeds without CodeQL present isn't a good indicator for whether it will work on a given machine once CodeQL is enabled.

[mat-sylvia-mark43]

@mbg Thanks! I see your WIP PR also. I'd love to test this if we can somehow orchestrate that.

[mat-sylvia-mark43]

@mbg This OOM issue may be bigger than just this original concern. For example, an 8KB tsx file (FailingOnThisFile.tsx) is making codeql run out of memory. This is the CodeQL analyze log (ubuntu-latest runner = 7GB memory):

[2024-06-25 19:09:43] [build-stdout] Done opening project /home/runner/work/...../tsconfig.json (5254 ms)
  [2024-06-25 19:09:43] [build-stdout] Extracting /home/runner/work/..../FailingOnThisFile.tsx
  [2024-06-25 19:11:02] [build-stderr] <--- Last few GCs --->
  [2024-06-25 19:11:02] [build-stderr] [2337:0x54898b0]    81539 ms: Mark-sweep (reduce) 3835.6 (3899.5) -> 3834.8 (3899.8) MB, 3377.4 / 0.0 ms  (average mu = 0.133, current mu = 0.002) allocation failure; scavenge might not succeed
  [2024-06-25 19:11:02] [build-stderr] [2337:0x54[89](https://github.com/mark43/.../actions/runs/9668023328/job/26671826284#step:5:90)8b0]    84915 ms: Mark-sweep (reduce) 3834.9 (3899.8) -> 3834.8 (3900.0) MB, 3375.7 / 0.0 ms  (average mu = 0.069, current mu = 0.000) allocation failure; GC in old space requested
  [2024-06-25 19:11:02] [build-stderr] <--- JS stacktrace --->
  [2024-06-25 19:11:02] [build-stderr] FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Note that I redacted some details out of this log.

[mbg]

Thanks for the extra information, @mat-sylvia-mark43!

Can you confirm that you run out of memory just with this one file? If so, could you share the file with us or see if you can find a minimal example that reproduces the issue?

[mat-sylvia-mark43]

@mbg Yes, it does indeed run out of memory on this file as the logs above attest to (extracting...then OOM). I can't share this file unfortunately.

[mbg]

What I'm trying to establish is whether there is a problem with this specific file or the project more generally. The log doesn't quite answer that since the whole project is being extracted and it could be that this file just happens to push things over the edge because it's a big project. Alternatively, there could be something specifically wrong with how we extract this file. The way to answer this would be to run an analysis with just this file in the workspace and see whether that also causes an OOM condition or not.

If that confirms that this specific file is responsible and you cannot share that file, then the options you have are:

• If there is something obviously unusual about the file (size, language feature that's only used in this file, etc.), you could let us know here.
• You could comment out the contents of the file and see if that causes the analysis to succeed. If it does, uncomment parts of the file bit by bit until the analysis fails again. Repeat this process until you have a minimal part of the file which causes the problem and share that with us.
• You could open a support ticket and share the file with us there.

[mat-sylvia-mark43]

@mbg After further review, it is not this specific file that the analysis fails on when reviewing more GHA runs...so we can rule that out. Has there been any movement on #16803?

This doc: https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/recommended-hardware-resources-for-running-codeql

indicates the following memory requirements:
Medium (100 K to 1 M lines of code) | 16 GB or higher | 4 or 8 cores

This fails on a runner that has 32GB ram and 8CPU, which is double the requirement above (and the project being scanned is likely <500k lines). This issue is a pretty big blocker. Should I open a support ticket on this again?

[mbg]

After further review, it is not this specific file that the analysis fails on when reviewing more GHA runs. [..] This fails on a runner that has 32GB ram and 8CPU, which is double the requirement above (and the project being scanned is likely <500k lines). This issue is a pretty big blocker. Should I open a support ticket on this again?

That makes things more complicated. Opening a support ticket (or emailing me at mbg@github.com) makes sense if that allows you to share code, logs, any partially generated databases, etc. that you may have to help us troubleshoot this.

Has there been any movement on #16803?

I am following up on this with the team to see if they are happy to merge it, but given that you are seeing considerably higher memory usage than expected during the extraction, I wouldn't necessarily expect this to resolve the cause of the issue.