Autobuild Error for Javascript-Typescript

[juleszembik]

Hi team,

Looks like my team is experiencing issues with codeql autobuild process. It seems to not be able to find this command:
codeql database trace-command --use-build-mode --working-dir

The full error we're seeing is this:

Error: 1-15 19:51:18] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/hostedtoolcache/CodeQL/2.20.1/x64/codeql/javascript/tools/autobuild.sh]) A fatal error occurred: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/2.20.1/x64/codeql/javascript/tools/autobuild.sh] Error: Encountered a fatal error while running "/opt/hostedtoolcache/CodeQL/2.20.1/x64/codeql/codeql database trace-command --use-build-mode --working-dir /home/runner/work/repo_name/repo_name /home/runner/work/_temp/codeql_databases/javascript". Exit code was 2 and error was: A fatal error occurred: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/2.20.1/x64/codeql/javascript/tools/autobuild.sh]. See the logs for more details.

Is there something missing from the workflow process? The codeQL workflow is pretty standard when running this step.

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v3
      with:
        category: "/language:${{matrix.language}}"

matrix language is javascript-typescript

We're not too entirely sure what could be impacting this issue as it is currently running on Github Hosted Runners. Any direction would be helpful! Thank you.

[redsun82]

👋 @juleszembik I'm sorry to hear you're encountering this issue.

Could you rerun the workflow with debug logging turned on? This will make the workflow upload a debug-artifacts.zip artifact. In there there will be a log directory. You can either attach those logs here or sift through them and report anything that seems related to the error.

That said, what already strikes me from what you wrote is that codeql is being called with --use-build-mode instead of the --index-traceless-dbs that I would expect for javascript analysis. Are you passing languages: javascript (or equivalently javascript-typescript or ${{ matrix.language}}) to your github/codeql-action/init step? Could you share your entire workflow alongside the logs?

[juleszembik]

Hi @redsun82 So from the recent debug-artifact I pulled this last portion of the database-trace-command log

[2025-01-10 21:08:40] [build-stderr] [2003:0x573a5a0]   185467 ms: Mark-Compact 3808.7 (3892.5) -> 3793.8 (3893.5) MB, 3045.50 / 0.00 ms  (average mu = 0.101, current mu = 0.046) allocation failure; scavenge might not succeed
[2025-01-10 21:08:40] [build-stderr] [2003:0x573a5a0]   188676 ms: Mark-Compact 3809.6 (3893.5) -> 3794.7 (3894.5) MB, 3101.72 / 0.00 ms  (average mu = 0.068, current mu = 0.033) allocation failure; scavenge might not succeed
[2025-01-10 21:08:40] [build-stderr] <--- JS stacktrace --->
[2025-01-10 21:08:40] [build-stderr] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
[2025-01-10 21:08:40] [build-stderr] ----- Native stack trace -----
[2025-01-10 21:08:40] [build-stderr]  1: 0xb8d0a3 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
[2025-01-10 21:08:40] [build-stderr]  2: 0xf06250 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
[2025-01-10 21:08:40] [build-stderr]  3: 0xf06537 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
[2025-01-10 21:08:40] [build-stderr]  4: 0x11180d5  [node]
[2025-01-10 21:08:40] [build-stderr]  5: 0x1118664 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
[2025-01-10 21:08:40] [build-stderr]  6: 0x112f554 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
[2025-01-10 21:08:40] [build-stderr]  7: 0x112fd6c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
[2025-01-10 21:08:40] [build-stderr]  8: 0x1106071 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
[2025-01-10 21:08:40] [build-stderr]  9: 0x1107205 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
[2025-01-10 21:08:40] [build-stderr] 10: 0x10e4856 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
[2025-01-10 21:08:40] [build-stderr] 11: 0x1540686 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
[2025-01-10 21:08:40] [build-stderr] 12: 0x7faf134d9ef6 
[2025-01-10 21:09:34] [build-stderr] Exception while extracting /home/runner/work/repo_name/repo_name/lib/client-server/content.ts.
[2025-01-10 21:09:34] [build-stderr] The TypeScript parser wrapper crashed, possibly from running out of memory.
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.ts.extractor.TypeScriptParser.getExceptionFromMalformedResponse(TypeScriptParser.java:396)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.ts.extractor.TypeScriptParser.talkToParserWrapper(TypeScriptParser.java:364)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.ts.extractor.TypeScriptParser.parse(TypeScriptParser.java:434)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.TypeScriptExtractor.extract(TypeScriptExtractor.java:24)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.FileExtractor.extractContents(FileExtractor.java:553)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.FileExtractor.extract(FileExtractor.java:460)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.doExtract(AutoBuild.java:1255)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.extract(AutoBuild.java:1241)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.extractTypeScriptFiles(AutoBuild.java:1194)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.extractTypeScript(AutoBuild.java:1072)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.extractSource(AutoBuild.java:758)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.run(AutoBuild.java:464)
[2025-01-10 21:09:34] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.main(AutoBuild.java:1360)
[2025-01-10 21:09:35] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/hostedtoolcache/CodeQL/2.20.0/x64/codeql/javascript/tools/autobuild.sh])
[2025-01-10 21:09:35] Exception caught at top level: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/2.20.0/x64/codeql/javascript/tools/autobuild.sh]
                      com.semmle.cli2.Commandline.executeAndCheckResult(Commandline.java:170)
                      com.semmle.cli2.Commandline.runWithoutReturn(Commandline.java:123)
                      com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:227)
                      com.semmle.cli2.database.TraceCommandCommand.executeSubcommand(TraceCommandCommand.java:126)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$5(SubcommandCommon.java:893)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:443)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:891)
                      com.semmle.cli2.picocli.SubcommandCommon.toplevelMain(SubcommandCommon.java:775)
                      com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:756)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:238)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:259)
                      com.semmle.cli2.CodeQL.main(CodeQL.java:115)

Last week the command also failed at [2025-01-10 21:05:27] This is codeql database trace-command --index-traceless-dbs --verbosity=progress++ /home/runner/work/_temp/codeql_databases/javascript command as well.

I am a little perplexed that this would fail out with an OOM issue as we use Github Hosted Runners.

[redsun82]

@juleszembik for bigger/more complex projects we do sometimes require use of larger runners to bear the grunt of analysis. On the other hand I'm not sure I ever saw that happen for javascript/typescript analysis, and we did occasionally experience OoM failures in unwarranted situations as the result of bugs that we needed to fix. Could you please share how many lines of javascript/typescript code we are dealing with here, to do an initial rough triage? I've also reached out to our javascript/typescript internal team about this.

[juleszembik]

@redsun82 I calculated around 1.47 million lines of code it looks like. 1479863 total

[juleszembik]

@redsun82 I ran the codeql against a github hosted large runner for the analysis - 8 cores - 32 RAM - 300GB SSD and was able to complete the codeql analysis just fine.

[redsun82]

@juleszembik While we're planning some work to reduce the resources required for javascript analysis, we have not started that work yet. That said, I'd say at the moment the size of the project is in the ballpark of indeed requiring a larger runner than the default one. As that indeed seemed to solve your problem, I'll go ahead and close this issue 🙌