stack size exceeded on M1/arm64 but not on x86

[ggascoigne]

Version

v16.13.1

Platform

Darwin USAGPM.local 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:41 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T6000 arm64

Subsystem

No response

What steps will reproduce the bug?

I have a project with about 1000 typescript files (it's private so I can't point you at it, sorry). It's a react project that uses create-react app.

The arm version of v16 gives:

Failed to compile.

undefined
fork-ts-checker-webpack-plugin error in undefined(undefined,undefined):
Maximum call stack size exceeded  TSINTERNAL

The x86 version works fine. I've tried every arm version of v16 that I can find, all have the same issue, every x86 version works with no problem. Running yarn tsc explicitly works fine with both the arm and the x86 version.

How often does it reproduce? Is there a required condition?

With the affected project it happens 100% of the time, with other projects I can't reproduce this.

What is the expected behavior?

x86 and arm behavior should be identical.

What do you see instead?

No response

Additional information

No response

[ggascoigne]

BTW I realise that without a project to test this on this will be difficult to do anything with, I'm more than happy to jump through hoops to try and get you any further information that can help pin this down if it comes to that.

[shauninman]

Building the open source Ace (Ajax.org Cloud9 Editor) also exhibits the problem. Steps to reproduce on an arm64 Mac:

git clone https://github.com/ajaxorg/ace
cd ace
npm install
make clean && node Makefile.dryice.js normal --m --nc

[bnoordhuis]

I can't test locally but I know the cause: V8 on arm64 assumes there's at most 864 kb of stack space (984 kb on other architectures.)

Check what ulimit -s prints and start node with node --stack_size=<kb> app.js. Important: if you set it too high, you risk crashes.

[targos]

@bnoordhuis can we close this issue or there's something Node.js can do?

[bnoordhuis]

Node should probably check getrlimit(RLIMIT_STACK) at start-up.

[mirabilos]

864 kb (kilobits)? This is highly unusual, stack is normally 4096 or 8192 KiB, in most settings.

I do have a public reproducer for you, first reported in Debian. I can reproduce this on the Debian/arm64 porterbox:

Linux amdahl 5.10.0-21-arm64 #1 SMP Debian 5.10.162-1 (2023-01-21) aarch64 GNU/Linux

processor       : 0                                                                                              
BogoMIPS        : 100.00                                                                                         
Features        : fp asimd evtstrm cpuid                                                                         
CPU implementer : 0x50                                                                                           
CPU architecture: 8                                                                                              
CPU variant     : 0x0                                                                                            
CPU part        : 0x000                                                                                          
CPU revision    : 1                                                                                              

Here’s it using the latest nodejs release (though I’m still using Babel etc. from Debian because that’s quicker for me right now than using npm):

(sid_arm64-dchroot)tg@amdahl:~$ wget https://github.com/danvk/dygraphs/archive/refs/tags/v2.2.0.tar.gz           
(sid_arm64-dchroot)tg@amdahl:~$ tar xaf v2.2.0.tar.gz 
(sid_arm64-dchroot)tg@amdahl:~$ cd dygraphs-2.2.0
(sid_arm64-dchroot)tg@amdahl:~/dygraphs-2.2.0$ env NODE_PATH=/usr/share/node_modules ../node-v19.6.0-linux-arm64/bin/node /usr/bin/babeljs --config-file $PWD/babel.config.json --compact false --source-maps inline -d tests5 auto_tests                                                                                                          
RangeError: Maximum call stack size exceeded                                                                     
    at isBlockStatement (/usr/share/nodejs/@babel/types/lib/validators/generated/index.js:381:26)                
    at isScope (/usr/share/nodejs/@babel/types/lib/validators/isScope.js:9:39)                                   
    at NodePath.isScope (/usr/share/nodejs/@babel/traverse/lib/path/lib/virtual-types-validator.js:107:10)       
    at NodePath.getScope (/usr/share/nodejs/@babel/traverse/lib/path/index.js:85:17)                             
    at NodePath.setScope (/usr/share/nodejs/@babel/traverse/lib/path/context.js:115:21)                          
    at NodePath.setContext (/usr/share/nodejs/@babel/traverse/lib/path/context.js:128:8)                         
    at NodePath.pushContext (/usr/share/nodejs/@babel/traverse/lib/path/context.js:183:8)                        
    at TraversalContext.visitQueue (/usr/share/nodejs/@babel/traverse/lib/context.js:78:14)                      
    at TraversalContext.visitSingle (/usr/share/nodejs/@babel/traverse/lib/context.js:65:19)                     
    at TraversalContext.visit (/usr/share/nodejs/@babel/traverse/lib/context.js:109:19)                          
(sid_arm64-dchroot)tg@amdahl:~/dygraphs-2.2.0$ env NODE_PATH=/usr/share/node_modules ../node-v19.6.0-linux-arm64/bin/node --stack-size="$(ulimit -s)" /usr/bin/babeljs --config-file $PWD/babel.config.json --compact false --source-maps inline -d tests5 auto_tests
Successfully compiled 59 files with Babel (9209ms).

An npm i after the cd should do the trick for you as well.

This probably affects many; I’ve seen reports of people trying to use some äpps on Android on arm64 running into this bug. I found it during Debian reproducible-builds testing. Others run into this because they use arm64 machines for development.

[mirabilos]

Hi James, (you might wish to Cc ***@***.***> so they actually get the reply…)

Are you able to determine whether #41163 (and/or any of the guidance within that thread) seems relevant to this bug?

It appears so. I commented there, thank you for finding that link. I guess there is even a… quick patch… to make from this. I admit I’m very confused by that statement: “if you set it too high, you risk crashes” That can’t be right. Searching through the nodejs source for where this stack size is set, I see multiple time bombs for all architectures. deps/v8/src/common/globals.h does set the default stack size to 864/984 KiB in order to achieve an about 1 MiB stack for JS plus C++ parts combined. I wonder if this shouldn’t be getrlimit(RLIMIT_STACK) - overhead, and then define the per-architecture overhead in a suitable way. That lower 1 MiB total limit seems to be for Windows. The lower arm64 limit is caused by “allocating MacroAssembler takes 120 [KiB]” but the total could still be raised I think… at least on unixoid platforms other than WebView-on-Android. Since the location of these defaults is in v8, it also applies for browsers and whatnot, but nodejs could indeed inspect the current ulimit and set a better default for at least nōn-Windows systems. I’m not, unfortunately, in the position to provide a quick patch, being a C developer, not CFrustFrust, and all that. I think that InitializeNodeWithArgs in src/node.cc, which already has a call to V8::SetFlagsFromString(NODE_V8_OPTIONS, …), is the likely place for adding code (suitably platform-ifdef’d) that does: - get the ulimit - subtract some arch-specific overhead target - check that that’s positive (or >= V8_DEFAULT_STACK_SIZE_KB even, that might be a good idea) - if so, pass this as synthetic --stack-size (or --stack_size?) to v8, overriding its default but allowing for a later option given by the user’s argv[] to override _that_, again Might need to adjust some tests, too :~ Good luck, //mirabilos -- <igli> exceptions: a truly awful implementation of quite a nice idea. <igli> just about the worst way you could do something like that, afaic. <igli> it's like anti-design. <mirabilos> that too… may I quote you on that? <igli> sure, tho i doubt anyone will listen ;)

[mirabilos]

ooooh and I think I get why “risk crashes” is mentioned here: setting --stack_size="$(ulimit -s)" omits the subtraction of the per-arch CFrustFrust overhead, so that’s not a consideration for the proposed way to fix this

[bnoordhuis]

“if you set it too high, you risk crashes”

That can’t be right.

--stack_size=... doesn't set the stack size, it just tells V8 to assume it's this big. Perennial source of confusion.

[mirabilos]

Ben Noordhuis dixit:

`--stack_size=...` doesn't set the stack size, it just tells V8 to assume it's _this_ big. Perennial source of confusion.

Aieee. OK, then setting it to the ulimit is probably right? Eh I’ll leave wondering about details to the devs. bye, //mirabilos --

> Why don't you use JavaScript? I also don't like enabling JavaScript in Because I use lynx as browser.

+1 -- Octavio Alvarez, me and ⡍⠁⠗⠊⠕ (Mario Lang) on debian-devel

[mirabilos]

James Addison dixit:

Maybe it's rare to propose 'do nothing' as a technical suggestion but I think it is worth considering here, since we are not the arbiters of Node.

It’s still a release-critical bug in Debian which impacts arm64 builders including reproducible-builds. I would see this fixed in bookworm, at least by a band-aid (raising these limits by default); the proper fix can be revisited after the release, and in coordination with upstream. We know the default ulimits for users in Debian, and they are higher than the 1 MiB assumed by v8, by quite some factor, so this won’t break things which are not currently broken (by that exception). This will do for the release I think. If a proper, upstream-supported, fix arrives within $time it’s even possible that a bookworm-security update includes that. Still thinking along the line of getrlimit + subtract known arch-specific value for the fix here, whatever makes sense with the v8 setting anyway… bye, //mirabilos -- Gestern Nacht ist mein IRC-Netzwerk explodiert. Ich hatte nicht damit gerechnet, darum bin ich blutverschmiert… wer konnte ahnen, daß SIE so reagier’n… gestern Nacht ist mein IRC-Netzwerk explodiert~~~ (as of 2021-06-15 The MirOS Project temporarily reconvenes on OFTC)

[mirabilos]

Package: nodejs Followup-For: Bug #1030284 X-Debbugs-Cc: ***@***.***, ***@***.*** mirabilos gesagt:

We know the default ulimits for users in Debian, and they are higher than the 1 MiB assumed by v8, by quite some factor, so this won’t break things which are not currently broken (by that exception). This will do for the release I think.

Relaying my understanding of this, so far: An increase in the V8 stack size should not cause earlier-process-exits for any processes that previously ran on Debian systems where the architecture-default-or-greater stack size is configured[1]. In other words: the same-number-or-greater of JavaScript processes should continue to run on any given Debian system where the configured stack size is greater-than-or-equal to the architecture's default, after the V8 stack size limit is increased. And we expect that it should repair a failing reproducible build test for at least one Debian package on arm64. [1] - see limits.conf

[jayaddison]

That comment was written and sent by me. I didn't realize that the GitHub reply-to email address is user-specific and that messages sent to it will appear with them as the author.