Cygwin build is broken again

[patacongo]

make[2]: Entering directory '/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/libs/libc'
.\assert/lib_assert.c:25:10: fatal error: nuttx/arch.h: No such file or directory
   25 | #include <nuttx/arch.h>
      |          ^~~~~~~~~~~~~~
compilation terminated.
ERROR: arm-none-eabi-gcc failed: 1
       command: arm-none-eabi-gcc -MT kbin\\lib_assert.o  -M '-fno-builtin' '-funwind-tables' '-Wall' '-Wstrict-prototypes' '-Wshadow' '-Wundef' '-Os' '-fno-strict-aliasing' '-fomit-frame-pointer' '-fno-strength-reduce' '-mcpu=cortex-m4' '-mthumb' '-mfloat-abi=soft' '-isystem' 'D:SpudaDocumentsprojectsnuttxmasternuttx_forkinclude' '-D__NuttX__' '-pipe' '-I' 'D:SpudaDocumentsprojectsnuttxmasternuttx_forklibslibc' '-D__KERNEL__' .\\assert/lib_assert.c
make[2]: *** [/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/tools/Config.mk:188: lib_assert.ddc] Error 1
make[2]: Leaving directory '/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/libs/libc'

Failure occurred on first compilation of stm32f4discovery:kostest configuration. The problem probably does not depend on configuration. Most likely all Cygwin builds that use a Windows native toolchain are broken. But perhaps this is limited to Cygwin 2 pass builds with a Windows native toolchain only?

[patacongo]

I took the time to do the git bisect to find the comment that caused this build failure. On each step of the bisection, I did the following:

make distclean
git checkout <revision>
tools/configure.sh -c stm32f4discovery:kostest
make staging/libc.a
git bisect <good|bad> <revision>

The final step of the bisection reported:

$ git bisect bad
d5b6ec450fde069a4e64569b0eb7e4fcb3b96e83 is the first bad commit
commit d5b6ec450fde069a4e64569b0eb7e4fcb3b96e83
Author: Matias N <matias@protobits.dev>
Date:   Wed Nov 18 13:44:58 2020 -0300

    Parallelize depend file generation

 arch/arm/src/Makefile       |  7 +++++--
 arch/avr/src/Makefile       |  7 +++++--
 arch/hc/src/Makefile        |  7 +++++--
 arch/mips/src/Makefile      |  7 +++++--
 arch/misoc/src/Makefile     |  7 +++++--
 arch/or1k/src/Makefile      |  7 +++++--
 arch/renesas/src/Makefile   |  6 +++++-
 arch/risc-v/src/Makefile    |  7 +++++--
 arch/sim/src/Makefile       |  7 +++++--
 arch/x86/src/Makefile       |  7 +++++--
 arch/x86_64/src/Makefile    |  7 +++++--
 arch/xtensa/src/Makefile    |  7 +++++--
 audio/Makefile              |  6 +++++-
 binfmt/Makefile             |  6 +++++-
 boards/Makefile             | 11 +++++------
 crypto/Makefile             |  6 +++++-
 drivers/Makefile            |  6 +++++-
 fs/Makefile                 |  6 +++++-
 graphics/Makefile           |  6 +++++-
 libs/libc/Makefile          | 12 ++++++++++--
 libs/libc/bin/Makefile      |  2 +-
 libs/libc/zoneinfo/Makefile |  6 +++++-
 libs/libdsp/Makefile        |  6 +++++-
 libs/libnx/Makefile         | 12 ++++++++++--
 libs/libnx/kbin/Makefile    |  2 +-
 libs/libxx/Makefile         |  6 +++++-
 mm/Makefile                 | 12 ++++++++++--
 net/Makefile                |  6 +++++-
 openamp/Makefile            |  6 +++++-
 pass1/Makefile              |  6 +++++-
 sched/Makefile              |  6 +++++-
 syscall/Makefile            |  7 +++++--
 tools/Config.mk             | 17 +++++++++++++++++
 video/Makefile              |  6 +++++-
 wireless/Makefile           |  6 +++++-
 35 files changed, 194 insertions(+), 54 deletions(-)

[protobits]

I will take a look and try to infer the failure, as I'm really not familiar with CYGWIN.
From the looks of it the include path is missing the delimiter.

[btashton]

Now that we have the minimal builds for macOS I can look at bringing back the Cygwin CI, previously it was super slow and I never got to the bottom of that.

[protobits]

That would be super useful since it would also allow me to validate the cmake changes

[btashton]

I'm taking a break for the US holiday weekend l, but I'll push this up the priority after along with the nxflat changes from last week that seem to be good and should be added to CI.

[protobits]

@patacongo did you continue the bisection to the end? If I look at the diff of that commit I only see changes to the depend stage, which should not affect the build itself. Or is that during depend target?

[patacongo]

From the looks of it the include path is missing the delimiter.

No, the delimiters were there but disappeared. For quoted values like \, one of the backslashes will disappear from the Bash command line each time it is processed. You can see the good values a little clearer with V=1:

make[2]: Leaving directory '/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/libs/libc'
make makekdepfile CFLAGS="-fno-builtin -funwind-tables -Wall -Wstrict-prototypes -Wshadow -Wundef -Os -fno-strict-aliasing -fomit-frame-pointer -fno-strength-reduce -mcpu=cortex-m4 -mthumb -mfloat-abi=soft -isystem "D:\Spuda\Documents\projects\nuttx\master\nuttx_fork\include" -D__NuttX__  -pipe -I "D:\Spuda\Documents\projects\nuttx\master\nuttx_fork\libs\libc" -D__KERNEL__"
make[2]: Entering directory '/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/libs/libc'
/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/tools/mkwindeps.sh --obj-suffix .o --dep-path assert --dep-path ctype --dep-path dirent --dep-path endian --dep-path errno --dep-path fixedmath --dep-path grp --dep-path inttypes --dep-path libgen --dep-path machine/arm/armv7-m --dep-path machine/arm --dep-path misc --dep-path net --dep-path pthread --dep-path pwd --dep-path queue --dep-path sched --dep-path semaphore --dep-path signal --dep-path spawn --dep-path stdio --dep-path stdlib --dep-path string --dep-path symtab --dep-path syslog --dep-path time --dep-path tls --dep-path uio --dep-path unistd --dep-path uuid "arm-none-eabi-gcc" -- -fno-builtin -funwind-tables -Wall -Wstrict-prototypes -Wshadow -Wundef -Os -fno-strict-aliasing -fomit-frame-pointer -fno-strength-reduce -mcpu=cortex-m4 -mthumb -mfloat-abi=soft -isystem D:SpudaDocumentsprojectsnuttxmasternuttx_forkinclude -D__NuttX__  -pipe -I D:SpudaDocumentsprojectsnuttxmasternuttx_forklibslibc -D__KERNEL__ -- assert/lib_assert.c > lib_assert.ddc
.\assert/lib_assert.c:25:10: fatal error: nuttx/arch.h: No such file or directory
   25 | #include <nuttx/arch.h>
      |          ^~~~~~~~~~~~~~
compilation terminated.
ERROR: arm-none-eabi-gcc failed: 1
       command: arm-none-eabi-gcc -M '-fno-builtin' '-funwind-tables' '-Wall' '-Wstrict-prototypes' '-Wshadow' '-Wundef' '-Os' '-fno-strict-aliasing' '-fomit-frame-pointer' '-fno-strength-reduce' '-mcpu=cortex-m4' '-mthumb' '-mfloat-abi=soft' '-isystem' 'D:SpudaDocumentsprojectsnuttxmasternuttx_forkinclude' '-D__NuttX__' '-pipe' '-I' 'D:SpudaDocumentsprojectsnuttxmasternuttx_forklibslibc' '-D__KERNEL__' .\\assert/lib_assert.c
make[2]: *** [/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/tools/Config.mk:209: lib_assert.ddc] Error 1
make[2]: Leaving directory '/cygdrive/d/Spuda/Documents/projects/nuttx/master/nuttx_fork/libs/libc'

[patacongo]

@patacongo did you continue the bisection to the end? If I look at the diff of that commit I only see changes to the depend stage, which should not affect the build itself. Or is that during depend target?

Yes:

$ git bisect good
Bisecting: 1 revision left to test after this (roughly 1 step)
[9ceb61d3a9ce3ca730d1e735349f7d5415d1b97f] risc-v 64-bit: Fix SCN/PRI.PTR definitions

$ git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[d5b6ec450fde069a4e64569b0eb7e4fcb3b96e83] Parallelize depend file generation

$ git bisect bad
d5b6ec450fde069a4e64569b0eb7e4fcb3b96e83 is the first bad commit
commit d5b6ec450fde069a4e64569b0eb7e4fcb3b96e83
...

[protobits]

I'm guessing it is due to CFLAGS not being inside quotes here:
https://github.com/apache/incubator-nuttx/blob/eb133ad527ebea7c190c1d0b3eb60f97e4d9349d/tools/Config.mk#L185
Could you add the quotes and see if that fixes it?

[patacongo]

Now that we have the minimal builds for macOS I can look at bringing back the Cygwin CI, previously it was super slow and I never got to the bottom of that.

Cygwin is slower than Linux for two reasons: First, the real time virus checking that you get with Windows. When I am am doing lots of builds, I normally turn of realtime virus checking during the build. The speeds thing up by maybe 40%

The other reason, which we can't do anything about, is due to the layer of simulated POSIX interfaces. In particular, fork(). All of the GCC build tools do lots of forks() but Windows does not support fork() and my understanding is that the processing to emulate fork() behavior is slow. You can see this if you make a tight loop in a Bash script and run some do-almost-nothing process as fast as you can. You will find that this is limited to something like 7 forks per second.