[BUG] Python interperter distclean isn't working

[PetervdPerk-NXP]

Description / Steps to reproduce the issue

Try to compile python interpreter using

$ cd nuttx
$ make distclean
$ ./tools/configure.sh rv-virt:python
$ make -j$(nproc)
$ ls -l nuttx

Run make distclean or make apps_distclean`
Python compile objects are still there.

Dry run doesn't show removal of the files

make apps_distclean -n
make -C /home/peter/nuttx/apps/interpreters/ distclean APPDIR="/home/peter/nuttx/apps"
rm -f  Kconfig
rm -f  .kconfig

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Ubuntu 22.04

NuttX Version

master

Issue Architecture

[Arch: risc-v]

Issue Area

[Area: Interpreters]

Verification

• I have verified before submitting the report.

[tmedicci]

Hi @PetervdPerk-NXP , were you able to build it properly in the last run?

I remember seeing something similar when I was developing and the sources were not being properly tracked. Can you try cleaning the nuttx and the nuttx-apps repository with git clean -ffdx, trying to build it and then using make distclean, please?

[tmedicci]

I tried and it worked as expected on our docker image:

docker run -it ghcr.io/apache/nuttx/apache-nuttx-ci-linux:latest /bin/bash -c "                                                               
  cd /tmp ; ls -la
  pwd ;
  git clone https://github.com/apache/incubator-nuttx nuttx;
  git clone https://github.com/apache/incubator-nuttx-apps apps ;
  pushd nuttx ;
  git checkout master;         
  echo NuttX Source: https://github.com/apache/nuttx/tree/\$(git rev-parse HEAD) ; popd ;
  pushd apps  ;
  git checkout master;
  echo NuttX Apps: https://github.com/apache/nuttx-apps/tree/\$(git rev-parse HEAD) ; popd ;
  cd nuttx;
  make distclean; ./tools/configure.sh rv-virt:python; make -j$(nproc);
  echo \"Python files before make distclean\"
  ls -la ../apps/interpreters/python;
  echo \"Python files after make distclean\"
  make distclean;
  ls -la ../apps/interpreters/python;
"

The build logs show the folder ../apps/interpreters/python after building the firmware and, then, after a make distclean:

Python files before make distclean
total 99564
drwxr-xr-x  6 root root     4096 Dec 18 13:06 .
drwxr-xr-x 14 root root     4096 Dec 18 13:03 ..
-rw-r--r--  1 root root        0 Dec 18 13:06 .built
-rw-r--r--  1 root root        0 Dec 18 13:05 .depend
-rw-r--r--  1 root root       86 Dec 18 13:03 .gitignore
-rw-r--r--  1 root root     1562 Dec 18 13:03 Kconfig
-rw-r--r--  1 root root     1322 Dec 18 13:03 Make.defs
-rw-r--r--  1 root root    12038 Dec 18 13:05 Make.dep
-rw-r--r--  1 root root     7259 Dec 18 13:03 Makefile
drwxr-xr-x 22 root root     4096 Dec 18 13:05 Python
-rw-r--r--  1 root root      601 Dec 18 13:03 Setup.local
drwxr-xr-x  4 root root     4096 Dec 18 13:05 build
-rw-r--r--  1 root root      723 Dec 18 13:03 config.site
drwxr-xr-x  4 root root     4096 Dec 18 13:05 install
-rw-r--r--  1 root root     4471 Dec 18 13:03 mount_modules.c
-rw-r--r--  1 root root  7633300 Dec 18 13:06 mount_modules.c.tmp.apps.interpreters.python.o
-rw-r--r--  1 root root  7633300 Dec 18 13:06 mount_modules.c.tmp.apps.interpreters.python_1.o
drwxr-xr-x  2 root root     4096 Dec 18 13:03 patch
-rw-r--r--  1 root root     5972 Dec 18 13:06 python.c.tmp.apps.interpreters.python.o
-rw-r--r--  1 root root     5972 Dec 18 13:06 python.c.tmp.apps.interpreters.python_1.o
-rw-r--r--  1 root root 47000166 Dec 18 13:06 romfs_cpython_modules.h
-rw-r--r--  1 root root  7621632 Dec 18 13:06 romfs_cpython_modules.img
-rw-r--r--  1 root root 31968446 Dec 18 13:04 v3.13.0.zip
Python files after make distclean
total 48
drwxr-xr-x  3 root root 4096 Dec 18 13:06 .
drwxr-xr-x 14 root root 4096 Dec 18 13:06 ..
-rw-r--r--  1 root root   86 Dec 18 13:03 .gitignore
-rw-r--r--  1 root root 1562 Dec 18 13:03 Kconfig
-rw-r--r--  1 root root 1322 Dec 18 13:03 Make.defs
-rw-r--r--  1 root root 7259 Dec 18 13:03 Makefile
-rw-r--r--  1 root root  601 Dec 18 13:03 Setup.local
-rw-r--r--  1 root root  723 Dec 18 13:03 config.site
-rw-r--r--  1 root root 4471 Dec 18 13:03 mount_modules.c
drwxr-xr-x  2 root root 4096 Dec 18 13:03 patch

Can you provide more details about your environment?

[PetervdPerk-NXP]

peter@SMW015329:~/nuttx/apps/interpreters/python$ git clean -ffdx
Removing Python/
Removing build/
Removing install/
Removing v3.13.0.zip
peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Setup.local  config.site  mount_modules.c  patch

$ ./tools/configure.sh rv-virt:python
$ make -j$(nproc)

# Lots of output

checking size of fpos_t... 0
checking size of size_t... configure: error: in `/home/peter/nuttx/apps/interpreters/python/build/target':
configure: error: cannot compute sizeof (size_t)
See `config.log' for more details
make[2]: *** [Makefile:107: /home/peter/nuttx/apps/interpreters/python/build/target/Makefile] Error 77
make[1]: *** [Makefile:57: /home/peter/nuttx/apps/interpreters/python_depend] Error 2
make: *** [tools/Unix.mk:630: pass2dep] Error 2
peter@SMW015329:~/nuttx/nuttx$
peter@SMW015329:~/nuttx/nuttx$ make distclean

Compile failure I'll try make distclean

peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Python  Setup.local  build  config.site  install  mount_modules.c  patch  v3.13.0.zip

So yeah most likely make distclean would work on a working build, but a build fail then you can't clean through make distclean

Also which compiler are you using? Because I think my failure has to do with my gcc version.

[tmedicci]

peter@SMW015329:~/nuttx/apps/interpreters/python$ git clean -ffdx
Removing Python/
Removing build/
Removing install/
Removing v3.13.0.zip
peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Setup.local  config.site  mount_modules.c  patch

$ ./tools/configure.sh rv-virt:python
$ make -j$(nproc)

# Lots of output

checking size of fpos_t... 0
checking size of size_t... configure: error: in `/home/peter/nuttx/apps/interpreters/python/build/target':
configure: error: cannot compute sizeof (size_t)
See `config.log' for more details
make[2]: *** [Makefile:107: /home/peter/nuttx/apps/interpreters/python/build/target/Makefile] Error 77
make[1]: *** [Makefile:57: /home/peter/nuttx/apps/interpreters/python_depend] Error 2
make: *** [tools/Unix.mk:630: pass2dep] Error 2
peter@SMW015329:~/nuttx/nuttx$
peter@SMW015329:~/nuttx/nuttx$ make distclean

Compile failure I'll try make distclean

peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Python  Setup.local  build  config.site  install  mount_modules.c  patch  v3.13.0.zip

So yeah most likely make distclean would work on a working build, but a build fail then you can't clean through make distclean

Also which compiler are you using? Because I think my failure has to do with my gcc version.

I see... Yeah, would need to investigate why it fails to clean the files if the build fails too...

About the toolchain, we always recommend using the same CI uses: https://nuttx.apache.org/docs/latest/platforms/risc-v/qemu-rv/boards/rv-virt/index.html#risc-v-toolchain

[PetervdPerk-NXP]

Any idea where the CI downloads the 64 bit of the riscv toolchain I only see this?

###############################################################################
# Build image for tool required by RISCV builds
###############################################################################
FROM nuttx-toolchain-base AS nuttx-toolchain-riscv
# Download the latest RISCV GCC toolchain prebuilt by xPack
RUN mkdir -p riscv-none-elf-gcc && \
  curl -s -L "https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/download/v13.2.0-2/xpack-riscv-none-elf-gcc-13.2.0-2-linux-x64.tar.gz" \
  | tar -C riscv-none-elf-gcc --strip-components 1 -xz

But when building it wants the 64 bit version

peter@SMW015329:~/nuttx/nuttx$ ./tools/configure.sh rv-virt:python
  Copy files
  Select CONFIG_HOST_LINUX=y
  Refreshing...
CP: arch/dummy/Kconfig to /home/peter/nuttx/nuttx/arch/dummy/dummy_kconfig
CP: boards/dummy/Kconfig to /home/peter/nuttx/nuttx/boards/dummy/dummy_kconfig
LN: platform/board to /home/peter/nuttx/apps/platform/dummy
LN: include/arch to arch/risc-v/include
LN: include/arch/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/include
LN: drivers/platform to /home/peter/nuttx/nuttx/drivers/dummy
LN: include/arch/chip to /home/peter/nuttx/nuttx/arch/risc-v/include/qemu-rv
LN: arch/risc-v/src/chip to /home/peter/nuttx/nuttx/arch/risc-v/src/qemu-rv
LN: arch/risc-v/src/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/src
Loaded configuration '.config'
Configuration saved to '.config'
peter@SMW015329:~/nuttx/nuttx$ make
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
Create version.h
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found

[tmedicci]

Any idea where the CI downloads the 64 bit of the riscv toolchain I only see this?

###############################################################################
# Build image for tool required by RISCV builds
###############################################################################
FROM nuttx-toolchain-base AS nuttx-toolchain-riscv
# Download the latest RISCV GCC toolchain prebuilt by xPack
RUN mkdir -p riscv-none-elf-gcc && \
  curl -s -L "https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/download/v13.2.0-2/xpack-riscv-none-elf-gcc-13.2.0-2-linux-x64.tar.gz" \
  | tar -C riscv-none-elf-gcc --strip-components 1 -xz

But when building it wants the 64 bit version

peter@SMW015329:~/nuttx/nuttx$ ./tools/configure.sh rv-virt:python
  Copy files
  Select CONFIG_HOST_LINUX=y
  Refreshing...
CP: arch/dummy/Kconfig to /home/peter/nuttx/nuttx/arch/dummy/dummy_kconfig
CP: boards/dummy/Kconfig to /home/peter/nuttx/nuttx/boards/dummy/dummy_kconfig
LN: platform/board to /home/peter/nuttx/apps/platform/dummy
LN: include/arch to arch/risc-v/include
LN: include/arch/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/include
LN: drivers/platform to /home/peter/nuttx/nuttx/drivers/dummy
LN: include/arch/chip to /home/peter/nuttx/nuttx/arch/risc-v/include/qemu-rv
LN: arch/risc-v/src/chip to /home/peter/nuttx/nuttx/arch/risc-v/src/qemu-rv
LN: arch/risc-v/src/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/src
Loaded configuration '.config'
Configuration saved to '.config'
peter@SMW015329:~/nuttx/nuttx$ make
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
Create version.h
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found

You need to export it to your PATH env var, just like it's done on CI:
https://github.com/apache/nuttx/blob/c1b665c5d5ca5eac130300395597e59d3add0803/tools/ci/docker/linux/Dockerfile#L465

So, basically, it doesn't matter where you download it, as long as you export it to your PATH. I created an alias on my ~/.zshrc (~/.bashrc for Ubuntu, if I'm not mistaken) to automate inserting it to my PATH when needed:

alias get_nuttx='export PATH="/home/tiago/Downloads/xpack-riscv-none-elf-gcc-13.2.0-2/bin:$PATH"'

[PetervdPerk-NXP]

Okay got it now turns out it fails on GCC 10.2, but GCC 13.2 works fine.
On a succesfull build distclean works.

Created /home/peter/nuttx/apps/interpreters/python/build/target/lib/python313.zip (3.11 MiB)
CPP:  /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script-> /home/peter/nuttx/nuttx/boards/risc-v/qeLD: nuttx
riscv-none-elf-ld: warning: /home/peter/nuttx/nuttx/nuttx has a LOAD segment with RWX permissions
Memory region         Used Size  Region Size  %age Used
peter@SMW015329:~/nuttx/nuttx$ make distclean
peter@SMW015329:~/nuttx/nuttx$ ls ../apps/interpreters/python/
Kconfig  Make.defs  Makefile  Setup.local  config.site  mount_modules.c  patch
peter@SMW015329:~/nuttx/nuttx$