Introduce Python virtual environments for PEP-668 compliance

[Eebit]

This PR is a potential partial fix for the issues surfaced in #715 (comment).

Python versions since 3.12 have enforced PEP-668, which recommends the use of virtual environments (using venv). Ubuntu 24.04 ships with Python 3.12 by default. This causes some issue with the build tools we have introduced recently (namely the TSA-processing utilities) that rely on the system-level Python. See the linked issue comment for details.

The other option we have (in theory) is to have users install our Python dependencies with --break-system-packages.

In this PR:

• I tried introducing a virtual environment to the Makefile.
  • If the user prefers, they can bring their own virtual environment using the environment variable VENV.
• I added a requirements.txt file which contains the Python dependencies that we have listed in our .github/workflows/build.yml.
• I renormalized the shebangs for the Python scripts we have in-tree.

Left to do:

• Bump the workflow build image to Ubuntu 24.04
• Test on a variety of machines (I tested on Ubuntu 22.04 using WSL2, and native Ubuntu 24.04)
• Update the README.md (I plan to do this if this PR is acceptable to other maintainers)

[CT075]

Seems to build after a make clean for me!

[Land0fChocolate]

I did a make clean and a make -j 8 and got a load of output with this at the end:

arm-none-eabi-as -mcpu=arm7tdmi -mthumb-interwork -I include -g sound/voicegroups/voicegroup090.s -o sound/voicegroups/voicegroup090.o
arm-none-eabi-as -mcpu=arm7tdmi -mthumb-interwork -I include -g sound/voicegroups/voicegroup091.s -o sound/voicegroups/voicegroup091.o
arm-none-eabi-as -mcpu=arm7tdmi -mthumb-interwork -I include -g sound/voicegroups/voicegroup092.s -o sound/voicegroups/voicegroup092.o
./scripts/arm_compressing_linker.py -o data/banim/data_banim.o -t linker_script_banim.txt -b 0x8c02000 -l arm-none-eabi-ld --objcopy arm-none-eabi-objcopy -c ./scripts/compressor.py
tools/mid2agb/mid2agb sound/songs/midi/song001_agbfe3_bgm_opening.mid sound/songs/midi/song001_agbfe3_bgm_opening.s -E -G000 -R020 -P010 -V051
tools/mid2agb/mid2agb sound/songs/midi/song002_agbfe3_bgm_op_theme_i.mid sound/songs/midi/song002_agbfe3_bgm_op_theme_i.s -E -G001 -R020 -P010 -V050
/usr/bin/env: ‘python3\r’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
make: *** [Makefile:198: data/banim/data_banim.o] Error 127
make: *** Waiting for unfinished jobs....

Not perfect, but very promising! It looks like I've nearly got a full build, just missing something.

[CT075]

This error appears to be due to a carriage return character (\r) in the script. I don't see one in my checkout of the file, so I suspect it's being added somehow, maybe by github desktop if you're using that?

[Land0fChocolate]

I used GitHub CLI to checkout this branch.

I've also noticed that when I pulled the branch, something changed one of the files:

Tom@DESKTOP-OLLJRDR MINGW64 ~/Saved Games/fireemblem8u/fireemblem8u (pep-668-fix)
$ git status
On branch pep-668-fix
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   include/constants/msg.h

no changes added to commit (use "git add" and/or "git commit -a")

Tom@DESKTOP-OLLJRDR MINGW64 ~/Saved Games/fireemblem8u/fireemblem8u (pep-668-fix)
$ git diff
warning: in the working copy of 'include/constants/msg.h', LF will be replaced by CRLF the next time Git touches it

Tom@DESKTOP-OLLJRDR MINGW64 ~/Saved Games/fireemblem8u/fireemblem8u (pep-668-fix)

Which is weird because I definitely didn't touch msg.h

[CT075]

Yeah, this is a windowsism. I believe there's a setting in git somewhere that will not replace LF with CRLF, but I don't know what it is offhand.

[Land0fChocolate]

Finding this: https://docs.github.com/en/get-started/git-basics/configuring-git-to-handle-line-endings, it looks like I can adjust line endings behaviors in .gitattributes. I'm seeing if I can add rules for LF line endings and make it work.

[Land0fChocolate]

I've added these rules to .gitattributes and committed them to the branch, and did a git add --renormalize . for good measure:

*.4bpp binary
*.c text eol=lf
*.h text eol=lf
*.s text eol=lf
*.py text eol=lf

Unfortunately, .4bpp doesn't want to play along.

./scripts/compressor.py graphics/portrait/portrait_Aias_tileset.4bpp fk
./scripts/compressor.py graphics/portrait/portrait_Amelia_tileset.4bpp fk
./scripts/compressor.py graphics/portrait/portrait_Anna_tileset.4bpp fk
/usr/bin/env: ‘python3\r’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
make: *** [Makefile:184: graphics/portrait/portrait_Aias_tileset.4bpp.fk] Error 127
make: *** Waiting for unfinished jobs....
/usr/bin/env: ‘python3\r’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
make: *** [Makefile:184: graphics/portrait/portrait_Amelia_tileset.4bpp.fk] Error 127
/usr/bin/env: ‘python3\r’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
make: *** [Makefile:184: graphics/portrait/portrait_Anna_tileset.4bpp.fk] Error 127

I'm puzzled as to how to get those files to behave. make adds a .fk and I'm not sure what that does.

[Eebit]

The *.fk files are just a signifier for a "fake compression" type used by the scripts/compressor.py script when generating the portrait graphics (based on the message of commit af4772b)

Based on the /r in your output after python3, it looks like that script (scripts/compressor.py) may somehow still have the wrong line endings. I'm wondering if you could convert the line endings using Sublime as you mentioned in the issue #716 -- maybe that will get us a bit further. As to why it wasn't renormalized by your .gitattributes rules, I'm not fully sure...

[MokhaLeee]

This problem is essentially a problem of WSL's adaptation to Linux.

A reasonable suspicion is that you did not do git pull directly under WSL, but used git tools under Windows or even unzip after download the pack under windows.

One possible method is to try to use WSL console directly to do git pull. The most reasonable method is not to use WSL but to use Ubuntu OS directly.

[Land0fChocolate]

Yeah, I use a Windows 11 machine so I use a Windows terminal for Git commands and a VM or Linux subsystem for things that require Linux.

So I changed the line endings to Unix manually for all the .py files in the scripts directory.

And SUCCESS!

Still, I wonder why trying to fix the line endings in .gitattributes didn't work.

[rayrobdod]

git add --renormalize only affects git's stage, that command has no effect on the working directory. To have git change the working directory, you need a git checkout command after the attributes have been set.