[GR-73086] additional libc backend

[ahgamut]

This patch adds support to build native-image binaries with Cosmopolitan Libc. More specifically:

• CosmoLibC added as a subclass of LibCBase
• --libc=cosmo added to SubstrateOptions
• POSIX headers and related C directives duplicated with modifications
• stub functions added to libchelper
• added BooleanSupplier classes to check for libc usage
• implementation of -fpatchable-function-entry flag (as in gcc/clang) to support debugging

This enables building binaries with native-image that can be converted to Actually Portable Executables.

1. build the cosmocc compiler suite with this patch of Cosmopolitan Libc
2. run make static-libs-image using this patch of labs JDK
3. build native-image as usual with this patch on the host system
4. use the built native-image with --libc=cosmo and provide necessary flags to build binaries

Example build recipes for Github Actions are given here.

[oracle-contributor-agreement]

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

• PR author: ahgamut
• ahgamut@gmail.com (@ahgamut)
• ahgamut@elide.dev
• gautham@elide.dev

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

[sgammon]

relates to #8350

[oracle-contributor-agreement]

Thank you for signing the OCA.

[oubidar-Abderrahim]

Thank you for your contribution @ahgamut
@wirthi could you please assign/review this PR?

[olpaw]

@sgammon thank you for this very interesting PR. We’re currently investigating whether merging it is feasible and practical. As far as I can tell, adding Cosmopolitan support in NI depends on changes in other projects that we don’t control. Is Justine Tunney willing to integrate your Cosmo changes as-is? From what I see, these are largely stub implementations of C library functionality that Cosmo is currently missing, but that LabsJDK (and the NI build) depends on at build time. Ideally, there would be an official Cosmopolitan release that includes your changes, which we could then build on.

Regarding the LabsJDK changes, I have another concern: they further increase our delta to OpenJDK and therefore increase our maintenance effort. Would you be willing to minimize these changes as much as possible and help resolve any merge conflicts that might arise in the future?

Also, could you elaborate on the need for the AArch64-specific register changes that Cosmo seems to require? Where can I read more about the rationale behind those changes?

Finally, if we integrate Cosmo support, we’ll likely need a setup to safeguard against regressions—e.g., a GitHub Actions–based CI job would be appreciated.

[ahgamut]

Is Justine Tunney willing to integrate your Cosmo changes as-is?

At the moment, the Cosmo changes are just headers and stubs for well known functions, so I would hope yes. I've submitted a PR here: jart/cosmopolitan#1474

Regarding the LabsJDK changes, I have another concern: they further increase our delta to OpenJDK and therefore increase our maintenance effort. Would you be willing to minimize these changes as much as possible?

The labs JDK changes as far as I remember are mostly just ifdefs to account for Cosmopolitan Libc. The only major change is with respect to the r28 register on aarch64, which brings us to

Also, could you elaborate on the need for the AArch64-specific register changes that Cosmo seems to require? Where can I read more about the rationale behind those changes?

on aarch64, native-image uses r28 to note the current isolate/thread. However, Cosmopolitan Libc currently uses r28 to remember thread-local storage, so there was a clash. I just chose to use r25 because it is also callee-saved.

Finally, if we integrate Cosmo support, we’ll likely need a setup to safeguard against regressions—e.g., a GitHub Actions–based CI job would be appreciated.

We have example CI actions here. I think in terms of CI we'd just need an action that creates static-libs for cosmo libc, like labs JDK currently provides for musl, and then a simplified version of this build script.

[olpaw]

However, Cosmopolitan Libc currently uses r28 to remember thread-local storage, so there was a clash. I just chose to use r25 because it is also callee-saved.

Sounds like we could possibly make this the default for all aarch64 images (not just cosmo). @lewurm do you agree?

[lewurm]

However, Cosmopolitan Libc currently uses r28 to remember thread-local storage, so there was a clash. I just chose to use r25 because it is also callee-saved.

Sounds like we could possibly make this the default for all aarch64 images (not just cosmo). @lewurm do you agree?

We have a tendency to choose the same register assignment as HotSpot to ease sharing of e.g. LIR instructions. After a quick look I don't see an unclean usages of the thread registers however, so it should be fine (and if not, it's certainly fixable).

The labs JDK changes as far as I remember are mostly just ifdefs to account for Cosmopolitan Libc. The only major change is with respect to the r28 register on aarch64, which brings us to

I'm confused about that (context elide-dev/labs-openjdk@7042bab ): Do you need to change it so that you can build libjvm with the cosmo toolchain? We still need a working labsjdk build, so I'm afraid to make that actually work on any AArch64 platform that will require some more work (we can discuss this on a labs JDK PR however; can you create one?).

[olpaw]

Thanks for graalvm/labs-openjdk#28

I left my comments: graalvm/labs-openjdk#28 (comment)

[olpaw]

@ahgamut regarding

1. build the cosmocc compiler suite with this patch of Cosmopolitan Libc

I was able to do that. But how to proceed from here?

I see that now the o folder got created and a lot of tests ran. But how do I use the result of this step to complete the next step:

2. run make static-libs-image using this patch of labs JDK

I need more detailed instructions than what you provide in top comment #12865 (comment).

• Please provide precise instructions for all the steps you mentioned above.

[olpaw]

At the moment, the Cosmo changes are just headers and stubs for well known functions, so I would hope yes. I've submitted a PR here: jart/cosmopolitan#1474

Sent a ping on the Cosmo PR jart/cosmopolitan#1474 (comment)

[olpaw]

2. run make static-libs-image using this patch of labs JDK

Using

export CC=/path/to/cosmo-repo/tool/cosmocc/bin/x86_64-unknown-cosmo-cc
export CXX=/path/to/cosmo-repo/tool/cosmocc/bin/x86_64-unknown-cosmo-c++

and

bash configure --without-{alsa,cups,fontconfig,x} --with-freetype=bundled

gives me

...
checking for jni library path... default
configure: Using default toolchain gcc (GNU Compiler Collection)
configure: Will use user supplied compiler CC=/home/pwoegere/OLabs/issues/GR-73086/cosmopolitan/tool/cosmocc/bin/x86_64-unknown-cosmo-cc
checking resolved symbolic links for CC... /home/pwoegere/OLabs/issues/GR-73086/cosmopolitan/tool/cosmocc/bin/cosmocross
configure: Using gcc C compiler version 14.1.0 [x86_64-unknown-cosmo-cc (GCC) 14.1.0]
checking whether the C compiler works... no
configure: error: in '/home/pwoegere/OLabs/issues/GR-73086/labs-openjdk':
configure: error: C compiler cannot create executables
See 'config.log' for more details
configure exiting with result code 77

In other words, how does this work in detail? See #12865 (comment)

[olpaw]

See comments

[ahgamut]

Updated the commits to avoid merge conflicts.

When opening this PR, I provided a repo containing all scripts and Github Actions used to build example binaries (the Envmap example, and the google-java-format JAR) via this patch.

Both example binaries appear to run without error even after the latest rebase here and here. I will re-run the actions in a while to confirm.

[ahgamut]

@olpaw I have described more details of the build steps below.

---

1. build the cosmocc compiler suite with this patch of Cosmopolitan Libc

this comment mentions cosmocc can be built using the above branch.

However, I found building the labs JDK required some additional headers and static libraries (at least libz, libpng, libjpeg, and libfreetype) that are not provided alongside cosmocc by default. I build these libraries using my superconfigure repo, which provides scaffolding for autotools-style builds.

Similar to the build script attempted here, it may be possible to build the static libs without having to rely on the superconfigure repo, by using the bundled option. I have not tested that build process yet.

Relevant scripts: patch-cosmo-repo.bash, build-cosmocc.bash, and build-deps-labs.bash.

2. run make static-libs-image using this patch of labs JDK

After collecting cosmocc and headers/libraries for (libz, libjpeg, libpng, freetype), I attempt to build the static-libs-image target with the labs JDK, using the patch under discussion at graalvm/labs-openjdk#28. The goal is to have static libs similar to what is currently provided for musl in the labs JDK Releases page.

I provide some additional arguments at the configuration stage (notably --disable-precompiled-headers and --enable-dtrace=no) to simpify the build. After calling make static-libs-image I just collect the relevant archives from the build (along with libz.a from earlier) for use when building with native-image.

Relevant scripts: build-labs-jdk.bash.

Note

A detail I failed to mention earlier that is provided in the repo: static libraries within native-image (liblibchelper.a, the libjvm.a stub, and libsvm_container.a) also need to be built with cosmopolitan libc. I assume it is possible to build these via a specific target in the mx tool (as is done for glibc and musl), but for now I just use the --alt-cc option with mx.

Relevant scripts: build-graal-helpers.bash

3. build native-image as usual with this patch on the host system

If I understand correctly, this step currently does not require cosmocc or any of the previous build artifacts. When writing the patch I used mx --env ni-ce build on my host system so I could have the diagnostic agent for testing.

Relevant scripts: build-native-image.bash.

4. use the built native-image with --libc=cosmo and provide necessary flags to build binaries

Once native-image has been built with this patch, and cosmocc along with the other build artifacts have been collected, we can use all of them together to build native-image binaries with Cosmopolitan Libc.

For now, the liblibchelper.a, the libjvm.a stub, and libsvm_container.a built using cosmocc from earlier need to be copied to a specific folder, namely lib/svm/clibraries/linux-$ARCH/cosmo before running a build. Once those files are in the correct spot, we can build static binaries via native-image --libc=cosmo and required flags.

Relevant scripts: build-envmap-example.bash, build-google-java-format.bash.

[olpaw]

Thanks for the updates. I will look into it in the next days (hopefully).

[olpaw]

Thanks for the info in #12865 (comment). I'm now able to successfully build the static libs with cosmocc. Next step, using these artifacts for review and test this PR.