[NativeAOT-LLVM] 'wasi' Linker options

[cyraid]

I'd like to include my own --max-memory and --initial-memory, could you change:

<IlcWasmGlobalBase>1024</IlcWasmGlobalBase>
<!-- ... -->
<CustomLinkerArg Include="-Wl,--max-memory=2147483648" />
<CustomLinkerArg Include="-Wl,--global-base=$(IlcWasmGlobalBase)" />

to

<IlcWasmGlobalBase Condition="'$(IlcWasmGlobalBase)' == ''">1024</IlcWasmGlobalBase>
<!-- ... -->
<CustomLinkerArg Include="-Wl,--initial-memory=$(EmccInitialHeapSize)" Condition="'(EmccInitialHeapSize)' != ''" />
<CustomLinkerArg Include="-Wl,--max-memory=$(EmccMaximumHeapSize)" Condition="'(EmccMaximumHeapSize)' != ''" />
<CustomLinkerArg Include="-Wl,--max-memory=2147483648" Condition="'(EmccMaximumHeapSize)' == ''" />
<CustomLinkerArg Include="-Wl,--global-base=$(IlcWasmGlobalBase)" Condition="'$(IlcWasmGlobalBase)' != ''" />

The EmccInitialHeapSize/EmccMaximumHeapSize are used in the browser-wasm for initial/max size, why not have them? So that we can include things like --relocatable (if it came to that, I know there's other things that need to be changed).

Can you also make the linking another target, so we can bootstrap more of the things in LinkNativeLlvm in the future if we need to?

Edit: Ultimately it'd be nice to be able to link a wasm created with this with another wasm with the linker. But it complains about not being relocatable.

[SingleAccretion]

wasm-ld accepts overrides for its options - the last one wins. E. g.:

& $env:WASI_SDK_PATH/bin/clang --% main.cpp -o main.wasm -Wl,--max-memory=2147483648 -Wl,--max-memory=131072

will link with --max-memory=131072. You can utilize this with:

<LinkerArg Include="-Wl,<any option you'd like to override>" />

That said, we don't support overrides for all options. In particular, overriding --global-base is UB.

The EmccInitialHeapSize/EmccMaximumHeapSize are used in the browser-wasm for initial/max size, why not have them?

Yes, that would make sense. There are a lot more options we could add from https://github.com/dotnet/runtime/blob/91f3eb9676d5362b1671d01c2fa23a257d57034e/src/mono/wasm/build/WasmApp.Common.targets as well. Ideally, they would be prefixed Wasm instead of Emcc for the ones that aren't emcc-specific, so that'd need some coordination with upstream.

Can you also make the linking another target, so we can bootstrap more of the things in LinkNativeLlvm in the future if we need to?

I am not sure what you mean here... LinkNativeLlvm is an internal target (the public target is LinkNative).

[cyraid]

Hey @SingleAccretion thanks for the reply,

    <Exec Command="$(WasmLinkerPath) @$(NativeIntermediateOutputPath)link.rsp $(EmccExtraArgs)" EnvironmentVariables="@(EmscriptenEnvVars)" />

That gets executed in the same 'breath' as the other property setting.

The --relocatable doesn't work if --global-base is on the same arguments list. Which is why I was asking for a conditional there. Ultimately, trying to have a 'runtime.wasm' that gets linked to another wasm. Tried wasm-merge with some mixed results, which is why I was hoping for a solution with linking.

[SingleAccretion]

I think what you're asking for is to make <NativeLib>static</NativeLib> work. We can certainly do that. Currently you need 'our' targets to be the root of linking.

Side note about --relocatable: it is not a fully reliable solution to 'packaging' multiple static libraries together. I think it will happen to work with what we currently produce, but it can be broken at any point if we for some reason need to use some more complex features of the linking format that depend on object file identity (weak symbols). The most reliable way is to copy 'our' command line as-is (I realize it is not the most ergonomic way, but AFAIK there is no way to make an archive that links identically to using raw object files, the best you can do is package all the archives together, excluding the object files).

[cyraid]

I'd actually be just as happy if the import memory (--import-memory) worked. Then I could have two wasms running (the runtime and the one that gets provided) and pass the "runtime" memory to the one that's provided. (env::memory)

[SingleAccretion]

making static work

This will be fixed by #3195.

[cyraid]

Thanks @SingleAccretion .. Is there any way to make --import-memory work? The goal is to use the Runtime.wasm separately, but have the provided wasm file (also built with NativeAOT LLVM) with an imported memory? Keeps complaining failed to satisfy "env::memory".

[SingleAccretion]

@cyraid I think it is going to be much more difficult than making static linking work, for two reasons:

1. We're producing components. Components can't import/export memories, though core modules 'inside' of them can. But we can't produce a core module that you could then compose separately (native toolchain problem).
2. How are you going to keep the memories / function pointer spaces separate? --import-memory only makes memory an import, it doesn't change the fact wasm-ld still lays out the memory as-if it is all owned by the module. To overcome that you'd need -fPIC/-pie, but we don't support that.

[cyraid]

Up until now I've been 'unbundle'-ing the modules, because barely any runtimes support components. Wasmtime does, but the dotnet embed version looked like it was being worked on, then someone either paused, is taking awhile, or gave up?

Component model also doesn't work in the browser. If only there was a way to toggle component support off until more runtimes support it. This is the reason why Extism for C# stays at .net 8.0.

[SingleAccretion]

If only there was a way to toggle component support off until more runtimes support it. This is the reason why Extism for C# stays at .net 8.0.

Agree, it has been an issue with components. This all comes down to the issue of "work", and work that is upstream of NAOT-LLVM, like https://github.com/WebAssembly/component-model/pull/378/files. We have also discussed workarounds in the past, but they're not particularly pretty (like building a wasip2 -> wasip1 'adapter').

(My understanding is that wasmtime-dotnet component integration is also a matter of "someone doing it".)

[cyraid]

Sorry for the late response. Someone already started work on it, but haven't heard from them in awhile. I just don't have the time to do it right now myself.

I wanted to debug what was happening to the linker when one of my wit files was not right, and trying to pass -v to clang $(WasmLinkerPath) was quite a feat in itself!

I ended up having to do the hacky:

  <Target Name="AddCLangArgs" BeforeTargets="LinkNativeLlvm" AfterTargets="CompileWasmObjects">
    <PropertyGroup>
      <WasmLinkerPath>$(WasmLinkerPath) -v</WasmLinkerPath>
    </PropertyGroup>
  </Target>

Could you please add a property that allows us to pass clang args? (like -v)?

[SingleAccretion]

Could you please add a property that allows us to pass clang args? (like -v)?

We already have the LinkerArg item group for this. If you need an MSBuild property specifically, we don't have an official one, but it can certainly be added.

[cyraid]

@SingleAccretion I think you're misunderstanding. I'm not wanting a LinkerArg (which I've tried before) to be passed to the linker, I want it for clang directly (catches wit errors, when the errors say use '-v' for clang).

Like I said, my workaround above actually sends -v to clang, and works.

[SingleAccretion]

As you can see in our targets, "the linker" is also just clang, we don't invoke wasm-component-ld directly:

runtimelab/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets

Line 442 in f9601de

<WasmLinkerPath>&quot;$(WASI_SDK_PATH)/bin/clang&quot;</WasmLinkerPath>

So I am pretty sure that your workaround is equivalent to <LinkerArg Include="-v" />. Just to be on the safe side, you do have the terminal logger disabled (/tl:off), right?

[cyraid]

I realize, I've been in that file many times, and I've spent a lot of time with this haha .. But when the flag is in link.rsp, it will not be a flag for clang itself, only for the linker (based on testing). Literally the only thing that worked (even tried environment vars for cflags), was just manually doing my work around to pass -v to clang itself BEFORE the link.rsp ..

Just to be on the safe side, you do have the terminal logger disabled (/tl:off), right?

And yep, I use with -tl:off