Add `:module-maps` to link against c libs without compiler knowledge

[dunkelstern]

To link against a c Library currently you'll have to add a compiler flag (-fmodule-map-file=) to the list.

This way a User is expected to know how the compiler works under the covers, I think this is a bad thing. You should be able to configure what you want if you are knowing what you're doing, but it should be as easy as possible to do the basic things.

The argument that you'll need to know how a module map works could be mitigiated by providing a repository of common module maps to be added as a dependency with atpm. The repo would contain a nop atbuild file and just a lot of module maps that could be used in your atbuild file then.

I will start such a repository if everyone will confirm that this is a good idea.

I propose a new key in atbuild's atllbuild tool: :module-maps which would be an array and just convert the paths in that array to -fmodule-map-file=<path> parameters for the compiler.

[drewcrawford]

I wonder if the interface we need instead is overlays rather than module maps per se.

e.g. for a long time (and still?) Dispatch needs -Xcc -fblocks in order to import it successfully. But there is no way to specify that in a module map.

However, the following design is powerful enough to specify it:

AT/randomlibs/build.atpkg

(package
:name "randomlibs"
:overlays {
    :dispatch {
         :compile-options ["-Xcc","-fblocks"]
    }
}
)

MyProject/build.atpkg:

(package
:name "MyProject"
:import ["randomlibs/build.atpkg"]
:tasks {
    :build {
        :tool "atllbuild"
        :sources ["src/**.swift"]
        ;;etc
        :use-overlays ["randomlibs.dispatch"]
    }
}

I wonder if something along those lines would be superior. Of course that design could use module maps as the source of its link directive, or emit arbitrary compile options as desired, but in any case it's a superset.

It's also worth pointing out -fmodule-map-file= might work differently than you expect:

Load the given module map file if a header from its directory or one of its subdirectories is loaded.

There is a way to pull off the behavior you want if we're satisfied module maps are the correct design. It is a little more complicated than just emitting one flag.

[dunkelstern]

So this means it only links against the file and imports that module if it is actually used. Sounds sane to me. You cannot re-export the API from a module map as I read from that.

But the overlay method sounds good too, so we would have a repo (or multiple sorted after what the libs do? graphics, compression, string manipulation, encryption, etc?) with a big atbuild file with probably thousands of overlays to choose from?

[drewcrawford]

it only links against the file and imports that module if it is actually used

meanwhile in the documentation

if a header from its directory or one of its subdirectories is loaded.

Saying import Foo in a Swift program does not cause any header to be loaded. If it did, then we would not need this feature, as we could just import C libraries directly without any new build options or keys.

There are several ways to get module maps useable from Swift, but none of them are -fmodule-map-file=, the header-based Clang flag. It only works if you have a way to import headers already, which is the thing we are trying to implement.

with a big atbuild file with probably thousands of overlays to choose from?

I would advocate smaller atbuild files that each list the overlay for a single library, but yes.

[dunkelstern]

Interesting, so why does my example in https://bugs.swift.org/browse/SR-685 work then?

[drewcrawford]

I don't know the answer to your question.

I do know that we should now support modulemap load via atbin (specify link-with-atbin). Does that solve this issue, or is there an argument for a direct use of modulemaps that have not been atbinned?