Should we do selective export for libHalide?

[alexreinking]

(FYI, there is a history here; we used to do selective export, but it proved hard to maintain, for various reasons, and we ended up abandoning it. Opening a new issue for future discussion would be worthwhile.)

Originally posted by @steven-johnson in #4644

---

CMake provides a GenerateExportHeader module that makes it somewhat easier to selectively export library symbols in a cross-compiler-compatible way. That, along with testing discipline to cover all the symbols we want to export, should ease the maintenance burden.

(Please feel free to edit the following two lists)

The pros of doing this include:

• Smaller binary sizes
  • Think of all the inlined template instantiations!
• Explicit API surface, which would make it easier to assign semantic version numbers, rather than dates.
  • If we remove an export declaration, bump the major version; if we add one, bump the minor; if we want to release changes without touching any of these, bump the patch.

Cons:

• Potentially difficult to implement in the Makefile, though it only supports POSIX OSes and GCC-compatible compilers.
• Maintenance was historically a burden. Why?

[abadams]

Halide is designed to be extensible (e.g. Func::add_custom_lowering_pass). People extending Halide may want to reuse all sorts internal tools from outside the compiler. It's also nice to be able to test internal components from an external test, and for external plugins like the autoscheduler to have access to all the internal tools. All this made explicitly exporting symbols a game of whack-a-mole where we were slowly creeping towards exporting everything for one reason or another. It also required littering the source with HALIDE_EXPORT macros. I believe the binary size savings were found to be insignificant (IIRC), largely because we were approaching manually exporting everything.

Regarding versioning - the intent is that namespace Halide is our public API, and changing something there must be done with care because it might break users. But for Halide::Internal we promise nothing (and people using those symbols just have to deal with changes).

So because the binary size difference was close to zero, and we already have an explicit API surface via namespaces, I claim that there are no real pros. Just exporting everything eliminates a whole class of failures at minimal cost.

[alexreinking]

Halide is designed to be extensible (e.g. Func::add_custom_lowering_pass). People extending Halide may want to reuse all sorts internal tools from outside the compiler. It's also nice to be able to test internal components from an external test, and for external plugins like the autoscheduler to have access to all the internal tools.

Between this and:

But for Halide::Internal we promise nothing (and people using those symbols just have to deal with changes).

Why would anyone want to depend on Halide::Internal? We might decide to, for example, stop exporting any symbols in that namespace. If we want to weigh those users' needs in our decisions about Halide::Internal, then we don't actually have an internal/external distinction.

But then everything else (tests, autoschedulers) is first-party and we can adjust our build processes accordingly to either give those parts access to the internal symbols, or promote relevant symbols to the public API.

I believe the binary size savings were found to be insignificant (IIRC), largely because we were approaching manually exporting everything.

Did we try applying the relevant linker flags (eg. -Wl,--exclude-libs,ALL) to LLVM, too? I'm surprised that dropping the over 120k symbols between LLVM and Halide::Internal had no impact on binary size or loading times (which would be observable in the tests)

[abadams]

Not sure what flags we tried, you may want to experiment. For llvm we already dead-strip all symbols not used by Halide itself in the makefile build via linker shenanigans.

We care about external users of both Halide:: and Halide::Internal::, but there are different expectations around each. Users of Halide::Internal:: should expect to have to tweak their code every time they update Halide (in the same way we have to tweak our code every time we update LLVM). Users of Halide:: expect more stability than that (e.g. there are deprecation warnings for a while before we delete things).

[alexreinking]

Not sure what flags we tried, you may want to experiment. For llvm we already dead-strip all symbols not used by Halide itself in the makefile build via linker shenanigans.

It is probably the right move, then, to do the experiment, and see if there are actual gains to be had. Those can be weighed against the ongoing maintenance burden of marking down exports.

[abadams]

The problem is to do the experiment you'd need to mark all the exports, which is more than just everything in Halide::, because of the extensibility stuff I mentioned. I think everything mentioned in Halide.h is the right set of symbols, as those are the things we've decided should have declarations in user code. Unfortunately that's nearly everything. We just exclude LLVM-using stuff and some built-in tables I think.

One thing you could do is roll back the repo to the point where we deleted all the export macros and check the size and loading time of libHalide before and after that commit. That should be a reasonable proxy for what it would look like to reintroduce the macros.

[alexreinking]

I've started running into issues on Windows with CMake 3.19 regarding exported symbols not being found. This might have to do with WINDOWS_EXPORT_ALL_SYMBOLS flakiness; remember: this is a CMake/Kitware-maintained convenience feature, not just another flag to pass to a Microsoft tool.

Still investigating. But if this is the problem, then selective export might be worth our while.