Addressing OpenEXRCore Symbol Conflicts When Using Multiple OpenEXR Versions in the same app

[vlazar-ilm]

Hello,

I've identified a problem when trying to use multiple versions of OpenEXR side-by-side in the same application.

The Problem

While the OpenEXR C++ API successfully avoids conflicts using its custom namespace feature, this protection does not extend to the C-based OpenEXRCore library.

When using dynamic libraries, the OpenEXR C++ library depends on OpenEXRCore. The dynamic linker resolves symbols for OpenEXRCore only once. This means that if multiple versions of OpenEXR are loaded, they will all incorrectly link against the first OpenEXRCore library that was loaded. This "symbol cross-talk" can lead to crashes or undefined behavior when a C++ library calls a C function from the wrong version of Core.

Proposed Solutions

To properly support side-by-side versioning, the C symbols in OpenEXRCore must also be uniquely identified. I thought about two potential solutions for this:

1. Opt-In C Symbol Prefixing

   • This approach would mirror the C++ custom namespace feature.
   • This would be an opt-in feature, allowing developers to ensure their custom builds do not conflict with other versions, while official builds will remain compatible with public builds
   • A new compile-time prefix would be added to all public facing C symbols.
   • Example: exr_encoding_run would become INTERNAL_EXR1_exr_encoding_run.
   • Downside is that it's a fairly intrusive change
   • Would work on all platforms

2. Symbol Versioning (Linux)

   • On Linux, we could use symbol versioning, similar to how glibc manages compatibility.
   • The official OpenEXR build could use a stable version (e.g., OPENEXR_1.0), and custom builds could use their own unique version string.
   • This would create a "hard link" binding each OpenEXR C++ library to its specific OpenEXRCore version.
   • Caveat: I am unsure of the intricacies or equivalents of this method on Windows and macOS dynamic linkers.

Any fresh insight on this issue is appreciated.

[RichardAtWeta]

I would suggest considering another option:

3. Wrap OpenEXRCore in a namespace

In this case, I mean wrapping the declarations and implementation of OpenEXRCore with a C++ namespace. Other than that, it would remain as C and use no other C++ features.

When compiled as C, there would be no namespace.

When compiled as C++, it would use a namespace (which could be switched for customized purposes).

There are more details and options to consider, but I wanted to make the suggestion.

For what it is worth, it probably is useful to also have symbol versioning for platforms that support it. In which case, I would suggest that symbol versions are always used on the platforms that support it since that helps the cases where you want to deliberately isolate a separate implementation in the same process.

[cary-ilm]

@RichardAtWeta, just to confirm, we would add a new library, OpenEXRCoreC++, which consists of namespaced C++ wrappers around each of the exported functions in OpenEXRCore, then statically link OpenEXRCoreC++ against libOpenEXRCore with symbols hidden. So like this:

namespace OPENEXR_3_4 {

exr_result_t 
exr_add_channel(
    exr_context_t              ctxt,
    int                        part_index,
    const char*                name,
    exr_pixel_type_t           ptype,
    exr_perceptual_treatment_t percept,
    int32_t                    xsamp,
    int32_t                    ysamp)
{
    return ::exr_add_channel(ctxt, part_index, name, ptype, percept, xsamp, ysamp);
}

}

for all the exr_* Core functions.

Is this what you're thinking? Or something different?

[RichardAtWeta]

It could follow the patterns for doing namespace control that the C++ portion of OpenEXR already uses.

[peterhillman]

Instead of the wrapper to hide the samples, the Core library itself could become a C++ library when custom namespaces are enabled. There could be OPENEXRCORE_NAMESPACE_ENTER and OPENEXRCORE_NAMESPACE_EXIT macros in each existing header. By default there's no change: the macros expand to nothing and Core compiles as a C library called libOpenEXRCore. If you want a custom namespace (e.g. Foo), it is instead compiled with C++ as a library called libOpenExrCoreFoo, and the macros expand to wrap each header in a namespace Foo {}. It probably should have a using namespace Foo somewhere too, to keep API compatibility.

The custom namespace library cannot be used from pure C code, but should "just work" from C++ code without any modifications. The namespace should be invisible except in the symbols themselves.

It seems like it would be a little fiddly to set up the cmake to provide the optional namespace, since it has to compile the library differently, but it keeps the header files easy to understand, and should work without changing libOpenEXR or any other "customers" of the Core library.

[cary-ilm]

I like that idea! In Vlad's situation, Nuke would be built against the vanilla OpenEXR/OpenEXRCore with symbols as they currently are. But for Vlad's plugin, he would build a custom OpenEXR/OpenEXRCore with -DOPENEXR_CORE_NAMESPACE=ON, which would compile OpenEXRCore as C++ in the same namespace as the OpenEXR library, which would result in different symbols from the ones in Nuke.

You suggested building it as libOpenEXRCoreFoo, but is that necessary? Since libOpenEXRCore shared library has the namespace suffix, e.g libOpenEXRCore-3_4.so, wouldn't the version/namespace suffix be sufficient?

[peterhillman]

If you build core with a custom namespace I think it might be wise to have that in the name of .so that's linked against. That would be helpful at link time to make sure the correct library is picked up (so the link command would specify -lOpenEXRCoreFoo). It is conceivable that software directly links against the customized library library, but also links against something that itself uses the standard OpenEXR library, in which case the link line would contain both -lOpenEXRCore and -lOpenEXRCoreFoo.

But yes, it's a good point that it should retain the same versioning as well , so it would be libOpenEXRCoreFoo-3_4.so.99

Looking at it closer, the headers are already wrapped in extern "C" {} so that would change to namespace Foo { when building with a custom namespace, alongside an #ifndef __cplusplus check that gives a useful error message when attempting to use a C compiler to build code that pulls in headers with a custom namespace.

[cary-ilm]

I had the mistaken assumption that the custom namespace was already appended to the output .so name, but it's just the hardcoded version that gets appended. But there is a OPENEXR_LIB_SUFFIX configuration variable, so you could get the Foo tacked on via that.

I just tried compiling the .c files in the core library with the C++ compiler and it fails with a bunch of missing type casts, so there's some repair work to do independent of the new statements to add.

I do like this idea, because it basically means the the Core library is still available in environments that require C and are OK with the accompanying limitations (no namespacing), but in environments in which the extra features of C++ (namespacing) are beneficial, with the flip of a switch you get the exact same behavior as the OpenEXR library, with minimal excess confusion.

[RichardAtWeta]

It probably makes sense to run all the tests against the core library in its C++-built form.

There are subtle differences in what is allowed in C as compared to what is allowed in C++.

[vlazar-ilm]

The namespace idea is great: not terribly intrusive (albeit might need changes to make C C++ compliant)
Would this be a mechanism instead of the symbol versioning or in conjunction with it ?
Is this something that could go inside 3.4 or is more of a 3.5 type of thing ?

[cary-ilm]

As I'm envisioning this, the default build behavior remains exactly as it is, with OpenEXRCore compiled as C, and we can add symbol versioning to that for situations that would benefit. Presumably, package managers and applications like Nuke will continue to build and distribute the library exactly as they have, so this can safely go into the v3.4 release. But we'll add a new cmake configuration option that compiles OpenEXRCore as C++ inside the same namespace as the OpenEXR library, available for the special situation in which plugin writers need to build and integrate an alternate version. If you build a custom OpenEXRCore that way, it won't be ABI compatible with the official v3.4 release, but then that's the point, you want the symbols to be different.

I'm working on implementing this now, I hope to have a solution in the next few days.

[peterhillman]

That's a good approach. Maybe we should go further and strongly discourage distributing libraries with custom namespaces unless there are unavoidable modifications to the library: it's a feature for experimentation or "internal use" only. Applications should distribute official releases built without namespaces, so that end users can replace distributed OpenEXR libraries with patch releases that may address security vulnerabilities (or even add new features) without causing a breakage.

I think if OpenEXRCore is modified and built with a custom namespace, then OpenEXR also needs a custom namespace, even if there are no changes. If only the OpenEXR library has been modified, then it would need a custom namespace but OpenEXRCore wouldn't. That case may be uncommon enough that it isn't worth supporting officially.

@kdt3rd also suggested that even official releases may benefit from symbol versioning, so that new libraries can provide both a new and backwards-compatible version of the same symbol. If that happens, then a non-standard build of OpenEXRCore should have a specific symbol version to indicate it is non-standard. Where a customized OpenEXRCore needs to stay a C library, OpenEXRCore could be built with a custom symbol version but without the namespace, skipping the build of the other libraries, but it might not be worth implementing support for that in the cmake setup either.

[vlazar-ilm]

This is exactly what I was thinking: in a perfect scenario where everybody used versioned symbols, I would skip the custom C namespace only if for the reason that it takes me further away from the beaten path.

[RichardAtWeta]

Indeed, I do recommend always using symbol versions on the platforms that support it.

Using namespace support, as we have been discussing here, also has the great benefit that it is using standard features.

[cary-ilm]

I submitted #2205 as a solution to this issue.