How to avoid edge color bleeding when rendering on transparent background

[stukowski]

I wonder if there is a way to prevent the background color from bleeding into edge pixels when rendering objects on a fully transparent background (alpha=0). This happens when you calculate more than one sample per pixel for anti-aliasing. The following images show the effect, once for a black but transparent background (background=(0,0,0,0)) and once for a white transparent one (background=(1,1,1,0)).

To demonstrate the issue, I read the rendered RGBA image from the ANARI framebuffer to compose it on top of a checkerboard pattern using "source over" blending.

The following reference picture was created using my own OpenGL-based renderer, employing a simple supersampling technique to achieve anti-aliasing. It shows how the result should ideally look.

I would like to allow users of my application to render transparent graphics that are saved as a PNG file, e.g. to insert them later into a PowerPoint presentation. At the time of rendering, the user may not yet know whether the graphic will later be placed against a dark or a light background.

A possible workaround could be to render only 1 sample per pixel in VisRTX, but with a higher image resolution (true supersampling). The application can then downscale the final image, similar to what I did with the OpenGL image. However, it would be nicer if VisRTX directly supported a rendering mode that produces a clean image with smooth edges.

[stukowski]

I think I can take this question back, because I've found the solution myself. If I render with background=(0,0,0,0) and then interpret the pixel data in the framebuffer as RGBA values with premultiplied alpha, I obtain correct results.

In terms of QImage pixel formats, which I am using in my application, the values in the ANARI framebuffer must be interpreted as Format_RGBA8888_Premultiplied, not Format_RGBA8888. When saving the image to a PNG file, it gets converted to a non-premultiplied alpha format (see https://www.w3.org/TR/PNG-Rationale.html, section "12.8. Non-premultiplied alpha").

Please close this issue if you think this is the end of the story.

[stukowski]

The edge bleeding problem is unfortunately not fully solved yet. When I activate denoising (right image), the background color creeps back in at the edges. Do you think this can somehow be avoided?

The following pictures have been rendered with background=(0,0,0,0) and then painted onto a solid white background, once without and once with active denoising.

[tarcila]

@stukowski I've been pushing a PR that I think should fix your issue. Any chance you can test and confirm?

[stukowski]

Thanks @tarcila, I will test your PR.

My first attempt failed due to a VisRTX installation issue after switching from an older commit to your more recent version of VisRTX.

cmake --build . --target install installs the following files in the <ANARI>/lib/cmake/VisRTX/ dir:

VisRTXConfig.cmake
VisRTXConfigVersion.cmake
VisRTX_Exports-release.cmake
VisRTX_Exports.cmake

However, VisRTXConfig.cmake contains an incorrect include statement, which points to a non-existing file:

include("${CMAKE_CURRENT_LIST_DIR}/anari_library_visrtx_Exports.cmake")

Which lets my app's find_package(VisRTX 0.12.0 REQUIRED) call fail.

Is this something you can fix, or should I open another GitHub issue for this?

[tarcila]

@stukowski I've pushed a hotfix to next_release for the CMake issue and rebased this PR, so you should be able to update and test it directly.

@jeffamstutz Keep me posted if there are other/better ways to address that VisRTXConfig.cmake issue.

[stukowski]

Thanks. I've checked out the your PR and compared it with commit 1f69142, which is the older VisRTX version we are currently using in our app.

The only difference I can see is that it no longer plays any role whether I set the renderer's background parameter to (0,0,0,0) or (1,1,1,0). For your new version, the resulting image seems to be the same in wither case, matching the image with background set to (0,0,0,0) from earlier VisRTX versions.

I am attaching two pairs of images to demonstrate this observation. All images show a test scene with purely white objects (denoise=off, depth-of-field=on) and have been produced by saving the contents of the ANARI frame buffer directly to PNG files using the Qt framework as follows:

const unsigned char* data = frame.map<unsigned char>(AnariRenderBuffer::Channel::Color);
QImage image(data, width, height, QImage::Format_RGBA8888);
image.mirrored().save("image.png");

The format QImage::Format_RGBA8888 tells Qt to interpret the pixel data as "straight" alpha (not premultiplied), which is the format used by VisRTX, I assume. Since the PNG file format also stores straight alpha colors, no transformation should take place and the RGBA values from the ANARI frame buffer are saved as is.

I am particularly interested in how the resulting images look when composited over a light background, because this has been the problematic case so far. To see the outcome, simply click on these PNG images to view them in your web browser, which (probably) renders them over a white background.

This first pair of images has been produced with VisRTX commit 1f69142, with background set to (0,0,0,0) and (1,1,1,0):

The left image yields perfect results when composited onto a black background, the right when composited onto a white background (but not vice versa).

When I do the same experiment with your new VisRTX version, I get two identical images:

They only look good on a black background.

What I would need, however, is an RGBA image that looks good on both dark and light backgrounds simultaneously.

[tarcila]

@stukowski I've been pushing an update to the related PR, where the output image is compensated for alpha premultiplication. This should improve the situation.
Here are pictures of before and after the update, with alpha (blending with the background color of the browser):

Blended onto a white background:

[stukowski]

Thanks. To successfully build your code, I had to replace the type uint with uint32_t on Windows.

After the new un-premultiply step, the result looks slightly better (a lighter halo around the white objects), but it's not perfect. I don't think this actually solves the problem.

I analyzed the functions __raygen__() and accumResults(), and I'm quite sure the issue is in accumResults(). This function blends the (premultiplied alpha) colors from multiple pixel samples and produces the final output color (supposedly also premultiplied, now that you have added the subsequent un-premultiply step).

I have the feeling that the problem could be caused by accumResults() calculating the arithmetic mean of the RGBA samples after tone mapping. This results in incorrect RGB components (alpha remains correct).

To show this, let's take a simple calculation example: Suppose N pixel samples get mixed. A fraction x (0<x<1) of the N samples hit an opaque object with shading color (1,1,1,1). A fraction (1-x) of the samples miss the object and yield the background color (0,0,0,0). The expected output for this case is exactly rgba=(x,x,x,x), i.e., white with opacity x (premultiplied).

Without tone mapping, accumResults() would exactly yield this expected result. However, the tonemap function turns the input color (1,1,1,1) into (1/2, 1/2, 1/2, 1). After arithmetic mixing with the zero background, this gives (x/2, x/2, x/2, x). Finally, inverseTonemap is applied, which yields (x/(2-x), x/(2-x), x/(2-x), x). This is different from (x,x,x,x).

I am wondering if this issue could be solved simply by omitting the tonemap/inverseTonemap steps. What do you think?

[stukowski]

And regarding the un-premultiply step that you've added: This processing step, which incurs additional computational costs, is not needed if I adjust all subsequent steps in our app to the fact that VisRTX outputs premultiplied alpha in the ANARI frame buffer. For example, I can simply wrap the data in a QImage with flag Format_RGBA8888_Premultiplied instead of Format_RGBA8888 and Qt will automatically interpret the data in the correct way in the subsequent processing steps.

[tarcila]

@stukowski Thanks for the quick turnaround and the additional feedback.

Regarding tone mapping, this component is necessary for properly handling HDR. I still have a task on my list to explore ways to mitigate some of its side effects. As for the "non-reversible" behavior you observed, that’s actually expected (though please don’t quote me on the term — the max3 tone mapper used here is considered a reversible tone mapper).

I’m in the process of cleaning up our renderers to bring the DirectLight (default) renderer back to a more SciVis-oriented target, where speed is more critical then realism. More advanced rendering features would be offloaded to other renderers. In most cases, this could help eliminating the need for tone mapping in the default renderer.

Concerning premultiplication, the desired behavior largely depends on how the final frame is intended to be consumed. Whether this is up to VisRTX or the client code to do the work is something to figure out. As we discussed, ANARI is intentionally vague about color semantics in this context. For now, I’ll revert to using premultiplied output until I have a clearer understanding of how RGBA should be treated in ANARI or determine whether this should become a configurable behaviour in VisRTX.

You also mentioned performance — did you observe any drop? I ran a quick test before submitting the PR and didn’t notice any meaningful differences.

As for this bug, here’s what I propose:

• Update the in progress PR to only include the premultiplied background fix. As you mentioned, Qt can handle premultiplied alpha, which should be better from a post-processing perspective anyway.
• Create a new PR introducing a new renderer parameter to disable tone mapping, so you can optionally turn it off.