Color Management of Gaussian Splats

There is increasing interest in how to insert Gaussian Splats (3DGS) into VFX pipelines. Currently, most Gaussian Splats are calculated directly on camera data in display-referred non-linear color spaces. Since most VFX work (e.g., CG rendering and compositing) should be done in a scene-referred linear color space, this poses a color management problem. Unfortunately, it is not as simple as just converting the values from the individual points/ellipsoids in the 3DGS file into a scene-linear color space. This is because the commonly used software for fitting Gaussian Splats apparently uses a rendering model that works directly on the display-referred color values. Applying that model with the fitted parameter weights but using colors that have been converted to a scene-linear color space would no longer give a visual match to the training data.

Work has been done in the area of Neural Radiance Fields (Nerfs) to update the loss function to allow training with a rendering model that uses linear values and it is anticipated that similar work will eventually find its way into commonly used Gaussian Splat fitting software. In the meantime, however, using existing Gaussian Splats in VFX is not straight-forward. This graphic illustrates various approaches to the problem. It assumes that the 3DGS file is rendered separately from a more traditional CG scene it is intended to be used in and then the two are composited together.