KHR_gaussian_splatting

[weegeekps]

Update: I've split out KHR_gaussian_splatting_spz_2 into it's own PR. There's been many changes since I wrote the summary, and while I'm leaving it at the moment. It will be updated soon.

---

This extension proposal, KHR_spz_gaussian_splats_compression, allows for efficient storage of 3D Gaussian splats data within glTF using the SPZ compression library from Niantic Spatial. The extension is applied to a primitive. The SPZ binary blob is stored as a buffer within the glTF file, and implementations can use the SPZ library to either decompress and then map the compressed Gaussians into placeholder attributes on the primitive or directly decompress into their rendering pipeline if preferred. Content creators have the flexibility to choose to use no Spherical Harmonics, or up to all 3 degrees of spherical harmonics depending on their use case.

We are currently working on an implementation in the CesiumJS engine based on this draft that we hope to have released soon.

[DRx3D]

It appears that this extension is tied to the Niantic Spatial library. Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?

Also note that the license for the Niantic Spatial library is MIT.

[weegeekps]

Is it necessary to specify a version or other unique identifier to ensure that the desired algorithm and calling sequence is used?

Good question. The SPZ library packs along a version number within the binary data that we store in the buffer, so it's unnecessary to have a version number stored in the glTF metadata.

[lexaknyazev]

this is the algorithm for quantizing to signed byte

That algorithm compresses the [-3, 3] range to [-1, 1]. Do renderers need to scale those values back?

[NorbertNopper-Huawei]

this is the algorithm for quantizing to signed byte

That algorithm compresses the [-3, 3] range to [-1, 1]. Do renderers need to scale those values back?

@weegeekps There is no scaling back to [-3, 3] or [-5,5] range for this extensions (SPZ makes some assumptions and this is the reason for this scaling in this extension). In this extension, the values are as is e.g. found in an uncompressed PLY file.
There is maybe the possibility, that the original, uncompressed data is in the range of [-1, 1]. This is today not done, but the normalization can be kept for such future approaches.

[NorbertNopper-Huawei]

From this website:
https://niujinshuchong.github.io/mip-splatting-demo/

The chair.ply asset, the KHR_gaussian_splatting:SH_DEGREE_0_COEF_0 are in the range [-2.525310, 6.694453].
For all the others degrees, the ranges go from [-1.024479, 0.996620].

For the drums the ranges are [-2.654668, 8.410955] and [-1.314519, 0.995610] correspondingly.

Just two random samples and please cross check, maybe with my quick test I did do something wrong.

Also, by keeping the extension as is (maybe with some explanation on the normalized data), this allows better the possibility for other compression methods like SPZ and other upcoming compression methods.

[NorbertNopper-Huawei]

@weegeekps @lexaknyazev
I looked at some code to precompute the color.
In short, I suggest to add an "offset" : 0.5 in the schema.
As long as the color is display referred and sRGB encoded, 0.5 works fine. However, for scene referred, probably this value has to be different. Also, just to be future proof, if the data is PQ or HLG display referred encoded, maybe one wants to use another value.

[NorbertNopper-Huawei]

@weegeekps Same problem in regards of quantization and normalized data with KHR_gaussian_splatting:SCALE

[lexaknyazev]

@NorbertNopper-Huawei SCALE is much less affected because it's log-encoded and supports non-normalized values.

[NorbertNopper-Huawei]

@NorbertNopper-Huawei SCALE is much less affected because it's log-encoded and supports non-normalized values.

We can keep it if folks manage to train such data, however no magic conversion please.

[NorbertNopper-Huawei]

@weegeekps Plus adding offset=0.5

[weegeekps]

Do you have an example of where you expect the 0.5f bias to be used in the renderer? I understand it's importance from the aspect of the training algorithm, but typically the renderer does not need to know this information, just that the color space is sRGB.

[NorbertNopper-Huawei]

EDIT: Did again some more research and had an internal discussion on this, so I keep what is important from the original post.

---

In general, the color is precomputed during load time, so one does not need to convert during runtime (but it is possible) or it is already stored in a specific file format e.g.
https://github.com/kishimisu/Gaussian-Splatting-WebGL/blob/main/src/loader.js#L96
https://github.com/playcanvas/engine/blob/main/src/scene/gsplat/gsplat-data.js#L73
https://github.com/antimatter15/splat/blob/main/convert.py#L32

In the implementation from the original paper this happens here:
https://github.com/graphdeco-inria/gaussian-splatting/blob/main/gaussian_renderer/__init__.py#L80
https://github.com/graphdeco-inria/gaussian-splatting/blob/main/utils/sh_utils.py#L117

During training, this goes forth and back e.g. using:
https://github.com/graphdeco-inria/gaussian-splatting/blob/main/utils/sh_utils.py#L114

---

The resulting colors are in general in the range of [0, 1], this is what the paper expects and is represented in this extension. So, this 0.5 is fixed by the given math and is added after the evaluation of the spherical harmonics. However, if it is not mandatory for 3DGS, a bias of 0.5 as default can be added.

For display referred images, the range of [0, 1] is given by default, independent if currently used sRGB encoding or for future use cases for PQ or HLG encoded images.

For scene referred images, if the linear color is in the [0, 1] range (e.g. content trained from SDR images and linear color space), this also works pretty well.
This fulfills the requirement for linear color space e.g. like here:
https://ubc-vision.github.io/stochasticsplats/

For the future, having other color spaces like Rec.2020, this also works well.

However, if the scene referred data is trained on HDR data, which contains values below 0 and/or larger 1, then it will not work.
However, the common approach is, that the color data is normalized to [0, 1] before training.
E.g. this paper uses RAW image data:
https://github.com/shreyesss/HDRSplat
To normalize the data, e.g. an offset with scale can be used. However, different approaches are available, to bring the data into the [0; 1] range e.g. using PQ for non-linear training.
This normalization and de-normalization and required parameters does belong into the KHR_gaussian_splatting_wide_gamut_color extension and needs to be further discussed there:
#2539

---

In summary:

• No offset and scale or alternatives in this extension needed, as long as the trained image data are in the [0, 1] range.
• As seen in the links above, the fixed 0.5 is given by the implementations of the paper and also depends on the required math. If required, a default bias of 0.5 can be added, however the text needs to be removed and/or rewritten in the README:

Wrong: "Non-normative Note: If the spherical harmonics are in the BT.709 gamut, the diffuse color can be computed from the KHR_gaussian_splatting:SH_DEGREE_0_COEF_0 attribute by multiplying each of the RGB components by the constant spherical harmonic value of 0.282095."

[NorbertNopper-Huawei]

@weegeekps @lexaknyazev ^^^

[lexaknyazev]

Can we guarantee that SH_DEGREE_0_COEF_0 * 0.282095 + 0.5 will always be in the [0, 1] range? If not, maybe the note should also require clamping so that COLOR_0 values are valid as per glTF 2.0 spec.

[NorbertNopper-Huawei]

And the initial color is loaded here:
https://github.com/playcanvas/supersplat/blob/0c0edf4d/src/shaders/splat-shader.ts#L105

[NorbertNopper-Huawei]

So, for the base glTF extension, we have this 0.5, like in the implementation of the paper.

Any additional parametrized offsetting and scaling, we do in the other extension.

[NorbertNopper-Huawei]

computeColorFromSH
https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/forward.cu#L20

Degree 0
https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/forward.cu#L30

Larger than degree 0
https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/forward.cu#L32

Offset of 0.5
https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/forward.cu#L63

[weegeekps]

I took a deeper look at what CesiumJS, BabylonJS, PlayCanvas, and Inria is doing because Inria is clearly are adding a 0.5f bias to the forward rendering pass and I wanted to ensure I fully understand. Ultimately, I think you are right @NorbertNopper-Huawei, but read on as I did a really deep dive into how all of this works. When I went back through everything this evening, as well as some of my notes from well over a year ago, I realized that this is actually a fairly important detail that I had forgotten.

The purpose of this bias within Inria's renderer is to ensure that the resulting color remains between [0, 1] before clamping. It's a numerical stability trick and as far as I can tell the authors mostly arbitrarily chose it because it works well enough.

As a refresher (because apparently I needed one) the way training works is the forward pass renders an image, compares to the ground truth, and then uses the backwards pass to adjust the SH coefficients. When you apply a bias, such as 0.5f as the authors have done, then the learned coefficients become intrinsically linked to the bias. Since a 3DGS renderer is simply the forward pass logic being run in real time, it needs to know the correct value of the bias.

None of the 3 real-time renderers appear to be doing this on the surface, but I did discover:

• PlayCanvas has it's own clrOffset and clrScale that they're setting as you pointed out, but these are related to brightness and black point values set within the renderer. Otherwise, I couldn't find anywhere that the 0.5f value was being set. I suspect it is getting handled elsewhere.
• CesiumJS and BabylonJS implicitly get this 0.5f value through SPZ, which I clearly forgot about. 🤦‍♂️
• After looking deeper into the math, the 0.5f bias is appropriate for all SDR and HDR content. The key difference between SRGB and any HDR content would be that after the bias any data outside of the [0, 1] range will be clamped and truncated for SRGB and with HDR it would not.

Given all of this, we probably do need to add this bias value as a property, but while I'm leaning towards adding it, I admit I'm still not fully committed yet. I want to talk to some others on the training side of things and get their thoughts as well; in the off chance this is a value that nobody will ever change, then maybe it doesn't belong in the base spec. It is a non-negotiable constant that both the trainer and renderer have to agree on, so maybe it does. Not sure, I'll figure it out in the morning.

Also, as an aside, we should not refer to this value as offset. It's a fine detail, but in the context of Gaussian Splatting, something like colorBias or shBias is more correct given it's purpose.

[NorbertNopper-Huawei]

Please re-read, as I learned as well and updated the document ☺️
#2490 (comment)

---

• For the base extension and probably other future extensions, we keep this 0.5 and it is not the offset anymore I mentioned initially
• For SDR(!) based content in scene referred space or display referred content in general, the values are between [0, 1]. So, this base extension works out of the box.
• For other color gamuts, this works as well with the base extension. However, these are handled in a separate extension as discussed
• For HDR content below 0 and/or above 1, the content is normalized before training, bringing into [0, 1] space (this is done in the HDRSPlat paper as far as I understand this). So, nothing needs to be changed in the trainer and/or the renderer for the general case. However afterwards, one could use an offset and scale to do the de-normalizing, but maybe others want to use PQ to do this. Or something different. This needs somehow defined in a separate extension and e.g. what PlayCanvas is doing with the clrOffset and clrScale is what I suggest in a general approach. And I assume as well, somewhere is the 0.5 visible in their code.

---

• What is also important, that SH_DEGREE_0_COEF_0 * 0.282095 can be precomputed, however it is not "yet" the color.
• SH_DEGREE_0_COEF_0 * 0.282095 + 0.5 can also be precomputed for only degree 0. This is the color.
• However, both optimizations are up to other extensions etc. and/or the fallback mentioned in the README.

---

• In regards of the 0.5 bias, the spherical harmonic coefficients are trained around 0 for numeric stability. If you have the degree 0 only case, the values are somehow around [-1.7, 1.7] as mentioned before in this discussion. If you multiply by 0.282095, you come around the [-0.5, 0.5]. Add 0.5, and one is in the color space.
• For higher degrees, the intervals are different. However, if summed up properly and added 0.5 at the end(!), the range is again in [0.0, 1.0]
• So no bias required, it is always 0.5. And if this really needs to be altered in the future, we can create another extension for this.

---

From my perspective, if folks are catching up with our insights, the base extension is ready to go. Everything else can be handled in the other extensions.

[javagl]

Do renderers need to scale those values back?

There is no scaling back [...] In this extension, the values are as is e.g. found in an uncompressed PLY file. There is maybe the possibility, that the original, uncompressed data is in the range of [-1, 1]. This is today not done, but the normalization can be kept for such future approaches.

and

Just two random samples and please cross check

I just added some log output for the SH0/0 "bounds" in my viewer, and dragged-and-dropped the given files into that. The values for those match the values that you posted. I also dropped a few other, random, unspecified PLY files into that.

What may be important: Theye PLY files are from different sources! (Random websearch results...). It might very well be that one particular splat creator/trainer does some form of "normalization" or targets a certain value range. But this wouldn't necessarily mean that this has to be the case.

Some outputs for different files:

// chair.ply:
MinMax SH (-2.2955, -2.4034, -2.5253)-(6.6945, 5.2129, 5.1132)
// drums.ply:
MinMax SH (-2.3606, -2.6547, -2.6196)-(6.8881, 7.318, 8.411)
// Others:
MinMax SH (-1.7725, -1.7725, -1.7725)-(1.7725, 1.7725, 1.7725)
MinMax SH (-1.7725, -1.7724, -1.7725)-(1.7725, 1.7725, 1.7725)
// This one looks odd when rendered, though...
MinMax SH (-18.3192, -16.9435, -20.2332)-(16.7837, 17.2307, 18.0589)
MinMax SH (-2.9584, -2.4775, -2.3873)-(4.0701, 4.1613, 4.7967)
MinMax SH (-3.0152, -2.8437, -2.784)-(7.2432, 6.6272, 6.3273)
MinMax SH (-1.7719, -1.7721, -1.7722)-(1.7723, 1.7723, 1.7723)
MinMax SH (-1.9665, -1.9804, -1.9763)-(2.8436, 2.6257, 2.8463)
MinMax SH (-1.7725, -1.7725, -1.7725)-(1.7725, 1.7725, 1.7725)

It looks like there is a cluster at that that ~1.7 range, but the values may be way larger, and apparently, still make sense. So I assume that the trainer could be configured to not generate this [-1.7,1.7] (or larger) range, but in fact, the [-1,1] range instead, and still represent "the same data".

In this case, the "signed intN/normalized" accessors could be used as a compact (quantized) representation of these values. So it looks like it would make sense to allow this in the base specification - maybe with a hint that ~"some trainers could be configured to generate that kind of data".

[NorbertNopper-Huawei]

@javagl Thanks for cross checking.

So, there is no problem to keep the quantized definition. If the data can be generated, all good. However, no magic numbers for conversions please.

My biggest concern as now is the 0.5 in the color conversion, as for linear data and/or encodings this will not work. So, to be future proof, this 0.5 should be a parameter inside the glTF.

[lexaknyazev]

@NorbertNopper-Huawei Where does the 0.5 offset come from?

[javagl]

I'm not entirely sure about the conversion from/to COLOR_0 either.

In my meticulously tested and mathematically verified library code where I just wrote down what seemed to make some sense at some point in time, there are directCurrentToColor/colorToDirectCurrent conversion functions that map back and forth between that [-1.7,1.7] and [0,1] range. The 0.5 that you are alluding to might seem to be inherent to that. But looking at some of the min/max values from these tests...
(-2.2955, -2.4034, -2.5253)-(6.6945, 5.2129, 5.1132)
this range obviously isn't even close to be symmetric around 0 (so I think that the question about "0.5 or not" doesn't even come up).

My "conversion" would just map these values to [0,1] (clamping them). Out of curiosity, I applied this back-and-forth conversion (with clamping) to the chair.ply. So the following is a comparison of the original
MinMax SH (-2.2955, -2.4034, -2.5253)-(6.6945, 5.2129, 5.1132)
and the clamped
MinMax SH (-1.7725, -1.7725, -1.7725)-(1.7725, 1.7725, 1.7725)
range (for all SH coefficients):

A subtle difference is visible (although not as strong as I suspected).

I think that the question is still the same: What is the value range?

If it was [-1.7,1.7], then the default conversion involving that 0.5 would make sense. If it was [0,3.4] then a different conversion (without some 0.5) would have to be applied. And an "offset=0.5" is not sufficient, because the range might also be [-10,10] or [0,20], for what it's worth...

[NorbertNopper-Huawei]

@NorbertNopper-Huawei Where does the 0.5 offset come from?

color = 0.282094 * coefficient + 0.5

I assume, the original paper used mainly display encoded sRGB images. So, to map the range to RGB, this 0.5 was added:
https://github.com/graphdeco-inria/gaussian-splatting/blob/main/utils/sh_utils.py#L114

However, if linear values are used, which are beyond 1 and/or different encodings are used e.g. HLG or PQ, I assume a different offset makes sense.

You can also cross check code, if PLY data is precomputed to color data.

[javagl]

color = 0.282094 * coefficient + 0.5

As elaboarted above: This assumes a range of [-1.7,1.7], which may not be the truth.

Actually, I'm not sure how important that is. One could just claim that the generator/trainer is responsible for storing values (in [0,1]) in the COLOR_0 in a way that sensibly reflects the data - similar to the requirement that iff the trainer is using signed intN/normalized accessors, the trainer is responsible for training the data in a way that the values fit into the [-1,1] range 🤷‍♂️

[NorbertNopper-Huawei]

the trainer is responsible for training the data in a way that the values fit into the [-1,1] range 🤷‍♂️

Please forget about this.

[javagl]

Please forget about this.

This contradicts your comment

So, there is no problem to keep the quantized definition. If the data can be generated, all good.

But... I'll abstain from further comments here for now.

[NorbertNopper-Huawei]

@javagl Please do not get me wrong. It is not possible to request this from the data. So, sorry, did not want to be impolite.

[weegeekps]

That algorithm compresses the [-3, 3] range to [-1, 1]. Do renderers need to scale those values back?

Yes, and my apologies for glossing over that. I was trying to make a point that there are algorithms that absolutely work with as it is today.

During the normalization step, the denominator is the absolute maximum value of your range of possible numbers. So, if you know your range is going to be $[-3, 3]$ for a particular set of values (like spherical harmonics) you can firmly state that the scale then would be $3$. This would be defined in the compression algorithm you're using and its related specification. There are obvious issues with this approach as it can cause some pretty significant loss of precision.

In practice you shouldn't use a pure SNORM8 quantization algorithm like this, but you can and it will work. Excluding the DC components, it's relatively safe to clamp the outliers when dealing with the spherical harmonics. Doing so will reduce specular effects but if you choose a slightly higher range such as $\pm 5$ the clamping will in practice have little difference.

All of that said, you should generally not use this algorithm. Which leads me to the point that I think I failed catastrophically hard at actually making yesterday:

We want to allow compression and quantization extensions, but we do not want to handle compression in the base extension.

We've stated multiple times over the past few weeks that compression is not a concern of the base extension, so perhaps this is another bit of scope that has creeped in that should be removed. If we include support for quantizing the spherical harmonics, we need to outline exactly how we expect it to be performed, and I'm not completely convinced that doing so is smart given the future-proofing goal we're trying to achieve with the base extension.

As an example of how this is not a straightforward decision: storing a scale factor with the INT8 quantization is one improvement, but at that point why not use just use something proven better overall for spherical harmonics like Vector Quantization, such as what CompGS? That quickly turns into a compression discussion.

This exercise has made me strongly in favor of removing the quantization aspect entirely unless someone has a very good argument as to why we need to keep it.

[lexaknyazev]

This exercise has made me strongly in favor of removing the quantization aspect entirely unless someone has a very good argument as to why we need to keep it.

As a general rule: the base glTF spec supports "basic" quantization, that is values in [0, 1] or [-1, 1] ranges with 8- or 16-bit precision. If expected values for a particular attribute fit within those ranges, then we can support quantization for that attribute essentially for free. If they don't fit, it would be better to require floating-point values for that attribute in the base extension.

This exercise needs to be done separately for every attribute.

[lexaknyazev]

As has been mentioned before, it's also possible to use non-normalized integers (that would be implicitly converted to the corresponding floats at runtime) when that makes sense.

[weegeekps]

As has been mentioned before, it's also possible to use non-normalized integers (that would be implicitly converted to the corresponding floats at runtime) when that makes sense.

If I am understanding you correctly, you'd still need a fixed scale in that case, so I don't think it makes sense for the spherical harmonics.

[weegeekps]

If expected values for a particular attribute fit within those ranges, then we can support quantization for that attribute essentially for free.

Rotation, scale, and opacity fit within those ranges. Technically, the degree-0 spherical harmonics should too, but I think I will go for consistency here and just make all of the spherical harmonics floats.

[javagl]

Highly preliminary test data:

SplatGltfTests-SNAPSHOT-2025-12-12.zip

The archive contains three glTF files that use the KHR_gaussian_splatting extension.

_mesh_in_splats.glb:

This is a glTF with a "'unit' cube splat mesh primitive", and contained in that, there's a cube (standard glTF) mesh primitive. So the splat data looks like this...

And the whole glTF looks like this:

where these "points" are supposed to be the splats, of course. (There currently is no renderer that can display this...)

_splats_in_mesh.glb:

The same, but with the standard glTF cube mesh being larger than the splat data. So... a conformant renderer would just display a cube ... and maybe parts of the splats near the origin? Who knows.

_gltf_with_grid.glb

This is a glTF that contains six splat mesh primitives. Each primitive consists of 9 splats: Eight of them at the corners of a cube, and one large one in the middle. The one in the middle is a single splat with spherical harmonics (degree 3) coefficients that cause it to look...

• red from the right
• green from the top
• blue from the front
• cyan from the left
• magenta from the bottom
• yellow from the back

These mesh primitives are attached to nodes with

• no rotation
• 90° rotation around x
• 180° rotation around x
• -90° rotation around y
• 90° rotation around y
• -90° rotation around x

and proper translation to create a "grid" where each of these cubes has a different side facing the viewer.

The following shows what this is supposed to look like:

(This was rendered with a not-yet-published state of JSplat, which supports rendering the splat data of glTFs, but not standard glTF meshes. I'll skip the question of whether that's a "standard compliant implementation of this specification")

[javagl]

The _gltf_with_grid.glb from the previous comment was highly unoptimized: It contained the same splats, converted to a mesh primitive multiple times. So there have been 6 meshes, each with one mesh primitive, and each with 20 accessors, with a total of 120 accessors (with lots of duplicate data).

To some extent, this was intentional. Renderers could make a ton of assumptions about the structure of the glTF. It is by no means self-evident that a renderer could cope with a splat primitive/mesh that is attached to two nodes, with different transforms. (I'd say that it is self-evident, but others have different priorities).

Given that all splats from a glTF will have to be sorted, globally (!) at some point, the renderer will have to shove these splats into a single collection, and sort this. Yes, in theory, other solutions could be possible. This would boil down to storing (only) the sorting order of the splats, once for each instantiation of this splat list, and then use this "list of sorting orders" to compute the final, blended result of all splat primitive instantiations. This computation of the sorting orders would have to take into account the whole global matrices of the nodes that the meshes are attached to. Depending on how common this case is assumed to be, the resulting complexity and implementation effort may not be warranted.

And... let's he honest: The case where a glTF contains more than one splat primitive will be very rare. The case that a glTF contains the same splat primitive, attached to different nodes, will probably never occur in practice. It will only occur in artificial test cases.

Like this one:

_gltf_with_grid_instanced 2025-12-13.zip

It is an "optimized" version of the _gltf_with_grid.glb, so to speak: One splat mesh that is attached to the 6 nodes with different matrices.

[NorbertNopper-Huawei]

Given that all splats from a glTF will have to be sorted, globally (!) at some point

We need to revisit this, as we need to clarify, what the impact of a scene referred and a display referred workflow for glTF means.

[javagl]

Question for the math guys: Is there anything that could be done with these "spherical harmonics lobes" that could make a sensible test asset?

Here, I just used the Y-function values for the positions, and the positions as a basis for computing the SH0/color.

Using the positions as the coefficients for the respective SH (based on l=0,1,2,3 and m=[-l,l]) looks fancy and nicely changes the colors when rotating it, but I'm not sure whether there's anything that can visually be derived/validated from that...