APPLE: Add Gaussian Splats API schema

[dgovil]

Note: For the latest update to the schema, please read #3716 (comment)

This latest update incorporates feedback from the AOUSD Emerging Geometries Interest Group. While we will of course also take feedback left on this PR, please feel free to join via aousd.org if you'd like to participate in live discussion.

Description of Change(s)

This PR adds our proposed schema for Gaussian Splats, based on feedback from the AOUSD Emerging Geometry Interest Group, and feedback from partners like Adobe, NVIDIA etc...
I would like to credit @ld-kerley who wrote most of this schema as well as the folks at Adobe and Michael B Johnson who provided earlier ad-hoc schemas.

The schema.usda should provide most of the information on the individual schemas, but some design overviews:

1. Our schemas apply on top of UsdGeomPoints because we felt that allowed for the most compatibility with the existing ecosystem.
2. We put the schemas in a light field schema for lack of a better name. We figured we didn't want to overload UsdGeom but also wanted to leave the path open for other representations. We aren't married to the name, so other suggestions are welcome.
3. We split the GaussiansAPI and SphericalHarmonicsAPI because it is possible to have them be used independently of each other. For example Gaussians without SH data, or Gaussians that use other radiance information.
4. The Gaussians have a shapes field that is non-comprehensive. I could not find a good facility for this in the schema gen but am open to better ways to present this.
5. We took inspiration from UsdGeomPointsInstancer and added half and float versions of attributes. We find most gaussians hold up when using Halfs just fine, and there are considerable space savings. We follow the same convention of suffixing the float version with an f, but perhaps it is better to use a variable type like XformOps do?

We shipped a preliminary version of this schema in the macOS 26 Tahoe developer beta to enable users using PLY data, but are keen to replace it with something more standard.
This aligns very closely with what Adobe has as an ad-hoc data definition in the adobe file format plugin.

Some more notes:

1. We do not plan to provide a gaussian renderer at this time. The gaussians will still render as points in Storm of course which provides some measure of view ability.
2. I can write tests and docs once we're through any technical feedback.
3. Everything is unmodified after usdGenSchema so you should only need to review schema.usda

Link to proposal

This was proposed in the AOUSD Emerging Geometry IG. A matching issue is created on the Proposals repo PixarAnimationStudios/OpenUSD-proposals#90

Checklist

• I have created this PR based on the dev branch

• I have followed the coding conventions

• I have added unit tests that exercise this functionality (Reference:
  testing guidelines)

• I have verified that all unit tests pass with the proposed changes

• I have submitted a signed Contributor License Agreement (Reference:
  Contributor License Agreement instructions)

[jesschimein]

Filed as internal issue #USD-11207

(This is an automated message. See here for more information.)

[meshula]

this would be redundant with the ColorSpaceAPI, let's not have a secondary mechanism, it would get confusing :)

[dgovil]

Is there a way to specify a default color space for a parameter if none is given? I think the ColorSpaceAPI defaults to linear if nothing is given, but the harmonics are often in sRGB by default.

Either way I'll add a note saying authors should make sure to author the correct color space.

[ld-kerley]

Or Is there a way for the splats schema to modify the ColorSpaceAPI schema when used in coordination?

[spiffmon]

@ld-kerley , yes the splats schema can include ColorSpaceAPI as a "builtin applied schema", and provide an override for the colorSpace:name attribute.

[meshula]

I'd request a brief description in both cases, in addition to sending someone to the API. It's not instantly obvious why this a half3[] or float3[] instead of a half[] or float[]

[dgovil]

I added a brief description to the schema doctoring. I'm not sure how in depth we should go on it though.

[meshula]

I have a general question.

Light Field, in graphics and optics, refers to a function that describes the radiance at every point in space, in every direction. Radiance is a physically meaningful, directional quantity used for physically-based rendering, simulation and so on. GS and any other splatting don't have canonical radiance or physical units, they try to match colors and bake together emission and reflection using learned densities, and are not energy conserving, and conceptually they are more like awesome billboards.

Is the idea that you are hoping to bias into plenoptic fields and the like in the future? I do realize there is a trend to write papers with titles like "Neural Light Field Representations" but here isn't the place for me to debate that accuracy ;) Gaussian Splats are more like awesome billboards and I worry about leaning into an academic trend that doesn't reflect what we actually do with splats, and an implication of further future functionality in the light field domain, where there may or may not be ambitions in that direction?

I'm totally willing to be argued out of being fussy about this term, but you can see I'm biased to something less evocative of physics, like UsdGeomGaussianSplat ;)

I was thinking that if there was a document page to go with the PR it might help contextualize the PR against questions like this. (on proposals maybe? Right now the proposal PR loops here without a "position" or goal statement.

[dgovil]

I can put together a document for this like the new user docs and link to it.

Regarding light fields as a name, it just felt like the only term that we could think of that covered possible other representations like Nerfs in the future if someone wanted to propose them. Having just usdGaussians felt too narrow a target name if that ever came to be. Especially since people in the USD community have asked about various ML representations as well.

There wasn't really any thought behind the light field name beyond that, and 5m of Lee and I trying to think of something else.

So happy to rename it to something else if someone can come up with a suitable name.

[meshula]

Nerfs are density maps, I think you are onto something with that line of thought Volumetric Density Something Something... Nerfs and GS are volume-sample technologies. Are they UsdVol related?

[dgovil]

@meshula yeah, I debated whether they are a volume. If I squint, it feels like they could be? Happy to move it there if it feels more appropriate. It's shorter to boot.

[BrianSharpe]

In PLY, these 3 are typically stored in an awkward machine-learning format (rather than more standard CG formats)

• opacity: in sigmoid form (graphics would prefer not sigmoid)
• scale: in logarithmic form (graphics would prefer not logarithmic)
• orientation: WXYZ non-normalized quaternion (graphics would prefer XYZW normalized quaternion)

How will these be stored in USD? original machine-learning format? Or CG-format? We should at least document whatever convention it is.
(ps: I prefer they would be the CG-format :) )

[dgovil]

@BrianSharpe ah great point, I will add that to the documentation in my next push of updates.

We store it in the CG way so that it works more or less out of the box in USD (albeit as points).

We convert from the PLY encoding to the CG encoding when writing to USD (and I believe Adobe do too? I forget if this was a change we made just on our end). If you have a Mac on the 26.0 betas, you can load a PLY file up in Preview and see the representation in Storm (albeit as points) and can save out a usda with our preliminary schema to see the values.

[mjtitchener-fn]

It's not clear what the order of the values in this attribute should be: Is it point-major or SH-major order? i.e. Is it all the harmonics for point 0, followed by those for point 1, or the order-1 harmonics for all points followed by all the order-2 harmonics...?

[dgovil]

I'll add a clarification once a do a doc pass. Thanks for calling that one out.

It's all the SH values for each point, because we figured that would allow for the easiest mapping to USD conventions.

[ld-kerley]

Hopefully this ordering will allow more straight forward pruning of points in the scene - if people ever try to edit this data in USD format.

[jerry-huxtable-foundry]

I would add that the elementSize metadata should be set for this attribute to indicate how many floats/halfs are used for each point rather than it being derived by dividing the length by the number of points.

[dgovil]

Is there a way in a schema to provide a default value for metadata? If so, should we be doing that, or should we require that authors of data explicitly always write it out? Or both I suppose could be done

[spiffmon]

elementSize can and should be specified in the schema for the primvars that are known to be tuple-like. @spitzak , hold that thought for a bit...

[spitzak]

I'm not very clear on whether this elementSize you are speaking of can be different than the size of a half3. I have not used it, I only used GetTypeName to analyze stored data and I think that will only tell me it is an array of half3.

If elementSize metadata is independent, then you do have some easy support of 2D arrays, where elementSize is the size of a row.

[jerry-huxtable-foundry]

To quote the docs:

ElementSize does not generally encode the length of an array-type primvar, and rarely needs to be authored. ElementSize can be thought of as a way to create an "aggregate interpolatable type", by dictating how many consecutive elements in the value array should be taken as an atomic element to be interpolated over a gprim.

For example, spherical harmonics are often represented as a collection of nine floating-point coefficients, and the coefficients need to be sampled across a gprim's surface: a perfect case for primvars. However, USD has no float9 datatype. But we can communicate the aggregation of nine floats successfully to renderers by declaring a simple float-array valued primvar, and setting its elementSize to 9. To author a uniform spherical harmonic primvar on a Mesh of 42 faces, the primvar's array value would contain 9*42 = 378 float elements.

So, yes, it does give you sort-of 2D arrays.

[dgovil]

That is an oddly specific/applicable documentation for this situation.
Now that USD allows for the metadata to specify elementSize (I was unaware of that change), I'll include it in our next update to the schema.

[spitzak]

I also was unaware of elementSize. IMHO this is all that is needed to support 2D arrays.

[spitzak]

Can this be determined from projectionModeHint?

[dgovil]

I'm not sure. I'll ask the folks who asked for both to be present.

[ld-kerley]

The two modes are subtly different, but could be related - and both are artifacts of the splatting render technique - where the splats are rendered from back to front.

The sortingModeHint is used to control the sorting stage of the algorithm. Either sorting the splats by distance between the center of the splat and the position of the camera (cameraDistance), or using the projected z-depth in the camera projection (zDepth).

The projectionModeHint is used to determine how the actual splat shape is projected in to the image. perspective follows the typical object rendering, projected and scaled based on scene depth. tangential projects the splat orthogonally in to the image (perhaps this could be called orthographic?), avoiding any distortion introduced by the perspective projection.

[jo-chemla]

Quick question as an outsider, producer of gaussian splats scans: why are these hints (sortingModeHint and projectionModeHint) part of the format specification and embedded within a file? Shouldn't this be let to the renderers choice whether to use orthographic-vs-perspective projection into camera/screen space, and which sorting mechanism to use?

[tcauchois]

Agreed with jo-chemla on this, although we've had questions in other contexts of serializing out display mode information (e.g., subdivide this mesh to X accuracy or whatever, as a hint). This could be a render setting on the example renderer, or if we do really want to serialize it I'd advocate for it being at least a separate API schema.

[cpichard]

I feel that calling the API "GaussianSplatsAPI" would be less ambiguous since the term "gaussian" can be used for many things different than splats.

[dgovil]

We had originally called it GaussianSplats but folks in the emerging geometry interest group had mentioned that splats assumed just the ellipsoid representation. So we switched to just Gaussian.

Happy to iterate on the naming though if folks feel differently.

[cpichard]

As a non expert in the field, I have always assumed splat was referring to the projection of whatever distribution the points were assigned to on the screen. I had a look at the original gaussian splatting thesis, and Lee Alan Westover defines splats:
" In the actual algorithm, the term splat refers to the process of determining a sample's image space footprint on the image plane, and adding the sample's effect over that footprint to the image. "
His thesis focus on gaussian distributions, which are naturally supported by sphere and ellipsoid in 3d.
But it feels to me that the term splat alone doesn't necessarily embed the notion of ellipsoid, it gives the idea that each points/volumes and their associated shape and distribution are thrown to the screen, and what we see in the final image is the accumulation of those splats. For that reason, it seems to me GaussianSplats is generic enough to embed the qualities of this schema.
"Gaussians" alone is ambiguous, it is used in so many things, if people feel strongly about not adding "Splat" to the name, could an alternative be to postfix with Geom ? GaussianGeoms ? or something in this vein ?

[cpichard]

I would even argue that "SplatsGeom" could be a sensible name.
When looking that this article: https://trianglesplatting.github.io/, I see they talk about Triangle splatting, not gaussian triangles.

[natevm]

State of the art seems to not use Gaussian kernels at all.

https://rongliu-leo.github.io/beta-splatting/
The above are "Beta" kernels, which have a parameter that controls their concentration.

Outside of computer graphics, a "Gaussian" generally has an unambiguous meaning.

The "x transpose C x" defines the distance metric, which is implicitly the "Mahalanobis distance" defining an ellipsoid.
But this can be generalized by swapping out the distance metric. For example, a Chebyshev norm gives a parallelepiped. Triangle splats do something similar with their barycentric distance metric.

"Gaussian" defines that the kernel be "exp(-distance^2)". This too generalizes, to something which acts as the "loss function" in machine learning. You might alternatively want something like a leaky ReLU (avoids the expensive exponentals and vanishing gradient issues), or you might want something more flexible, which better supports neighboring particles.

At least personally, I worry that the abuse of the "Gaussian" terminology here risks the schema becoming outdated, like how OBJ materials don't accomodate physically based materials.

[natevm]

All my rambling above was meant to conclude that I like the name "Light Fields" because it's general.

"Light" seems to be implied by the spec talking about spherical harmonics encoding radiance/irradiance. At the very least, one could argue the extension is generally meant to be used for applications creating pictures which require light to be seen.

"Fields" generalizes reasonably well to arbitrary learned "density field" geometries (nerfs, radial basis functions which may or may might not be Gaussian, may or may not be ellipsoidal / triangular / etc).

Also sounds cool :)

So, not GaussiansAPI, rather, LightFieldsAPI.

[ld-kerley]

Naming is hard :)

We considered adding this schema to UsdGeom - but wanted to try and make things easier for USD implementations to communicate that the splats aren't supported, by keeping is separate. If thats less of a concern - I could make an argument that the GaussiansAPI could live in UsdGeom. But then it would be unclear to me if the SphericalHarmonicsAPI would live along side or not.

Really happy to defer to whatever the community thinks is best here.

[jo-chemla]

Impressive discussion and insights, nice to see the convention naming being debated so it's as future-proof as possible!

Before the initial release of the gaussian splatting paper by GraphDeco at SIGGRAPH 2023, splats could be used to represent standard pointclouds, by projecting to screen discs (oriented towards normal or screen), spheres, or alternative primitives (eg paraboloids & co, see nice @m-schuetz thesis around p. 33).

Now splats can indeed mean gaussians, triangles, or any other form of density function like mentioned Beta-Splats, with renderers either rasterizing or ray-tracing the primitives like EVER or 3DGRT.

[cpichard]

typo represeted -> represented

[natevm]

I'd argue what people call a "splat" is the projection of an ellipsoid. The word "splatting" is a friendly way to describe "projection". When we project ellipsoids onto the screen, this produces 2D "ellipses".

Ellipsoids are 3D solids. For reference, the EVER paper here also uses ellipsoids, though they are splat onto a 1D line (the ray) rather than the 2D rectangles (the pixels on your screen).

The "plane" token here is a bit ambiguous. A plane is an infinite flat surface defined as where dot(n, x-p) = 0. Triangles, rectangles, disks, and any other cutout of the 2D plane would themselves also technically define "planes", but are more than just that. Planes also imply some 2D surface embedded within 3D space.

I'd propose replacing "plane" with "ellipse", and then if necessary, we could add "sphere" and "circle" if the particles define symmetric 3D/2D cutoffs within a volume/plane vs asymmetric 3D/2D ellipsoids/ellipses.

I'd probably suggest we consider adding a "parallelepiped / box" too, as that would generalize to what you might use to define the cutoff of a transformed NeRF / voxel grid.

I'd also propose that this be renamed to "cutoffShape" rather than "gaussianShape". That would generalize better to splatters which don't use Gaussian falloff distributions like "Deformable Beta Splatting" does. Then, "falloffShape" would become another uniform token type, from which "Gaussian" would be one option, "Linear" might be another, "Beta" could be a third, etc.

The difference between "stop the pop" and 3DGS style splatters then becomes more a matter of how that cutoff geometry is projected onto the screen (or ray).

[natevm]

I'd probably also argue that all of these cutoff shapes should be convex. So a crescent would not be allowed as a "cutoffShape", and instead, we'd require that a crescent be composed from eg multiple ellipses.

[ld-kerley]

+1 I like cutoffShape. Or maybe just shape ?

+1 renaming plane - I'd propose rectangle - I think the intention here was to suggest a quad as the rasterized shape.

Do users already have use-cases for falloffShape? That isn't something that has come us so far for us. Maybe we should wait to add it until we have concrete use-cases?

[ld-kerley]

we should probably match the capitalization of the tokens in the description here.

[meshula]

@ld-kerley Here's a way to compatibly use a normals primvar without breaking the primvar :)

/// Converts a scaled axis-angle vector (axis * angle) to a unit quaternion.
/// Compatible with OpenUSD GfQuatf and GfVec3f.
///
/// If the input vector is zero, returns identity quaternion.
pxr::GfQuatf ScaledAxisAngleToQuat(const pxr::GfVec3f& scaledAxisAngle) {
    float theta = scaledAxisAngle.GetLength();
    if (theta == 0.0f) {
        return pxr::GfQuatf(1.0f, pxr::GfVec3f(0.0f)); // identity quaternion
    }

    pxr::GfVec3f axis = scaledAxisAngle / theta;
    float halfTheta = 0.5f * theta;
    float sinHalfTheta = std::sin(halfTheta);
    float cosHalfTheta = std::cos(halfTheta);

    return pxr::GfQuatf(cosHalfTheta, axis * sinHalfTheta);
}

/// Converts a unit quaternion to a scaled axis-angle vector (axis * angle).
/// Compatible with OpenUSD GfQuatf and GfVec3f.
///
/// If the quaternion is identity, returns zero vector.
pxr::GfVec3f QuatToScaledAxisAngle(const pxr::GfQuatf& quat) {
    const float w = quat.GetReal();
    const pxr::GfVec3f v = quat.GetImaginary();

    float sinHalfTheta = v.GetLength();
    float cosHalfTheta = w;

    if (sinHalfTheta == 0.0f) {
        return pxr::GfVec3f(0.0f); // no rotation
    }

    float halfTheta = std::atan2(sinHalfTheta, cosHalfTheta);
    pxr::GfVec3f axis = v / sinHalfTheta;

    return axis * (2.0f * halfTheta);
}

[dgovil]

Thanks for the great feedback everyone. We'll do a pass on these after discussing some details and post an update soon to reflect what is hopefully a good aggregation of everyones thoughts

[spiffmon]

A few more late thoughts.

[spiffmon]

Typed vs API

While I see the desire to have the new schema encoding "just work" as Points in non-updated renderers (and even old versions of USD), I think the compromises we are making will seem unfortunate in five years, and I also think that it will be very useful for users, GUI's and workflows for this to be a concrete, typed schema - the Gaussian-splattiness really is the core identity/behavior here. This would avoid complexity/ambiguity with widths, normals, and displayColor being used for spherical harmonics coefficients (which I'll talk about more for that schema). I think it will be sufficient to address "conversion to Points data for unprepared renderers" to be something that a Hydra scene index filter takes care of; while that leaves us with no support in older versions of USD, that's no different from how most newly introduced gprims will come in (e.g. Breps).

Someone else noted the similarity to PointInstancer, and I am curious if there is value in exploring a "PositionBasedAPI" that would be builtin to both a GaussianSplat (I'm not wading into the name game more than that) and PointInstancer schemas? The main potential awkwardness I see, if the concerns precision concerns expressed here hold up is that while in PointInstancer (and followed here) we have the "main" orientations in half, while scales are float, so if we wanted half-precision scales introduced in Gaussians, it would be with an h suffix, which is opposite to orientations.

Compression

Those encoding decision also lead to consideration of compression and alternate encodings. If we want GPU's to be able to get access to compressed data directly, then we cannot rely on fancy FileFormats to provide compression unless it's literally a reference out to an OpenVdb asset or something (which would also be nice to see discussed? Should we be thinking about these like Volume s?) it must be represented explicitly at the schema level. Do we believe compressed forms will be entirely different schemas? Or should we be thinking ahead about adding a forward-looking attribute that declares compression scheme, with alternate encodings provided by future API schemas added on?

Animation

The common real-world way of getting this data generally (I think?) does not allow for animated data, though we can generate such synthetically from renderers. If animated gsplats completely blow people's' minds, we could make the data uniform, but if there are potential interesting use-cases, we just make sure renderers can handle it...

Material Bindings and Transparency

Presumably material bindings have no effect at all on gsplat rendering? If so, we should make a general statement about that. Relatedly, and just a point of personal curiosity, is there another expectation with gsplats that they do not handle transparency? Or rather that anything transparent in the dataset is locked into whatever was behind it at capture time? Probably also worth stating for folks less familiar with them.

[spiffmon]

This is way too strange to be in something that claims to be a general SphericalHarmonics schema. Another thing resolvable by detaching from Points as a fallback representation, this schema should contain all the coefficients. If it is useful for LOD and fast-path rendering to have the zeroth coefficient separated out, let's do it, but do it here as a separate attribute. Would all three coefficients want to be separate so you could efficiently read a prefix of all or any of them without pulling the entire thing into memory?

It's also worth discussing whether they want to be encoded as primvars, made directly available to the shaders, or whether they are regular attributes consumed by the renderers themselves? Regarding the discussion around elementSize I can relay that we will, soon, be adding a core arraySizeConstraint field to UsdAttribute that allows any array to declare itself either fixed-size or constructed of tuples, as elementSize does.

[dgovil]

Thanks for the comments, @spiffmon

I think some of the recent changes to OpenUSD that you listed will let us do what you suggest.

Also, thanks everyone else for your notes. They're all heard and we'll do our best to integrate whatever feedback we can. We're just waiting on some notes to be posted from a few other places

I'll discuss with Lee and we will start working on the requested changes that are here and we've received offline. We'll probably start on that work after SIGGRAPH.

[raymondyfei]

There's a growing trend of (stochastically) raytracing Gaussian splats as well as works on stochastic rasterization of the splats. In those rendering methods, the Gaussians are sorted per-pixel instead of per-splat (i.e., only sorted with the center of the splats). This makes a difference, even though both per-pixel and per-splat sorting methods utilize the camera distance (i.e., per-pixel refers to the distance from the camera/ray-origin to the pixel or hit point between the ray and the splat, while per-splat is the distance from the camera to the center of the splat hit by the ray).

It would be helpful to differentiate whether the Gaussians are sorted per-pixel or per-splat during creation.

[ld-kerley]

I just pushed a pretty big update. This comes as a result of several conversations in different groups, including the AOUSD Emerging Geometries Interest Group. The AOUSD EG IG is where most of the active conversation is happening, and I would encourage anyone who is interested to attend the meetings there (the next one is Thursday, 16th October at 12 p.m. PST).

As with the previous update - the only significant file here is schema.usda - the rest are all generated by usdGenSchema.

The community has expressed a desire to support both a practical, concrete schema as well as having a system that is extensible in the future. To allow for this, we have currently settled on a concrete schema type to provide a base ParticleField. This base contains no data but is the point that a number of different appliedAPI schemas can be attached to. AppliedAPI schemas are used to define the required characteristics of a given type of ParticleField. These currently fall broadly into the following categories:

• Base attributes - these define the particles themselves, with attributes such as position, scale, and orientation.
• Kernel definition - these define the shape and falloff used for each particle in the ParticleField.
• Radiance definition - these define the appearance of the ParticleField.
  This is the current set of categories, but the system does not preclude new categories being added in the future as the technology evolves.

This allows extension and experimentation with different implementations of each of these. For instance, some stakeholders have expressed interest in defining the position of the particles using a multilayer perceptron (MLP). We do not propose an appliedAPI schema for this yet, but we can imagine someone defining a PositionMLPAPI appliedAPI schema that could be used in place of the currently suggested PositionAttributeAPI schema to define the locations of the particles.

These extensibility/configurability do mean it is possible to construct an "incomplete" ParticleField. We do not currently propose taking an opinion on what "incomplete" means here, as it may evolve. We do, however, offer an opportunity for avoiding the possible complications of an "incomplete" ParticleField, by providing additional concrete type schema for explicit types of ParticleFields. Specifically, in the PR, we see a ParticleField_3DGaussianSplat schema, that pre-applies the necessary appliedAPI schema to create what we consider the USD representation of the classic 3DGS data. This concrete class has been defined to be complete to describe this specific "flavor" of ParticleField.

We hope this approach strikes a good balance between extensibility, robustness, and practical application.

[spiffmon]

Final review notes:

1. To aid in creation of user-docs (which themselves would be nice to have as contribution), and especially since there's no other proposal artefact, it's important to have a overview.md in this PR that discusses the motivations/purpose of LightField, the modular design that ParticleField takes (and why), and if we have ideas, what other kinds of future concepts might find a home here?
2. Particularly given the decision to name this domain LightField, we either need to discuss why the schemas it contains are not meant to be used in combination with UsdLuxLightAPI, or inform implementors that it well may, going as far as (as with Mesh and Volume), defining a ParticleFieldLightAPI schema here?

[spiffmon]

sp: desribe

[spiffmon]

"appliedAPI" isn't core terminology - can we just say "applied schemas" here and throughout?

[spiffmon]

sp: atleast

[spiffmon]

no need for "it" here

[spiffmon]

Question about the general pattern being proposed here... how do we do configuration validation for ParticleField prims? If we require that all new applied configurations/flavors must be contributed directly to usdLightField, then a validator written for ParticleField can perform the validation. But if we allow plugins to add schemas that are alternatives to core ones, then we have a problem. As USD gets more serious/industrial, it would be good to consider this early, before the ecosystem gets complicated.

[spiffmon]

One thought here that derives from some of our earlier thinking about being able to identify "families of API schemas that are alternates to each other, but you shouldn't have more than one-of applied" that I recalled when looking at the "Shape" schemas below is... if there is a "base", LightFieldParticleShapeAPI applied schema, then each user-facing shape schema can include that schema as a "builtin" applied schema of its own.. then validation can be formulated in terms of "there must be exactly one shape API applied" ?

Note the UsdSchemaRegistry may be missing some API to answer exactly that question, but that's work we should do, if this is an approach we think is useful, here.

[spiffmon]

tricky interplay between schemas, here, and illustrates the need for and kinds of validation we'll want.

[spiffmon]

(Maybe ironically?) To my mind, this seems like an instance where we do not need a ParticleField (or LightField) prefix for the schema name? Do we really believe there are likely to be other 3DGaussianSplat schemas?

[ld-kerley]

Totally happy to defer to what people think is best here - my thinking with the "ParticleField" prefix was to indicate that this is a concrete instance of a ParticleField - but if thats unnecessary, or too verbose I'm open to shortening or dropping it.

I might leave this as is for now - to allow others to weigh in - but will raise it as a discussion point at the next AOUSD meeting

[spiffmon]

This morning I had another naming question about this schema... given how it is (nicely) constructed with a particular configuration of add-on schemas to provide everything it requires... what do we think would happen in the future if we coalesced on a GaussianSplat that was driven by MLP (or something else) - what would that be called? Would we be making things too awkward at the inception to start off naming this thing 3DGaussianAttributeSplat or something like that?

[ld-kerley]

I think I want to say "no" here. The original paper refers to these things as 3D gaussian splats, and describes per-point attributes, as is typically seen in the common ply/splat file formats.

I think if in the future there is a GaussianSplat that is driven by MLP then we would name that concrete schema with whatever new term was coined to describe that. I know that the NVIDIA folks are further along with their ParticleField MLP work, so perhaps they have some insight as to possible names here. I'll reach out and get them to offer feedback here I they have it.

[spiffmon]

without any other references to kernels anywhere, this namespaceing seems to come out of left field... if it makes sense to the practitioners, maybe a note of explanation is sufficient?

[ld-kerley]

Maybe I missed one of the applied schema somewhere - but my thought process here somewhat aligns with the base applied schema idea you raise above.

There are (currently) three different families of schema, kernelShape, kernelFalloff, radiance.

The thought process here is that each of the applied schema in those families would declare any attributes necessary inside its respective namespace - hopefully making it clearer in the concrete particle field case, which attributes contribute to which "part" of the particle field. As well as potentially protecting our future selves from attribute name clashes.

Again, if this feels un-USD like, or there are objections I'm happy to remove it - I don't think its fundamentally important to the data, just felt like it might be a little future proof?

Again - I'll leave as is for now - so others can weigh in.

[spiffmon]

Since none of these properties identified as primvars will be inheritable down namespace anyways, do they actually need to be primvars? I don't think we want to overly-select on saying they are direct inputs to shaders (which primvar implies). In 25.11 and beyond, we have introduced arraySizeConstraint at the core Sdf-level, which allows specification of the same element-bundling that primvar elementSize does. Hydra isn't yet consuming it, but if that's the reason for using primvars here, it need not be.

[spiffmon]

Same observation for uses of primvars below...

[ld-kerley]

Happy to remove primvar categorization - didn't realize it was no longer necessary. thx

[spiffmon]

sp:zDeph

[ld-kerley]

@spiffmon - thanks for the great feedback (and proof reading my terrible typos). I left inline comments for the discussion points that aren't obviously resolvable, and will raise those as discussion points at the next AOUSD meeting (in about 90 minutes time :) )

[ld-kerley]

@spiffmon - we had a lively discussion in the last AOUSD meeting.

One of the topics we discussed was that of which usd domain this belongs in. UsdLightField was really just an initial proposal to get the conversation going, and several at the meeting, including myself were swaying in the direction of actually moving all of these ParticleField schema to the UsdGeom domain. The ones we got a little hung up on were the radiance providing schema, like the spherical harmonic ones. One idea was to present the idea of these radiance providers as analogous to the current displayColor/displayOpacity attribute. Meaning an indicator of how a ParticleField might be rendered if there is no material bound to the location. We do think that future technological advances will want to bind "materials" to a ParticleField, so this feels like it would align well. I'm guessing if things get moved to UsdGeom then all the schema would have a ParticleField prefix.

Before I reframe the PR in UsdGeom I wanted to check in with you and see if you had any thoughts about this direction.

The other discussion point that I'd love your feedback on was if the ParticleField base case should inherit from GPrim or Boundable. It's unclear if concepts like doubleSided or orientation from the GPrim schema are really all that applicable to ParticleFields at least in terms of current research, and with the idea that perhaps radiance providing schema replace things like displayColor/displayOpacity it feels like Boundable might make more sense, but I'd love your perspective incase I'm missing something.

[spiffmon]

@ld-kerley , I think Boundable makes alot more sense than Gprim, actually.. here's another reason: every gsplat I've ever seen represents something that, if explicitly modeled with actual UsdGeom geometry, would consist of many gprims, many materials, etc.

The other question is going to take more time and talking to folks. My initial concerns are about UsdGeom size and complexity, and what the unifying concepts are. Two related notes:

1. we formulated the single-schema UsdProcProcedural out into its own domain, and that has a similar "represents many geometries and materials/looks" flavor
2. It's quite likely that the number of schemas in UsdGeom will nearly double when we add support for double-precision geometry, rather than going the direction that this proposal is going with support for multiple precisions put into multiple properties in the same schema (which does have precedent in PointInstancer, but we have concerns about scaling that pattern up.

[spiffmon]

@ld-kerley , we spent time discussing unifying concepts, and the TL;DR is that we believe a LightField domain makes the most sense of the options on the table (UsdLightField, UsdGeom, UsdLux, UsdVol) - particularly if we look ahead potentially to nerfs, which fit even less in UsdGeom than ParticleField. Details:

1. Other than primvars:displayColor and primvars:displayOpacity, which should be thought of as "UI hints" that a renderer can choose to use or ignore if it has no capacity to process higher-level material characteristics, @gitamohr notes that UsdGeom deals only in geometry - or put another way, as stated by @nvmkuruc to me, things that can occlude. While a PointInstancer can instance complete asset descriptions (and will be able to instance ParticleFields) that contain higher-level concepts, it does so in a blind way: it instances prim hierarchies.
2. @meshula notes also that everything in UsdGeom submits to Euclidean characteristics that every element can be defined with an exact distance from every other element of a Geom primitive. Given the falloffs and density/volumetric nature of the ParticleFields being proposed here, we don't think they adhere to that property.

[tcauchois]

Solid stuff all around! Left my own idiosyncratic opinions about the schema and some small procedural notes on the hydra stuff.

[tcauchois]

This doc is copied & pasted from the kernel API, & needs an update. Something along the lines of "defines a radiance function parameterized by position and angle on the field prim it's applied to"?

[tcauchois]

As a renderer author, I hope all particles have the same spherical harmonics degree? Which makes the "maximum" here a bit confusing.

[tcauchois]

"The SH coefficients are grouped in the array by particle, meaning each particle has N contiguous coefficients, Y(m,l) sorted first by order (m) and then within the order by index (l). A renderer can compute an element size per particle based on the SH degree, and use that to stripe the array by particle."

[tcauchois]

Let me know if I'm misinterpreting, but these two hints seem akin to "display style" for a mesh or curve, e.g. how many times you subdivide a catmark hull for display? But not intrinsic to the scene data itself. I understand the utility of serializing these for reproducibility, but if they are indeed viewer hints I'd rather see them in an optional API schema.

[tcauchois]

Here & elsewhere in this directory: I think keeping with the "ParticleField" (i.e. base class) naming is preferable, and distinguishing between variations in hydra either via prim type (have a helper on the ParticleFieldAdapter that comes up with prim type based on prim.HasAPI() or whatever, as in UsdLuxLight); or by a token "kernelType" on the prim telling hydra which kernel to use (as in UsdGeomBasisCurves). All of this to minimize code duplication where we can.

[tcauchois]

To expand on that, the class itself is nicely parameterized in terms of telling you which data you're on the hook for pulling down, meaning it's possible to write a single particle field adapter that can ingest data for any specialization, saving a bunch of LOC (much like we do for e.g. lights, shaders, sample & display filters in Renderman, etc.). Since this is the only current specialization, this isn't an urgent note.

[tcauchois]

Coding convention is "_" indicates a module-private object/definition, at least in C++, so I was confused to see one here. Not sure if others feel the same way?

[spiffmon]

Good call, @tcauchois . While schema names are not prohibited from having underscores in them (nb. schema versioning uses "_INTEGER" schema name suffixes to define schema versions), we request that any codeful schema contributions to OpenUSD itself, like this one, refrain from containing them.

[tcauchois]

Convention is to have a CMakeLists.txt for each directory... My inclination would be to promote the USD adapters to pxr/usdImaging/usdLightFieldImaging directly? And then put the whole hydra implementation just directly in pxr/extras/imaging/examples/hdParticleField.

[tcauchois]

I know this is example code, but the OpenImageIO include here is troublesome, and it doesn't look like you're using it for anything other than allocating scratch space + iterating? Or are you using it for type conversion too? I'm wondering if there's a way to rewrite this simply without the dependency.

[tcauchois]

Neat implementation, I like it! :)

[tcauchois]

High-level semantic ruminations:

1. I skimmed some discussion of the lightfield name, but it seems appropriate to me? With the SH radiance closure, splats are totally a weird discrete approximation of a plenoptic function. My read of the schema was that we left the door open to other representations of radiance closures, but insisted on some kind of representation.
2. For splats specifically, I really like the particle field name, as the thing that's specifically interesting about them (vs a rando VDB representation) is that they can be efficiently stored and rendered as particles with spatial basis function support. I know true particles are spatial impulses (i.e. 1d points), but particle systems have been around for a while and make a good analogy.
3. The particle field class bakes in light field semantics, but is also parameterized so that in the future you could maybe have a non-particle-based lightfield subclass of it? UsdVol splits the semantics up into Volume (i.e. a light transport algorithm) and Field (i.e. a data storage specifier that you can sample). I'm wondering whether that split would make any sense here, although I suppose since the rendering algorithm is pretty hardcoded the "Volume" analogue would contain nothing but a relationship to the "Field" analogue.
4. I do think there's a case to be made for these schemas being specializations of the "usdVol" concepts. Like, conceptually a splat is a basis decomposition of a volume field with the outputs "presence" and "radiance closure", so there's a case to be made for ParticleField subclassing VolField; although ParticleField also defines its own rendering algorithm, which VolField doesn't, and practically I'm not sure if there's utility in relating the two.

[tcauchois]

One more note I forgot :). We're switched over to datasources in UsdImaging now, and this only maps "positions" and won't correctly fetch the other parameters you need e.g. "orientations", "scales", "opacities", "SH coefficients". Should just be a matter of adding them to this mapping list though?