Animating morph targets individually

[donmccurdy]

The animation spec currently allows a channel.path value of "weights", for a sampler that must provide keyframe values for all available morph targets on the mesh. It would be useful to provide a mechanism of animating the weights of a single target, or perhaps an arbitrary subset, for two reasons:

1. Filesize: If only one target needs to be animated for an effect, providing 0s for other targets is wasted space.
2. Composability: Animation like facial expressions may be composed from multiple targets. For example, consider an animation that moves the character's mouth, and another that animates the eyebrows. Ideally those animations could be played together or separately, but playing them together may have unwanted effects if the eye animation includes weight=0 for the mouth targets.

I mentioned this in #1301, as I think that extension (if it moves forward) would be a good place to address this feature. Filing a new bug, though, just so that it's tracked for reference.

[Bug-Reaper]

I think the above post summarizes things quite nicely. As someone working on animation editing software for morph target lip sync animations, I can confirm the above statements to be real pain-points.

I'd also like to add:

3. Practicality: If you intend to do any additional modifications to an animation, it is ideal to have separated tracks for each individual morph target. Editing an all in one track is significantly more complex/difficult.
4. Understandability: A track structure where morph targets are tracked individually would allow you to understand the purpose of that animation by just looking at the track data. Conversely, the merged track structure makes the purpose of that track very hard to understand by just looking at the data.

In the interest of making this format more accessible and easier for developers to work with, I believe this would be an extremely welcome addition.

[donmccurdy]

Editing an all in one track is significantly more complex/difficult.

I think this one might be specific to how threejs imports the animation – while the glTF file stores all the morph target keyframes in one "sampler", glTF has no opinion about what defines an editor's "track", and when THREE.GLTFLoader creates a THREE.AnimationClip with a single THREE.KeyframeTrack for all targets, that could also be split into a clip with one THREE.KeyframeTrack for each morph target. Might be a little less efficient to play back, but YMMV.

A track structure where morph targets are tracked individually would allow you to understand the purpose of that animation by just looking at the track data.

👍

[zeux]

Just as a data point, morph target animations were pretty painful for me to handle in gltfpack - it's the only animation type that has variable length, unbounded, keyframe type; the layout for cubic spline is "all value tangents; all values; all value tangents" which is also kind of messy. This took time to get right, and also the weight storage layout doesn't lend itself too well to compression because values for different targets are intermixed together.

I would suggest that if glTF is extended to provide support for animating individual weights, that it should only allow for single-weight animations, not a subset - it feels like subsets, while generic, are an odd middleground that just increases the complexity further. An application would be able to decide if it needs to use a few single weights or a complete track independently.

[donmccurdy]

... it should only allow for single-weight animations, not a subset ...

That sounds like a good simplification. Most engines only support animating a limited number of morph targets at a time, so number of morph targets that could reasonably be active at once is bounded.

[vpenades]

@zeux I feel your pain when dealing with morph target animations.

Something I noticed in the specification is that depending on the vertex attribute definition, the actual maximum number of simultaneous morph targets that can be blended together ranges between 2 and 8. Being 8 the absolute maximum of morph targets that can be blended at a given time.

I believe it's designed that way because in a vertex shader, you have to bind all the morph target attributes, and there's a limit of 16 attributes per vertex shader.

So using that knowledge as an advantage, I wrote a helper structure that holds the morph weights for a given animation frame, but limited to a max of 8 weights. Which is somewhat easier to handle than an unbounded array.

[zeux]

@vpenades Yeah I considered limiting the number of supported targets to 8, but this wasn’t truly practical.

Most valuable applications of morph targets need more than 8 targets in the data set. Depending on the quality requirements you may need a few dozen shapes for facial deformation and a few more for lipsync.

Often only a few shapes need to be active (weight > 0) at a given time - engines that use vertex shader for morph target blending typically pick the 4-8 targets with the highest weight dynamically (per frame) for display. You can also use compute shaders to apply an arbitrary number of targets iteratively.

All in all a fixed limit may be appropriate when you have a specific application/data set in mind, but it doesn’t work for a general library/processing tool.

[vpenades]

@zeux sorry, I didn't explain well.... the structure I'm using for morph target animation look like this:

struct ActiveMorphWeights
{
  int Index0;
  float Weight0;

  int Index1;
  float Weight1;

  int Index2;
  float Weight2;

  int Index3;
  float Weight3;

  int Index4;
  float Weight4;

  int Index5;
  float Weight6;

  int Index6;
  float Weight6;

  int Index7;
  float Weight7;
}

So, for animations, I use Vector3 and Quaternion, and for morph targets, I use ActiveMorphWeights instead of an Unbounded array.

Obviously, this structure is able to work with animations that can have many morph targets, but limiting the number of active morph targets to 8.