I want an general extension for spatialized audio, but have no business writing one

[amwatson]

Hey all,

As per some discussion at glTF next and W3C, I want to put forward a schema for adding spatialized audio support for glTF. Given the great work and interest in both audio and glTF coming out of today's session, I'd love to have something linkable for tomorrow. With that in mind, I think we could be more productive with something that isn't a vendor pull request. How do we feel about this as a start?

Overview

This extension adds general support for spatialized audio in glTF scenes. Audio sources are specified at the node level, and are colocated with the location of their node/joint in worldspace.

Property	Type	Description	Default Value
source	string	ID of the audio source container	None
volume	float	The factor by which the audio source's volume is scaled for playback. An Audio source with volume 0.0 is considered to be disabled, and may be ignored	0.0
reflection	string	Audio source attenuation mode	false
attenuation	boolean	Denotes whether reflection should be enabled (true) or disabled (false)	'NONE'
diameter	float	Virtual diameter of the spherical audio source in meters. Larger sizes provide more envelopment for volumetric sounds	0.0
maxSpeed	float	Max speed at which the audio source can travel, in meters/second. Unanticipated jumps in motion may produce artifacts during positional interpolation	0.0
startOffset	float	The number of seconds between the time at which audio playback is requested, and the start time of the given audio source. If 'loop' is set to true, the audio source will restart playback 'startOffset' seconds after the end time of the previous iteration	0.0
loop	boolean	When true, the audio source is played continuously in a loop, restarting 'startOffset' seconds after the previous iteration has completed	false

Schema

glTF (gltf.XXX_audio.schema.json)

{
    "$schema" : "http://json-schema.org/draft-03/schema",
    "title" : "Spatialized Audio extension",
    "type" : "object",
    "properties" : {
        "audio" : {
            "type" : "object",
            "description" : "A dictionary object of audio sources.",
            "properties" : {
            },
            "additionalProperties" : {
                "$ref" : "audio.schema.json"
            },
            "default" : {}
        }

    },
    "additionalProperties" : false
}

audio (audio.schema.json)

{
    "$schema" : "http://json-schema.org/draft-03/schema",
    "title" : "audio",
    "type" : "object",
    "description" : "Data used to create an audio source",
    "extends" : { "$ref" : "glTFChildOfRootProperty.schema.json" },
    "properties" : {
        "uri" : {
            "type" : "string",
            "description" : "The uri of the audio source.",
            "format" : "uri",
            "required" : true,
            "gltf_detailedDescription" : "The uri of the audio source.  Relative paths are relative to the .gltf file.  Instead of referencing an external file, the uri can also be a data-uri.",
            "gltf_uriType" : "Application"
        }
    },
    "additionalProperties" : false
}

node (node.XXX_audio.schema.json)

{
    "$schema" : "http://json-schema.org/draft-03/schema",
    "title": "XXX_audio node extension",
    "type": "object",
    "properties": {
        "source": {
            "extends" : { "$ref" : "glTFid.schema.json" },
            "description" : "The audio source ID",
            "required" : true
        },
        "volume" : {
            "type" : "float",
            "description" : "The factor by which the audio source's volume is scaled for playback.", 
            "gltf_detailedDescription" : "The factor by which the audio source's volume is scaled for playback. An Audio source with volume 0.0 is considered to be disabled, and may be ignored",
            "minumum" : 0.0,
            "default" : 0.0,
            "required" : true
        },
        "reflection": {
            "type" : "boolean",
            "description" : "Denotes whether reflection should be enabled (true) or disabled (false)",
            "default" : false,
            "required" : false
        },
        "attenuation" : {
            "type" : "string",
            "description" : "Audio source attenuation mode",
            "enum" : [ "NONE", "FIXED", "INVERSE_SQUARE" ],
            "default" : "NONE",
            "required" : false
        },
        "diameter" : {
            "type" : "float",
            "description" : "Virtual diameter of the spherical audio source in meters",
            "gltf_detailedDescription" : "Virtual diameter of the spherical audio source in meters. Larger sizes provide more envelopment for volumetric sounds",
            "minimum" : 0.0,
            "default" : 0.0,
            "required" : false
        },
        "maxSpeed" : {
            "type" : "float",
            "description" : "Max speed at which the audio source can travel, in meters/second",
            "gltf_detailedDescription" : "Max speed at which the audio source can travel, in meters/second. Unanticipated jumps in motion may produce artifacts during positional interpolation",
            "minimum" : 0.0,
            "default" : 0.0,
            "required" : false
        },
        "startOffset" : {
            "type" : "float",
            "description" : "The number of seconds between the time at which audio playback is requested, and the start time of the given audio source",
            "gltf_detailedDescription" : "The number of seconds between the time at which audio playback is requested, and the start time of the given audio source. If 'loop' is set to true, the audio source will restart playback 'startOffset' seconds after the end time of the previous iteration", 
            "minimum" : 0.0,
            "default" : 0.0,
            "required" : false
        },
        "loop" : {
            "type" : "boolean",
            "description" : "When true, the audio source is played continuously in a loop, restarting 'startOffset' seconds after the previous iteration has completed",
            "default" : false,
            "required" : false
        }

    },
    "additionalProperties": false
}

[tparisi]

The purist in me thinks this should be in a separate spec.

We do need to define some node-like aspects e.g. Model matrix but otherwise all the other sound properties, even the spatial aspects, e.g. Diamater, feel like they should be specified in a completely different spec and the leave it up to the app to bind the gltf node and spatialized source together e.g. By node name, metadata or some other application specific scheme

[tparisi]

My $.02 after two minutes of thought. Let's discuss tomorrow

[pjcozzi]

I don't know the first thing about audio, but I think this would fit very well as a KHR_ extension. The proposed schema is inline with glTF conventions.

Putting this into the core glTF spec would be harder to justify since many engines do not use sound and would not want the implementation burden.

[tparisi]

Everyone needs to look at the Web Audio API. There is a session on it right now at W3C VR workshop.

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

We should be able to declaratively represent the properties represented in this API. e.g. they have a Node concept as well.

But again, maybe this should be in a separate spec or at a higher declarative layer such as A-frame or React...

[amwatson]

Putting this into the core glTF spec would be harder to justify

I would fight very hard against it being core spec, fwiw :)

We do need to define some node-like aspects e.g. Model matrix but otherwise all the other sound properties, even the spatial aspects, e.g. Diamater, feel like they should be specified in a completely different spec

I'm open to considering more external solutions -- in fact, I think what you're describing is more-or-less the hack I cribbed this schema off of. Originally, when I was prototyping my audio engine, I had a sounds.json file that listed the above properties for a sound source, along with the name of a glTF node.

leave it up to the app to bind the gltf node and spatialized source together e.g. By node name, metadata or some other application specific scheme

I don't have a lot of experience with spatialized audio, and could be convinced pretty easily that, for instance, no general list of properties can define a sound source across applications. Here are my concerns:

• In my prototype setup, the audio engine was indeed kind of an app-defined "drop-in"; however, since I needed to know, for instance, the position of an animated joint in worldspace, I essentially needed to know the end of the rendering pipeline at each frame -- that didn't seem to justify decoupling spatialization from rendering
• In my experience working with audio folk, the properties specified (and this is coming from the Oculus audio SDK) are describing the model/scene, not the audio data. I think of the audio properties almost as a material definition, and trying to keep that application or platform specific doesn't seem in-line with the spirit of glTF.

[tparisi]

@amwatson where is the authoring taking place? probably not in a modeling package like Maya, right? except maybe that artist putting in a placeholder node to position/orient it. Is that correct? I want to know because to me this is also about enabling the best/easiest pipelines. The common case to me seems to be that, with the exception of an artist putting in a node placeholder, the audio is added in a separate step, basically in the application. If that's true then it would be better to have the rest of the audio data in, say, a React or A-frame component

[amwatson]

in my limited experience, an audio artist came in and specified those properties by hand in the json file; the audio SDK took care of the rest. It would be interesting to get input from W3C folks what audio tools, if any, they use, but I think we can assume it is not going to be a part of a major modeling package.

It's possible there's a sensible division between application-specific properties and model/artist/scene-specific ones.

[amwatson]

The pipeline problems for me would be 1. the rendering/spatialization division, since trying to keep audio at an application level wasn't practical, and 2. How the scene container is being defined, since a model that's loaded without these properties wouldn't be the "scene" in full.

[tparisi]

right, so we should strive to keep the audio data in a separate piece of content, with the exception of these placeholder nodes I'm talking about... the rest of the audio information should be specified elsewhere and there are different groups talking about all this now - Web Audio, Dolby etc.

[amwatson]

What I was getting from the talks seemed mixed. I'll try to ask around and refine my understanding.

[lexaknyazev]

glTF aims to be a JPEG-for-3D, in other words, a self-contained 3D-multimedia package (with emphasis on models). I can see at least two use cases where deeply integrated audio is reasonable:

• An asset for use in "metaverse": model with associated sounds (like ticking clock or car engine). It makes sense to keep audio with other media (such as textures) and describe all of them in one glTF.
• Complete scene with with models, environment, animations, sounds, etc, intended for non-gaming "exploration". Think of other rich-media containers: MP4, PDF, even PPT - they could contain several heterogeneous media streams suitable for easy end-user consumption.

[zellski]

We were just discussing whether or not anybody had ever wanted to include spatialised audio in glTF files. Did this extension ever get further traction?

[pjcozzi]

I don't believe there was any addition work on this, but it would be great to pick up if you have the bandwidth.