Implement audio absolute time (DSP time) and scheduled play

[PizzaLovers007]

Note: This PR is superseded by #107226.

---

Implementation is based off of @fire's notes 4 years ago: https://gist.github.com/fire/b9ed7853e7be24ab1d5355ef01f46bf1

The absolute time is calculated based on the total mixed frames and the mix rate. This means it only updates when a mix step happens.

Specific play_scheduled behaviors:

• If a sound is playing, play_scheduled() will stop that sound (with single polyphony). This matches the behavior of play().
• If a sound is scheduled, then paused, then resumed before the schedule happens, the sound still plays at the correct scheduled time.
• If a playing sound is paused, then play_scheduled() is called, the sound will restart from the beginning. This matches the behavior of play().
• With a higher max_polyphony, multiple sounds can be scheduled, and playing sounds can continue playing.
• play_scheduled is unaffected by pitch scale.
• play_scheduled does not support samples. The "Stream" default playback type is required for Web builds (ideally with threads enabled).

Scheduled stop is not implemented due to limited use cases.

• Demo project: https://github.com/PizzaLovers007/play-scheduled-test
• Web build: https://pizzalovers007.itch.io/godot-play-scheduled-demo?secret=l7sWHu1qrjbd5DQcEO1mpm4EQE

Fixes godotengine/godot-proposals#1151.

[fire]

I removed breaks compat because we decided to only change the new parameters to double.

Edited:

At least that's the approach we want to take. It may still be breaking compat, but that's not by design and is a bug.

[PizzaLovers007]

Added, thanks for the pioneer work and review!

[Mickeon]

Part of what originally confused up on the purpose of this PR is that there's no documented reason for this method to exist.

What I mean is, at a glance, most users will look at both play and play_scheduled. They will see that the latter has a longer name, longer description, and takes longer to set up. As such, they may stick to _process() callbacks, Timer nodes, etc. for accurate audio purposes. There's barely any mention, or any examples as to why this should be used over play().

I have no suggestions about that, but I would heavily recommend to at least provide one.

[Mickeon]

I'm generally questioning an implementation detail. I have not looked much at the prior conversation, so I'm not sure if this has ever been brought up.

If AudioServer.get_absolute_time is so integral to AudioStreamPlayer.play_scheduled, why should it be fetched every time in user code? Why should play_scheduled require an absolute time, instead of relative time?
Essentially my suggestion is as follows:

# Before:
var future_time = AudioServer.get_absolute_time() + 1
player1.play_scheduled(future_time)

# After:
player1.play_scheduled(1)

Regarding the precision of these methods, I wonder if #81873 would be of any inspiration, or if it would even supersede AudioServer.get_absolute_time in some situations?

• See #81873

[PizzaLovers007]

Here's a trimmed down version of the metronome in my demo project:

var _start_absolute_time: float
var _scheduled_time: float

func _process(_delta: float) -> void:
	# Once the currently scheduled tick has started, begin scheduling the next
	# one.
	var curr_time := AudioServer.get_absolute_time()
	if curr_time > _scheduled_time:
		var next_tick := ceili((curr_time - _start_absolute_time) / beat_length)
		_scheduled_time = _start_absolute_time + next_tick * beat_length
		play_scheduled(_scheduled_time)

func start(start_absolute_time: float) -> void:
	_start_absolute_time = start_absolute_time
	_scheduled_time = start_absolute_time
	play_scheduled(_scheduled_time)

If we were to change the API to be relative, reimplementing it would look something like:

var _delay: float

func _process(delta: float) -> void:
	_delay -= delta
	if _delay > 0:
		return

	# Now that the currently scheduled tick has started, begin scheduling the
	# next one.
	var curr_time := song_player.get_playback_position()
	var next_tick := ceili(curr_time / beat_length)
	var relative_delay := curr_time - next_tick * beat_length
	play_scheduled(curr_time)
	_delay = relative_delay

func start(delay: float) -> void:
	_delay = delay
	play_scheduled(_delay)

For scheduling sounds at the very start, using relative seems a lot easier since you only need to care about the initial delay. However once the game logic advances past that first frame, you need some sort of "current time" in both cases in order to know when to schedule sounds (in the above example, the next metronome tick).

Using player.get_playback_position() without any correction would actually be ok today since the value only changes on mix steps. This would align the delay param with the mix buffer. The issue is that play_scheduled becomes strongly coupled with an active player, so you would essentially require something to be playing at all times, even if it's a pure silence track. The reliance on the value only changing on mix step is also quite brittle since this can change at any time like with the merge of #81873.

Time.get_ticks_usec() is another option as a "current time", but it's out of sync with the mix steps and thus you would lose out on any sample-accurate audio timing (the whole reason for this PR).

To me, the monotonic and audio thread synced nature of AudioServer.get_absolute_time() provides the best functionality even if the ergonomics can be a bit clunky needing to fetch it in user code. Quoting what @goatchurchprime said above:

It looks like the most common application of this feature is to play a sound on the next beat. If we were to look back in 5 years time on all the uses of this function in people's code, it might turn out that the function play_on_next_absolute_beat(beats_per_minute, skip_next_beats=0) would have been a more targeted implementation that didn't require get_absolute_time() to be exposed. Unfortunately we can't know this at this point in time.

IMO this follows best engine practices with this low-level API exposed to users that need it so they can build their own solutions.

[adamscott]

(I'm so sorry for the review delay. 🙇) I will summarize here the thread I made on the Godot Developers Chat.

First and foremost, thank you @PizzaLovers007 for your great PR. You did put a lot of work on it, and it shows!

So.

The need for a new PR

I think a new PR is needed, as a little twist is needed on how to implement the functionality. I think nobody here is doubting of the necessity to add such feature to Godot.

And @PizzaLovers007 you should totally let this one intact (just to keep the history of the code and such), and start a new one if you're still interested!

The problem with the current implementation is that it relies heavily on the AudioStreamPlayer and its underlying AudioStreamPlayback. And it introduces hacks to make it work with the new absolute time start parameter.

We need to ponder about the fact that

• AudioStreamPlayers aren't built to handle "future" playback. And I don't know how we could make it work with that in mind.
  • Like, what happens when you mix and match play_scheduled() and play() / stop()? It is not even clear conceptually.
• Except that AudioStreamPlayers are actually frontends to AudioStreamPlayback, and it's those that are actually registered in on the AudioServer.

I think we should inspire ourselves from the Web Audio API AudioScheduledSourceNode. The Web Audio API doesn't really work like our current way of handling sound, but I think it can give us a great insight on how to handle precise scheduled playback.

An AudioScheduledSourceNode only plays once. Once ended, it cannot be restarted. This is great because it limits weird edge cases. And fortunately enough, it's quite easy on resources to create new Web Audio nodes.

And the API is so small: there's nothing really you can do once it started. All you can do is to disconnect the node from it's audio destination.

An alternative way to handle scheduled play

What about AudioStreamPlaybackScheduled? Instead of adding new parameters to support internal scheduled time and such (if kept, the current PR would have needed to add a new parameter for when to "stop" the sound also (it's currently missing from the PR), we should instead have a brand new playback that is specifically made to handle such tasks.

We can keep using AudioStreamPlayers too, but we could change the function name to really hammer in that you don't "play" the stream when "scheduling".

// When `p_start == -1.0`, it would be the equivalent of "now"
// I initially thought about `AudioStreamPlayer.schedule_play()`, but I think the new name states better what it does... and especially what it returns.
Ref<AudioStreamPlaybackScheduled> AudioStreamPlayer::start_scheduled_stream_playback(double p_start = -1.0, double p_stream_offset = 0.0);

// No default here because it's an optional call. Thanks @PizzaLovers007 for the suggestion.
void AudioStreamPlaybackScheduled::set_scheduled_stop(double p_end);

This has the added bonus to give the control to the user. The user can summon any number of AudioStreamPlaybackScheduled instances, but will have to handle them.

It can be useful to summon multiple streams in advance too. Currently, in this PR, the metronome plays on-time when the process is at 1FPS, but fails a lot of time to actually play, because the main code didn't have the opportunity to queue up new ticks. As a metronome is predictable, you could easily whip up something like this:

var player_in_range: = true
var playback_instances: Array[AudioStreamPlaybackScheduled] = []
const MAX_PLAYBACK_INSTANCES: = 10

func _on_playback_end(playback: AudioStreamPlaybackScheduled) -> void:
    playback_instances.erase(playback)

func _process() -> void:
    if player_in_range:
        while playback_instances.size() < MAX_PLAYBACK_INSTANCES:
            var playback: = $player.start_scheduled_stream_playback(AudioServer.get_current_time() + playback_instances.size())
            playback.connect("playback_ended", _on_playback_end.bind(playback), CONNECT_ONE_SHOT)
            playback_instances.push_back(playback)
    else:
        for playback_instance in playback_instances:
            AudioServer.stop_playback_stream(playback_instance)

[PizzaLovers007]

Thanks for the review! I've given this some more thought after trying to implement it (WIP) as well as seeing some of the post-rework discussions in the chat.

I can agree that the scheduling logic should be moved to its own playback. However, I think my main worry about dissociating the playback from the player is that there are some settings like volume and panning that as soon as you schedule the sound, you'd need to interact with AudioServer directly. This doesn't seem too bad for the simple AudioStreamPlayer, but then as soon as you consider the panning/doppler effects from the AudioStreamPlayer2D and AudioStreamPlayer3D, it becomes very apparent that the scheduling API becomes useless there. AudioServer APIs related to playbacks are also not yet bound in GDScript (not hard to do, but is blocking).

Additionally from your example:

1. player_in_range implies that the sound trails as you get further from the audio player. This would need to be controlled maunally.
2. "playback_ended" is usually signaled from the player, which pulls its data from the AudioServer. Having the playback emit the signal would mean the AudioServer needs to tell the playback that it's done playing (no polling option since playbacks aren't nodes).

If we want to expand on the playback idea but keep it tied to the player, I came up with few alternatives:

Alternative 1: Add the playback to the player

void AudioStreamPlayer::add_playback(AudioStreamPlayback playback);

Benefits of this would be that the user is free to create any kind of playback from any stream but still reap the benefits of the player frontends. The drawback here would be that the playback itself may have nothing to do with the attached AudioStream. I'm also not sure if there would be many (or any) other AudioStreamPlayback types that would make use of this.

Alternative 2: AudioStreamScheduled

With this, the user can set the start/end time on the player, then call play() to snapshot those values. They can be changed later when retrieving them from the player.

class AudioStreamScheduled : public AudioStream {
  Ref<AudioStream> base_stream;
  uint64_t scheduled_start_frame;
  uint64_t scheduled_end_frame;
}

class AudioStreamPlaybackScheduled : public AudioStreamPlayback {
  Ref<AudioStreamPlayback> base_playback;
  uint64_t scheduled_start_frame;
  uint64_t scheduled_end_frame;
};

On the GDScript side, it'd looks something like:

var scheduled_stream = AudioStreamScheduled.new(original_stream)
player.stream = scheduled_stream

var playbacks = []
for i in range(10):
  scheduled_stream.scheduled_start_time = AudioServer.get_absolute_time() + i + 1
  player.play()  # play() snapshots the start/end settings
  playbacks.append(player.get_stream_playback())

# Settings can be changed later
var scheduled_playback = playbacks[0] as AudioStreamPlaybackScheduled
scheduled_playback.scheduled_start_time = ...
scheduled_playback.scheduled_end_time = ...

This wouldn't require any changes to player code, but the usability of this seems worse.

Alternative 2A: Single playback

Similar to alternative 2, but interacting with it would be more like AudioStreamPlaybackPolyphonic.

class AudioStreamScheduled : public AudioStream {
};

class AudioStreamPlaybackScheduled : public AudioStreamPlayback {
  struct Schedules {
    uint64_t start_frame = 0;
    uint64_t end_frame = 0;
    double from_pos = 0;
  };

  Ref<AudioStream> base_stream;
  LocalVector<Schedules> schedules;

  int add_schedule(double start_time = -1, double from_pos = 0, double end_time = -1);
  void set_start_time(int index, double start_time);
  void set_from_pos(int index, double from_pos);
  void set_end_time(int index, double end_time);
};

I'm not very convinced by any of these options though, and I think your suggestion with start_scheduled_stream_playback + associating the scheduled playback with the player is the one that makes the most sense to me. Let me know what you think!

[adamscott]

@PizzaLovers007 We discussed your proposal during the last audio meeting and I had an idea of a counter proposal. I just didn't have the time yet to do so. I'll try to do it today or in the next days.

[PizzaLovers007]

Just wanted to ping this thread! I did read the meeting notes, but I'm not sure if "forked synced" node refers to Godot nodes or Web API nodes like AudioScheduledSourceNode. I think just even a small explanation would help me get started, it doesn't need to be as in depth as your original reply 🙂

[bonjorno7]

Until this is merged, here's how I've been working around it; accurate within a millisecond.
Add a second of silence to your sample, then calculate how far into the sample you should start playing.
position here refers to a time in your music, between now and one second from now.
It looks ahead by AudioServer.get_output_latency() so that the sound effect plays in the next audio buffer.

var silence = position - music_player.get_playback_position()
sfx_player.play(clamp(1.0 - silence, 0.0, 1.0))

Oh yeah using an AudioStreamPlaylist to add silence before your sample doesn't work; perhaps you could do it with an AudioStreamGenerator, but for my use case it was easier to just modify the sample itself.
I can provide more implementation details if anyone has questions.

[adamscott]

@PizzaLovers007 So my idea would be to create ScheduledAudioStreamPlayer, ScheduledAudioStreamPlayer2D, and ScheduledAudioStreamPlayer3D nodes.

These nodes are destined to be derived from existing related AudioStreamPlayer nodes and not really destined to be created from the editor.

func _ready() -> void:
    var scheduled_player = $AudioStreamPlayer.create_scheduled()
    add_child(scheduled_player)

How does it sound?

[Mickeon]

I may be missing some pieces of discussion, but why new nodes and not new AudioStreamPlayback types (still created from the existing nodes)?

[adamscott]

I may be missing some pieces of discussion, but why new nodes and not new AudioStreamPlayback types (still created from the existing nodes)?

See @PizzaLovers007's comment:

I can agree that the scheduling logic should be moved to its own playback. However, I think my main worry about dissociating the playback from the player is that there are some settings like volume and panning that as soon as you schedule the sound, you'd need to interact with AudioServer directly. This doesn't seem too bad for the simple AudioStreamPlayer, but then as soon as you consider the panning/doppler effects from the AudioStreamPlayer2D and AudioStreamPlayer3D, it becomes very apparent that the scheduling API becomes useless there. AudioServer APIs related to playbacks are also not yet bound in GDScript (not hard to do, but is blocking).<

(emphasis is mine)

[PizzaLovers007]

@PizzaLovers007 So my idea would be to create ScheduledAudioStreamPlayer, ScheduledAudioStreamPlayer2D, and ScheduledAudioStreamPlayer3D nodes.

These nodes are destined to be derived from existing related AudioStreamPlayer nodes and not really destined to be created from the editor.

func _ready() -> void:
    var scheduled_player = $AudioStreamPlayer.create_scheduled()
    add_child(scheduled_player)

How does it sound?

Making new nodes sounds good, though I'm not sure about the player.create_scheduled() aspect of this API. From a usability perspective, I imagine users would never use both versions at the same time on the same player instance, so it would make sense for the user to pick ScheduledAudioStreamPlayer in the editor. (Is it possible to hide specific node types in the Create New Node dialog?)

If we want to keep this behavior, I think a "copy" constructor would work better since that would break the circular dependency from the inheritance. So something like ScheduledAudioStreamPlayer.new($AudioStreamPlayer).

Is there a stronger reason to not make ScheduledAudioStreamPlayer and friends available in the editor?

[PizzaLovers007]

I've rewritten the PR based on the discussions here, and will close this in favor of #107226.