Add sample playback support

[adamscott]

⚠️ Here be dragons ⚠️

When this PR will be merged for 4.3, this feature will be marked as experimental and could change in 4.4.

tl;dr

This PR adds (back) the concept of samples to the Godot Engine. Currently, the PR enables only the web platform to play samples.

It principally fixes #87329, as that issue would plague any non-threaded web releases with crackling audio.

Example

Single-threaded web example using streams (old way)	Single-threaded web example using samples (this PR)
https://adamscott.github.io/2d-platformer-demo-main-thread/	https://adamscott.github.io/2d-platformer-demo-main-thread-samples/

Introduction

Godot uses streaming to mix game audio. Each active stream is registered and then the engine mix on-the-fly the needed audio frames together to output audio based on the audio latency parameter. It works very well on modern platforms.

Samples are another way to handle sound instead of mixing streams. Instead of handling mixing sound and music by the game processes, it relies on off-loading it to the host system. While it doesn't permit full access to the mixing apparatus, it's super useful on systems that don't have a lot of processing power.

To use samples, you register a sample, and then tell the system to play it when needed. And to stop it. It's like a music player, you set the file, then you click on play. You don't control how the software do it, but you know it does.

Godot used to have samples back in Godot 1 and 2, especially to support platforms like the PSP, and the web (thanks to the Web Audio API).

As newer console platforms let developers handle their own mixing logic and that SharedArrayBuffers were introduced in browsers (permitting WebWorkers (web threads) to share memory with the game), samples support was dropped from Godot. Everything was fine.

Anyway, the implementation was somewhat lacking. You had to specifically want to play samples, you couldn't use common nodes to play both streams and samples.

The problem

But on the web platform, Spectre and Meltdown happened. And it completely changed where SharedArrayBuffers were able to be used. Enter "cross-origin isolated" websites, where it's impossible to contact other websites or display ads, and complicating hosting of simple games, greatly reducing the appeal for our web builds.

Hence the work on #85939 in order to compile Godot to run on the main thread. This enables exporting Godot games on the browser without having to cross-origin isolate your website. Unfortunately, this brought an unexpected issue: software mixing is pretty much incompatible with single-threaded games. Especially running on older/less powerful hardware.

Wanna hear for yourself? Try the single-threaded platformer demo (without this PR applied) on your phone or on a computer that doesn't have a great CPU.

The investigation

My colleague @Faless and I considered every solution imaginable: augmenting latency for the web, traced the processes on the web and on mobile and refactor the AudioWorklet processing the audio. But alas. Nothing substantial could have been done.

The only solution we found was to resort to Web Audio samples.

And it's not uncommon for web game engines. We were the uncommon ones not using web audio samples. So, a few weeks ago, I began work on this PR.

The sole requirement: seamlessness

My main focus was to reuse as possible as many features that already exist. It means that in order to play samples, I wanted the UX to keep as close as possible to existing tools.

Godot strives itself to offer the same experience for every target that it exports to. Imagine making the developer choose between having samples for the web export and streaming nodes for the rest. And having to manually add or remove nodes based on the platform with scripts.

This has such a big impact that it's a clear no go for us. We don't want that poor UX.

The solution

My solution is a big hack. (But it does works wonderfully.)

The idea is to reuse all the existing stream nodes and systems. And make the stream elements capable of producing samples.

This story begins with the new project setting audio/general/default_playback_type (hidden currently in the advanced options). Usually, it should stay with the value "Stream", as normally, that's how Godot works currently. But the magic happens with audio/general/default_playback_type.web set as "Sample".

That's because AudioStreamPlayer, AudioStreamPlayer2D and AudioStreamPlayer3D now have a new property called playback_type, which is set by default to... "Default". That's where the magic happens! On standard exports, the nodes will be defined as "Stream", but on web exports, "Sample" will be used instead!

The magic operates behind the scenes though.

The man behind the curtain

Essentially, when a stream is considered a "sample", it doesn't get mixed at all in the mixing phase. Instead, it relies on callbacks by the StreamPlayer nodes.

The StreamPlayer nodes, when their play() method is called, are calling internally AudioServer::start_sample_playback(). All the AudioServer does is to call AudioDriver::start_sample_playback(). If the driver doesn't implement that function, it just doesn't play any sound. But if it does, the driver can now tell the backend to play that sound.

The same thing happens for stop, pause, etc. You can even update the sample, like when the position of the node changes!

Isn't this fascinating?

Registering samples

Before playing the samples, it's important to register them first.

If played without previous registration, the player will make sure to register it first. Though, it's recommended to register manually streams. That's because, on single threaded games, memory transfer is synchronous, so it may make your game stutter. You register a stream as a sample by calling this method:

# optional step
const my_stream_resource = preload("res://assets/my_stream.wav")

AudioServer.register_stream_as_sample(my_stream_resource)

Under the hood, Godot will call the mix() method of the stream playback for the entire duration of the clip. This makes it so that it's possible to play any type of sound media that Godot supports (WAV, mp3, ogg vorbis).

Is it really seamless, though?

These demos were exported to the web (single-threaded) using samples without ever touching the project nodes, resources, nor files.

Demo title	Playable link	Source
2D Platformer	Play	Link
Dodge the Creeps	Play	Link
3D Platformer	Play	Link
Truck Town	Play	Link
Hell of Mirrors	Play	Link
Catburglar	Play	Link

Bugs yet to fix before merge

• Buses don't chain properly
• Only forward loop is supported (fix may not make it to the final release)
• Autoplay doesn't work right now.

Known limitations

• Effects don't apply (will certainly not be part of the initial release and GDScript based effects cannot be used)

Technical diagrams

Registering and playing samples

sequenceDiagram
    participant Script
    participant AudioStreamPlayer2D
    participant AudioStreamPlayerInternal
    participant AudioServer
    participant AudioDriverWeb
    participant JavaScriptAudioLibrary

    Script ->> AudioStreamPlayer2D: set_stream(Ref<AudioStream>)
    AudioStreamPlayer2D ->> AudioStreamPlayerInternal: set_stream(Ref<AudioStream>)

    Script ->> AudioServer: register_stream_as_sample(Ref<AudioStream>)
    activate AudioServer
    AudioServer ->> AudioServer: register_sample(Ref<AudioSample>)
    AudioServer ->> AudioDriverWeb: register_sample(Ref<AudioSample>)
    deactivate AudioServer
    AudioDriverWeb ->> JavaScriptAudioLibrary: godot_audio_sample_register_stream()

    Script ->> AudioStreamPlayer2D: play()
    activate AudioStreamPlayer2D
    AudioStreamPlayer2D ->> AudioStreamPlayerInternal: play_basic()
    activate AudioStreamPlayerInternal
    opt _is_sample() && stream->can_be_sampled() && stream_playback->sample_playback is null
        AudioStreamPlayerInternal ->> AudioServer: is_stream_registered_as_sample(Ref<AudioStream>)
        activate AudioServer
        deactivate AudioServer
        AudioServer ->> AudioStreamPlayerInternal: 
        opt stream is not registered
            AudioStreamPlayerInternal ->> AudioServer: register_stream_as_sample(Ref<AudioStream>)
            activate AudioServer
            AudioServer ->> AudioServer: register_sample(Ref<AudioSample>)
            AudioServer ->> AudioDriverWeb: register_sample(Ref<AudioSample>)
            deactivate AudioServer
            AudioDriverWeb ->> JavaScriptAudioLibrary: godot_audio_sample_register_stream()
        end
        AudioStreamPlayerInternal ->> AudioStreamPlayerInternal: set stream_playback->sample_playback
    end
    AudioStreamPlayerInternal ->> AudioStreamPlayer2D: Ref<AudioStreamPlayback>
    
    deactivate AudioStreamPlayerInternal

    opt stream playback sample exists
        AudioStreamPlayer2D ->> AudioStreamPlayer2D: Update stream_playback->sample_playback for specific AudioStreamPlayer2D stuff
        AudioStreamPlayer2D ->> AudioServer: start_sample_playback(Ref<AudioSamplePlayback>)
        deactivate AudioStreamPlayer2D

        AudioServer ->> AudioDriverWeb: start_sample_playback(Ref<AudioSamplePlayback>)
        AudioDriverWeb ->> JavaScriptAudioLibrary: godot_audio_sample_start()
    end

Samples and streams

classDiagram
    namespace AudioStreams {
        class AudioStream
        class AudioStreamWAV
        class AudioStreamPlayback
        class AudioStreamPlaybackWAV
    }
    namespace AudioSamples {
        class AudioSample
        class AudioSamplePlayback
    }

    %% Inheritance
    AudioStream <|-- AudioStreamWAV
    AudioStreamPlayback <|-- AudioStreamPlaybackWAV

    %% Links
    AudioStream ..|> AudioSample
    AudioStreamWAV ..|> AudioSample

    AudioStreamPlayback ..|> AudioSamplePlayback

    %% Cardinality
    AudioStreamPlaybackWAV "1" --> "many" AudioSamplePlayback

    AudioSample "1" --> "many" AudioStream
    AudioSamplePlayback "1" --> "many" AudioStream

    %% Classes
    class AudioStream {
        +can_be_sampled()*
        +get_sample()
    }

    class AudioStreamWAV {
        +can_be_sampled()
        +get_sample()
    }

    class AudioStreamPlayback {
        +set_is_sample()*
        +get_is_sample()*
        +set_sample_playback()*
        +get_sample_playback()*
    }

    class AudioStreamPlaybackWAV {
        -bool _is_sample
	    -Ref~AudioSamplePlayback~ sample_playback

        +set_is_sample()
        +get_is_sample()
        +set_sample_playback()
        +get_sample_playback()
    }

    class AudioSample {
        +Ref~AudioStream~stream
        +data
        +num_channels
        +sample_rate
        +loop_mode
        +loop_begin
        +loop_end
    }

    class AudioSamplePlayback {
        +Ref~AudioStream~stream
        +float offset
        +float volume_db
        +PositionMode position_mode
        +Vector3 position
        +StringName bus
    }

Fixes

Fixes #87329

[adamscott]

When testing, I always get an exception in the console.

Bus.count; -> return GodotAudio.Bus._buses.length; -> GodotAudio.Bus is null?

tmp_js_export.js:10089 Uncaught (in promise) TypeError: Cannot read properties of null (reading '_buses')
    at Function.get (tmp_js_export.js:10089:29)
    at Object.GodotAudioClassesInit [as classesInit] (tmp_js_export.js:10116:8)
    at tmp_js_export.js:14275:16
    at tmp_js_export.js:16297:8

I don't do anything crazy with the audio buses:

@Maran23 Can you reproduce the bug with my latest changes? If so, can you join a minimal reproducible project?

[adamscott]

@Faless I changed the "language in" to UNSTABLE in order to support static class methods. Otherwise, it doesn't compile.

/var/folders/_5/6_c63ny97m554cs3yqq3djxh0000gn/T/emscripten_temp_u8l09o95/godot.web.template_debug.dev.wasm32.nothreads.js:9645:4: ERROR - [JSC_LANGUAGE_FEATURE] This language feature is only supported for UNSTABLE mode or better: Public class fields.
  9645|                 static _samples = new Map();
                        ^

/var/folders/_5/6_c63ny97m554cs3yqq3djxh0000gn/T/emscripten_temp_u8l09o95/godot.web.template_debug.dev.wasm32.nothreads.js:9854:4: ERROR - [JSC_LANGUAGE_FEATURE] This language feature is only supported for UNSTABLE mode or better: Public class fields.
  9854|                 static _sampleNodes = new Map();
                        ^

/var/folders/_5/6_c63ny97m554cs3yqq3djxh0000gn/T/emscripten_temp_u8l09o95/godot.web.template_debug.dev.wasm32.nothreads.js:10064:4: ERROR - [JSC_LANGUAGE_FEATURE] This language feature is only supported for UNSTABLE mode or better: Public class fields.
  10064|                static _buses = [];
                        ^

/var/folders/_5/6_c63ny97m554cs3yqq3djxh0000gn/T/emscripten_temp_u8l09o95/godot.web.template_debug.dev.wasm32.nothreads.js:10066:4: ERROR - [JSC_LANGUAGE_FEATURE] This language feature is only supported for UNSTABLE mode or better: Public class fields.
  10066|                static _busSolo = null;
                        ^

[Faless]

UNSTABLE doesn't seem something we should target in stable release though.

Do we really need all those? None of the other libraries needs to use the static keyword. Why not use the usual:

const Lib = {
  $Lib: {
    myProp: 1
  },
  godot_js_get_my_prop() {
    return Lib.myProp;
  },
};

mergeInto(LibraryManager.library, Lib);

[Faless]

Was this file committed on purpose? I don't understand why we need type definitions for interoperability with TypeScript for an internal library.