Web Audio Synthesis & Visualization

Matt DesLauriers introduces the course by providing an overview of the course material, a walkthrough of the course repo, and some prerequisites. The demos provided in this segment involve mp3, buffered mp3, gain, waveform, meter, and frequency.

Matt discusses the basics of how audio and digital audio works including a description of sound, waveforms, and frequency. A closer look into what a waveform represents is also covered in this segment.

Matt walks through some artistic examples that involve the use of web audio. Personal projects with web audio including art, audio visualization, a game, and work from other developers are provided to showcase web audio's many applications.

Matt provides a brief overview of what the Web Audio API is, a graph of how audio data flows from input to output, and how the API works.

Matt walks through the JavaScript being used to stream an mp3 file through the browser overtime using the Web Audio API. Streaming audio is a better option for audio files with a longer track length to allow the file to be played as it downloads instead of waiting for the file to be completely loaded in. A brief walkthrough of a typical course sketch is also provided in this segment.

Matt discusses buffering an audio file to fully download and decode the file to allow the audio to immediately be played. To help avoid errors from edge cases a best practice is to make sure the audioContext is resumed before playing audio.

Matt demonstrates passing an AudioNode through a second GainNode before the end AudioDestinationNode using the createGain function to allow for simple adjustments to the audio's volume. Gain values should be adjusted gradually to avoid pops and clicks in the audio.

Matt discusses mapping a waveform into a digital space with arrays and JavaScript using an analyserNode and the Web Audio API function getFloatTimeDomainData. The getFloatTimeDomainData takes in a dataArray which designates the window of time ahead of the playhead to be analyzed.

Matt walks through the code being used in the waveform demo to visualize a waveform including creating a Float32Array to store the waveform data. A brief discussion on the advanced waveform demo, which builds on the first example with extra features and browser support, is also provided in this segment.

Matt demonstrates using the waveform data to create a visualization based on the volume of the audio track. The advanced meter demo uses the Web Audio API to help split the audio track into different colored frequency bands to help create a more interesting visualization. A brief discussion regarding the BiquadFilterNode is also covered in this segment.

Matt discusses what it means to analyze the frequency, provides multiple visualizations of the frequency spectrum, and discusses how frequencies can be used. A demonstration of using the tool spectrum to visualize the frequency graph and display a chosen decibel and hertz value of any audio file passed to it is also provided in this segment.

Matt walks through the code provided in the frequency demos and discusses how to use frequencyToIndex, indexToFrequency, and audioSignal to help visualize parts of an audio track without using a the biquad filter. A student's question regarding if fft is the only frequency analysis supported by the Web Audio API is also covered in this segment..

Matt discusses he advanced filter demo which builds on the previous frequency demo by using the frequency to map to a generative artwork. Student questions regarding how to adjust the fixed parameters for a live audio source and how to treat files at different bit depths for dB FS are also covered in this section.

Matt discusses synthesizing audio from code without using any source files and demonstrates some examples of synthesizing audio with code. Examples of using Tone.js for interactive and procedural sound is also covered in this segment.

Matt discusses the open source JavaScript library Tone.js, how the base code is laid out for creating a Tone.synth, and provides the API documentation. A walkthrough of each part of code involved in synthesis in the tone demo and a demonstration of a game that uses a procedurally generated soundtrack is also provided in this segment.

Matt discusses the various effects included in Tone.js such as FeedbackDelay and walks through how to apply those effects to an audio track. A walkthrough of a demonstration using FeedbackDelay, Reverb, Filter, and editing the envelope is also covered in this segment.

Matt discusses creating a sequencer using PolySynth to create a richer sound with multiple oscillators, Transport for timing events, and a randomizeSequencer function to randomize the played notes. A walkthrough of the code used to create the sequencer is also provided in this segment.

Matt walks through the various resources provided in the learning synths playground by Ableton including some material covered in this course such as amplitude, pitch, and effects. A demonstration of the synth recipes section, which provides settings for some common synth sounds, is also provided in this segment.

Matt provides some inspiration for future web audio projects by providing various artists works that involve or could possibly be made with web audio.

Matt discusses some links and provides a section on the course repository for further reading about web audio including GitHub pages that collect web audio resources, math, learning audio, fun web audio sites, and tools.

Matt wraps up the course by providing well wishes on continued learning of web audio and answering student questions regarding web audio limitations, kick or beat detection, and tools to combine audio and a mobile device's gyroscope. Student questions regarding using web sockets along with audio to sync users in real time, routing signals to and from outside of the browser, and how to find inspiration for future projects.