Visualizing Audio Overview

[00:00:00]
>> So we've talked a little bit about playing audio just mp3 files and then buffering it decoding it and then playing it with low latency. Now let's talk a little bit about visualizing that audio, how do we take the audio data and split the nodes up so that we have some nodes that are playing the audio through the speakers.

[00:00:20]
But then another node that's analyzing the audio as it's playing and maybe then we can start to do some interesting data visualization with it some sort of generative piece. Maybe something just like music visualizer whatever it is you really wanna do, so to understand how to do this.

[00:00:37]
First we have to understand how do we think of this waveform in a way that is more digital rather than just being this graphing of the wave going up and down. How do we map this into a digital space with arrays in JavaScript and things like that? So what we're gonna use is a Web Audio API function called getfloatTimeDomainData very long wordy function.

[00:01:03]
And that's from the analyserNode which I'll show you how to set up in a second. And what that does is it gets us back the data of the waveform as it's playing based on whatever it is that's playing through that analyserNode. So remember here how I was talking about waveforms earlier what it's going to give us are these little sample points where if you zoomed really far in those little dots on the graph.

[00:01:29]
Are the values that are gonna come through this function and be put into an array and so over time if we were to store all of the little samples that it gives us from this function. Eventually we'll be able to draw the same graph that you see here zoomed out.

[00:01:47]
But because we're streaming this audio in and because we're playing the audio and because we can't just save every single sample all at once into a big array that would be this massive array. What we tend to do in audio sort of programming is we tend to think about it in terms of a window so we have a playhead which is where the audio is playing.

[00:02:06]
And then there's a window which is some slice of time that's ahead of the playhead, and it's already loaded that audio into memory and it's getting ready to play it. And because it's already there in memory we can start to analyze it so that's kind of how this works is that.

[00:02:22]
It just takes a little window a slice of the audio as is playing rather than trying to take all of the data from the entire audio track. So just to see it in another way let's imagine for example that we're going from the 35th millisecond to the 36 millisecond in an audio track.

[00:02:38]
I mean those are arbitrary timestamps it might actually be much smaller slices of time or bigger depending on your settings. But then what we're gonna get from this API function we're just gonna get the data within this little window here the little data points of the wave and of the waveform.

[00:02:56]
So here's how it looks in an array when we call this function we pass in the array that we want to receive the data. So we're just reusing the same array every time we would call this function and the array has the index of zero starting. This is the zero element of the array and then it goes all the way to whatever the maximum index of the array is.

[00:03:21]
We can set this to be a bigger array if we wanna have a bigger window and we wanna have more data, or we can do a smaller array if we don't need that much data. And inside each element of the array is the actual signal so if I was just to go back to this.

[00:03:36]
Remember I talked about how this graph is always positive 1 to negative 1 it's always normalized between those two values. So these float values are always between negative one and positive one. And here on the left, this first index represents the start of the window. So this line here, and this last index, whatever the value is here represents whatever the value point in the sample point is at the end of the window there.

[00:04:08]
So every time we call this function as the audio is playing we're probably gonna get a different waveform because it's always gonna just be a single little slice of audio.