Frequency Analysis Demo

[00:00:00]
>> Now let's jump into some of the code and just see how we can use this frequency analysis. And try to use it in a way that allows us to visualize different parts of the track without having to rely on something like a biquad filter. So in our demos, I have two here that have to do with frequency analysis, specifically in a way that is kind of manual.

[00:00:31]
Just analyzing the frequency and then acting on that data. The first one just isolates the signal and gives us different ranges of frequency. So here we have an audio file that is really strong in terms of the kick drums and in terms of the low frequency bass sounds.

[00:00:51]
And from that, we can start to visualize a single circle, and then using the higher mid tones, the voice, we can use that to map those signals to another circle.

[00:01:02]
[MUSIC]

[00:01:09]
And then the second demo, which is really just me sort of trying to go further with this. It's not really something I'm gonna dissect today, but it's something that you might want to look into how it all works, is using frequency to sort of map that to something like a generative artwork almost.

[00:01:25]
This is based on Albers, so Josef Albers artist, and who created this a lot of work around color and around these primitive shapes. And basically, just taking those ideas and seeing how do we reinterpret his work through the lens of an audio visualization of some kind.

[00:01:43]
[MUSIC]

[00:01:50]
It's the kind of work actually which in such a small window, it doesn't look like much. But if you were to open it up full screen and play it through projector, it could be a potentially powerful audio visualization in a physical experience. So that's kind of what the end goal of this is, is not just to create something that looks good in this little tiny thumbnail here or runs just on your local host.

[00:02:15]
But maybe create something that you can share and create a sort of emotional experience for the audience. So let's dive into the frequency example just to start.

[00:02:30]
[MUSIC]

[00:02:33]
So this frequency analysis demo is very similar to some of the looping streaming audio demo that we started with, except it's using a different audio files, using this blue jeans.mp3, which is sort of an electronic track. And then down here, we're setting a few parameters on the analyzer node, we're setting fftSize to be a little bit higher because we want to capture more resolution.

[00:03:02]
So a larger number here should give us more data to work with, especially in the low end, where there's not a lot of data compared to the rest of the graph. And because there's so much kick drums here, we really wanna capture the detail. We're gonna just use a larger value.

[00:03:19]
Just make sure whenever you're setting the fftSize, you have to use a power of two number. So it has to be 1024, 2048, 4096, etc. And then here, we're changing the minimum and maximum decibels that we're expected to be working with. So actually, I've just set them now to the default values, which is the same as what Web Audio API gives us.

[00:03:46]
But you might find if you're using a different audio source, you might want to tweak these values, because maybe your audio source is never getting this loud. And so you're losing some of that information if you're just using the default values. And then we set up our float array as normal.

[00:04:07]
We have a couple functions here that are new. And they're a little bit sort of mathy in the sense that they have to use this nyquist value, which as I mentioned briefly earlier, is half of the sample rate of the audio context. The sample rate is how many samples there are per second.

[00:04:30]
So if you remember one of my slides was zooming in on the waveform and seeing those little dots. Well, imagine in a single second of time, there's gonna be 44,100 of those dots. That's the sample rate. And each of those dots represents a different sample of, let's say, the waveform data, that we then gonna convert into this frequency using the FFT, the Fast Fourier Transform.

[00:04:55]
But again, this is usually set for us. We don't need to worry about what the sample rate is. We just divide it by 2 to get this nyquist. And then we use this math here to figure out the actual index based on a frequency band. You don't really need to memorize the inside of this function or the math involved.

[00:05:14]
All you really need to know is that this function allows you to pass in hertz. So like we were saying in the spectrum analysis, something like 200 hertz or maybe 440 hertz if you're trying to find the index in the array of that C node. And then it returns the index of the array between zero and whatever it is that the array has been defined as.

[00:05:36]
So like up here, we're using fftSize as the array length. And then the other way around is also possible, just taking an index somewhere in the array, and figuring out what is the frequency band for that index. And this third function here, and again, these functions are all in the snippets.

[00:05:57]
So if you want to just copy and paste, that's fine. Quite often what I do with this is I just write it once, and I copy and paste it or I install it from NPM or something like that. So I don't really always remember what's inside this function, but inside this function here, audio signal.

[00:06:14]
The idea is that it gives us back the signal between 0 and 1, based on a specific band of frequency. So we pass in the analyzer node. We pass in the frequencies data which is the float 32 array. Pass in the minimum and maximum hertz that we want to get the signal between.

[00:06:36]
So this is very similar to this slide where we pass in the minimum which is the left end of the band, and the maximum, which is the right end of the band. And then we're just gonna compute the average signal of all of the data in between those two points.

[00:06:59]
And now, the way that works is we take the frequency, convert it to an index in the array. And then we look through and get the actual average from that. And this average is in decimals, there may be other ways of doing this rather than just taking an average, but maybe some sort of root mean squared type of thing that might be superior.

[00:07:21]
This is just the way I've been doing it and it seems to work okay, it's definitely worth maybe exploring that a bit further. And it's one of the reasons I think that using the biquad filter is actually maybe a little bit better, because the math that they use in terms of the windowing isn't just a hard start and a hard stop.

[00:07:40]
It's something more of a fall off, and so it's a bit smoother and a little bit better. But this is nice just to sorta get an idea of what's going on under the hood. So we take the average, and then we map that average from the minimum to the maximum decibels.

[00:07:55]
So the frequency data that we get is in decibels, but we wanna convert it to a 0 to 1 value, and that's what we're doing here. And so from that, you just put those things together. You have your analyzerNode.getfloat frequency data, which is different than get float time domain data, pass it into the array.

[00:08:16]
And then we use this function audio signal, and we pass in our minimum and maximum hertz. And it gives us back some signal between 0 and 1 that we can then scale.

[00:08:28]
[MUSIC]

[00:08:36]
Just as a little example of how to do this with a different track, let's say all of a sudden, I wanted to visualize something that has much different shape, much different signal. Let's say I wanted to visualize a piano. You'll see that the yellow circle is no longer really changing.

[00:08:57]
And that's because we're looking at different frequencies. So, let's say we open up our spectrum which is in the bottom of the repo by the way. Spectrum.sh, but you can also find it if you scroll all the way down, Tool, Spectrum Analyzer. And so let's open this up and we'll drag in our piano.

[00:09:18]
Is FFT the only frequency analysis supported by the Web Audio API? With the Web Audio API, it gives you this FFT out of the box, but there's not really any other way to analyze the frequency than that. But if you wanted to do something based on the frequency, you could use the biquad filters to sort of shape the signal.

[00:09:41]
So that it's only giving you some signal that's within a certain frequency band or that it's only giving you the top half of the frequencies or something like that. So that's how you can use frequency analysis without having to rely on actually getting the frequency data and then doing things with the data samples themselves.

[00:10:02]
Okay, so let's pop open this spectrum analyzer and just drag in your mp3 file or your WAV file that you want to analyze. And we're gonna listen to it and look at the graph as it's playing.

[00:10:18]
[MUSIC]

[00:10:20]
And again, we'll see that if we use our mouse, we can see where the signal's happening. And it's happening between 120 hertz and maybe 2.5k. And so let's just use that for now as, As the input here. So, I'm just gonna go back to my demo here. And I'm gonna just use just one circle for now.

[00:11:03]
Let's just say 150 hertz to 2,500. And you can see that now it's very clearly picking up only the piano nodes.

[00:11:14]
[MUSIC]

[00:11:18]
And we could try something else where maybe we're only picking up let's say, the very low piano nodes.

[00:11:25]
[MUSIC]

[00:11:40]
So now, the yellow circle's representing the low tones which are fairly constantly there despite what node is playing. Because there's always this sort of low vibration that's happening whenever the piano's getting struck. But when the piano's getting struck, there's also a higher frequency sound that is specific to the node that's being played.

[00:12:06]
And so that's how we can start to create these interesting visualizations based on these different audio sources and different types of frequencies.