Wrapping Up

[00:00:00]
>> Thanks for listening, thanks for tuning in, happy to take any questions. And if you're on this repo and you see any issues with the repo or you have any particular problems or questions, feel free to pop an issue open and if I have some time, I'll try and get to you there.

[00:00:15]
>> What kind of limitations have you met with Web Audio?
>> So there's a lot of, I mean, it's been getting a lot better in the last couple years. But when I started doing Web Audio, there was a lot of different browser problems across different browsers. So Safari wouldn't support certain things, and mobile phones wouldn't support certain things, and it was this nightmare to try and manage all these different browser incompatibilities.

[00:00:41]
It's gotten a lot better though and recently, I'd say it's starting to get to a point where you just are able to code it in the way that it shows on the MDN website. And you don't have to worry too much about incompatibilities. So that's one thing just to be aware of if you are trying to support, let's say older browsers, or older phones.

[00:01:04]
Other problems sometimes have to do with things like popping and clicking, that's always been a bit of a weird thing where certain devices they're just not able to handle dozens of SINTs overlapping with effects and reverbs. It starts to slowed down, especially if you're doing that at the same time as really pushing the browser in terms of maybe the GPU and the CPU.

[00:01:28]
And you're just really maxing out the capabilities of the browser and the device. Sometimes you might start to get some issues in terms of lag or in terms of popping and clicking that's happening in the Web Audio. And then also as much as it's, there's a lot of resources and there's a lot of information, a lot of libraries, it's still kind of a complex beast.

[00:01:58]
There's still a lot to unpack, because it's an API that's really powerful and able to do a lot of amazing things. But it's also an API that is sometimes so low level and requires you to do so much more on top of it in order to achieve certain things.

[00:02:19]
So I can't think of any really specific examples, but in the past I've had to do some pretty wonky things with decoding WAV files because Web Audio API for some reason didn't support a certain type of WAV file or something like that. Or potentially having to do some weird stuff where like timing, because the Web Audio API doesn't always give you like a super easy set timeout type of function, so you have to really build out these complex schedule and things like that.

[00:02:57]
So, yeah, that's one thing, I guess, is that it's low level and it takes a bit of time to build on top of it and that's why these libraries like Tone.js are helpful. But they also have very specific goals in mind that maybe won't apply very well for your application or your game.

[00:03:18]
>> What about kick detection or beat detection?
>> Yeah, so there's some libraries that do this kind of work very specifically and they have some really smart algorithms for how to detect the kick drum in any arbitrary track. So kind of like that problem that came up earlier is like, how do you do visualization when you wanna support any type of audio instead of just the audio that's always within a specific range?

[00:03:49]
And so these beat detection libraries, they tend to use these complex algorithms that use a lot of math and it's beyond what I know how to do kind of in this kind of, fast Fourier transform type of math is a little bit above me. But it's pretty powerful and you can use those to augment your visualizers.

[00:04:13]
There are some libraries that are really designed for specifically doing visualization on something like an electronic track that supports kick drum detection and things like that that are beyond what I've been sort of talking about today. So yeah, it's a useful thing to look into.
>> Are there any tools with helping combining audio, visuals, and mobile device gyroscope?

[00:04:46]
>> Well, if you have a mobile device gyroscope APIs support, if it's enabled on the phone and you can use the gyroscope, then you can kinda do what we've been doing today in the course. Where you have maybe filters or effects and the parameters of those filters are changing based on the value of the gyroscope.

[00:05:04]
So you could do something like that and it could be quite fun where maybe depending on how the phone is being positioned, it's changing the audio. So that wouldn't really require its own library. You could do it already just with Tone.js, you might have to just google search gyroscope JavaScript and find some of the details around how do you access this.

[00:05:29]
And it would be something like that.
>> Have you experimented with using WebSockets along with audio to sync users in real time?
>> No, I haven't tried using WebSockets to sync audio. I imagine, actually I might have, I wasn't really directly part of this project. But there was a project many years ago at an agency I was working at where there needed to be a sync where one person would push a play button and many devices would play the audio.

[00:06:06]
And I think for the most part with WebSockets it was okay. But I think because audio is so specific, that if it's a fraction of a second out of sync, your brain can really, really detect the problem, your brain is really good at detecting small flaws with these kinda things.

[00:06:26]
And so I think there was some challenges with getting that to work perfectly because if even one was out of sync to the other just a little bit, it wouldn't sound great. And I think what they ended up doing was having to sort of forge it a little bit by making it so that even though you say play it at one millisecond from now.

[00:06:48]
The phone would actually say okay, I'm gonna play it a little bit earlier than that or something like that, just to make sure that it's playing at the perfect moment. I think generally with the web, one of the things that's lacking in not just audio but also visual, video sort of playback, is really fine grained, very accurate level controls of media playback.

[00:07:13]
So for example, saying I want to play this MP4 or MP3, whether it's video or audio or both, I wanna play it from a specific timestamp that is down to the exact frame that I wanna play or something like that. The web's not so good at those kind of exact playbacks, unfortunately.

[00:07:33]
But it probably is possible at a basic level and you might be able to create some really cool projects out of it still.
>> Are there ways to route signals to and from outside of the browser for external heavy processing?
>> So that's something I haven't explored too much of.

[00:07:54]
I mean, typically what we would do if we're doing some sort of installation or something that's using Web Audio, is it would just be using the output device of the computer or laptop. I mean, basically from the browser to the output device, and then sending that along to something that might be totally unrelated to the browser.

[00:08:18]
Maybe even it could be some sort of visual output like sending the signal to a physical device, like a physical RGB strip or something like that. So it's still coming from the browser and you're just using different physical plugs and different sort of interfaces that are physical audio interfaces to reroute that to different places.

[00:08:43]
Where you might be able to explore as well, and this is a different path that I haven't really looked into, is WebRTC has been coming a long way. And there may be ways of piping the signal directly from an audio node into a WebRTC stream of some kind, which may allow you to stream audio to appear in a peer-to-peer kind of situation.

[00:09:12]
And one other thing, I guess, worth mentioning as well is offline audio context. So, so far all the audio that we've been exploring is real time playback of audio for the purpose of you're hearing it, you're interacting with it, you're able to play with the audio and hear it at the same time.

[00:09:35]
But if you wanted to do something like, let's say you wanted to build a MP3 converter or something like that or you wanted to build a tool like Audacity or like Ableton. Or some sort of audio tool, a digital audio workstation, and you wanted to render out an MP3 file or a WAV file with some audio nodes and using the same structure that we've been exploring.

[00:09:58]
And so you don't want it to be in real time cuz you don't want it to just you have to wait for a whole two minutes, you want it to run faster than real time or as fast as it possibly can. And you don't really need it to go through the speakers because it's just running internally to convert, let's say one format to another or to add effects to something and then produce a file.

[00:10:20]
You can use this OfflineAudioContext, which is only really in new browsers, like Chrome and things, but it's quite useful for that.
>> There's a more general question from online, how do you go about finding inspiration for creating the things that you do?
>> Yeah, so a lot of the inspiration that I find is actually through online, through Twitter, through Instagram, following other artists and creators, and sometimes game developers.

[00:10:48]
Or sometimes people that are just making these kinds of audio visual experiences. It took me a while to sort of build up this this group of people that I've been following that seem to always be producing really amazing work. Or at least every time I look at my Twitter and Instagram feeds, I'm just constantly blown away.

[00:11:10]
And I guess that's because of many years of searching and following and then looking at who people are following and who they're resharing and then finding more through them. So that's probably the main source of most of my inspiration these days. But also especially as things are beginning to open up a little bit, I kind of realized how much I missed being at events in person and finding these kind of events that are maybe art exhibitions or music shows or something like that.

[00:11:45]
That's a little bit off the beat and track, an example of that is Ryoji Ikeda, I'm not sure if he's still, spell his name, Ryoji, there it is, not sure if he's still doing his show in London. He might be, 180 The Strand, but this is a good example, is I heard about this show in London and it's really the kinda thing where you look at the site and it looks kinda interesting.

[00:12:16]
It seems like they might have changed from his to a different exhibition, but you go to this and suddenly you're kind of presented with an exhibition that really blows you away. And I think just being at these kinds of events, you are really inspired more than you really can be in some ways just by browsing your phone cuz it just transports you to another world almost.

[00:12:43]
And so, yeah, I guess going to actual exhibitions and trying to actively seek these installations and trying to find out how to actually visit them in person. That's been pretty transformative, and just being able to access that lately has been really nice. So I managed to go see this one and this was a couple months back.

[00:13:07]
And you leave the exhibition with your ears ringing in your eyes kind of a little blurry because it was just so much raw visuals and noise. But you're kinda leaving as well inspired to do more things and also to bring the work back into this physical space, rather than just keep it purely digital.

[00:13:29]
So, yeah, mix of Instagram, Twitter and then also going for art exhibitions and galleries and museums and things.