Web Audio & Web MIDI

[00:00:00]
>> Well, what else? We have Web Audio. Web Audio is a low level API that it's been for a while, that will let you generate dynamic audio on the standard output. So you need to be some kind of an audio engineer to understand actually what's going on inside this audio, but you can use this to generate dynamic audio to create a spatial 3D audio.

[00:00:26]
And it's even useful for, this is an idea, right? For ultrasound communication with devices. This is just a fun idea. Not really for something that you will do on your next website, but I have a demo of that. So, the idea of Web Audio is a low level API in JavaScript where you can work with the audio wave, both for receiving audio from the output, and to generate audio to the speaker, okay?

[00:00:55]
So very low level. There is a library known as Sonic socket, which I think is pretty cool, where you can define your alphabet, so which characters are you going to use? And then you can send and receive data between two separate devices that are reachable by audio using ultrasound, okay?

[00:01:19]
So you can actually like send private data, hidden data, without connection, without 3G, 4G, without Wi-Fi, without SIM cards, just by using audio and ultrasound. Be careful if you have animals around you, right? Anyway, pets and so on, because you will work with ultrasound. But anyway, this is how it works.

[00:01:40]
So for example, this is an example, we turn off the Wi-Fi. That's not the ultrasound, that's normal sound. [SOUND] Now, when you go ultrasound. You can log in. By the way, there are some devices at your home currently using this technique, I'm not sure if you know that.

[00:02:15]
So, similar techniques, not websites, but similar techniques are being used, for example, to Chromecast. Or when you're setting up your Apple TV for the first time, Chromecast, they're sending ultrasound messages to your devices to know that it's near, its close. So, in this case, you can use that on your websites.

[00:02:36]
Again, do you understand why what's going on? We have one device, let's say, a phone that is generating a sound over the speaker, but in a frequency that we don't hear. But another computer opens the microphone, receives that frequency, and then convert that back into a message. It's just for fun.

[00:02:58]
But it's interesting that you can do that just with the Web Audio API, okay? So, have in mind you have that and there is a new version of the Web Audio API, that's by the way it's green, so it works on every device, even old devices it works.

[00:03:14]
And now there is a new version that will let you create audiolets, it's a thread. So you can create a worker, a thread, specifically targeting audio. So there is a whole world around audio, I'm not an audio engineer, but there are much more to learn about Web Audio.

[00:03:31]
I think it's pretty cool. Web MIDI. So, we mentioned Web HID, right,? Human Interface Device. MIDI is something similar, the difference is that it's much older than HID. So I have a piano at home, it's been with me for 40, no 35 years, and it's a MIDI interface.

[00:04:04]
So actually, I can connect that musical instrument to my computer and use it from JavaScript. The thing is that it's also an interface that you need to understand. It's not a simple interface only because it's not actually modern, right? But anyway, it's a low-level API. And typically, you use this to connect to music devices, synthesizers, keyboards, guitars, drum machines, and also a lot of lighting systems.

[00:04:34]
Systems that organize lights in a showroom are sometimes MIDI. It's an old spec, so today most of these systems are moving to a new protocol. But anyway, it's still there. And Web MIDI is available on Chrome and Firefox. Firefox already implemented that, but not Safari. Safari started implementation but then stopped.

[00:04:56]
So in the source code of WebKit, you can find references to Web MIDI, but then it's not there. And then you can receive and send MIDI messages, and you need to understand what that protocol is. This is how it looks like. Navigator, request media access, and then you pass two callbacks.

[00:05:14]
You can see that even the API is kind of weird. It's not exactly promise base, it's promise base but most of the time it's kind of old, so it's weird. But anyway, all the APIs look the same once you get into that.