Real-Time Web with Node.js WebRTC Introduction

Transcript from the "WebRTC Introduction" Lesson

[00:00:00]
>> [MUSIC]

[00:00:04]
>> Kyle Simpson: We have one more thing that we're gonna work on. It's gonna be a divergence from this one. So 11.js will start from a different point but I'm gonna introduce a topic and do a couple minutes more of lecture before we talk about our final set of technology.

[00:00:26] So the goal here today was to give you confidence that you can start to tinker around with node JS, understand how the mechanisms work without having to just rely upon these big frameworks that you don't know how they work. Now you understand the underlying fundamental principles and how you can write if statements and even do command line tools and things like that.

[00:00:45] We've wired up HTTP servers and even web socket servers with two way communications near real time. So we've come quite a long way in the last six to seven hours that we've been together. But there's one last step that I want us to take because it's a really exciting step, it represents a step towards the future of what we're gonna be doing with the web.

[00:01:05] And I hinted at that earlier. It's this inflection point where we're moving towards a peer to peer based web. And that hasn't happened yet from an architectural perspective on the web. We're still fundamentally a client server. But I think with the concerns that we've recently had about security and privacy and all of that.

[00:01:24] I think we're going to see a fairly significant change to the makeup of the way the web is architected over the next 5, 10, 15 years. And while that may seem like forever away and we don't need to worry about it. The people that are gonna be making that sort of thing happen are already right now in our industry and starting to think about these things.

[00:01:44] And I would like to encourage you and challenge you to be one of those people to start to think about where the future of our web is headed. Because it's not just what we've always done, there's a lot of neat stuff coming. WebRTC stands for Web Real Time Communications, again it's just another way of moving us even closer because we're removing the server middleman and allowing two computers to talk directly to each other.

[00:02:10] So imagine the scenario, you and a friend decide you wanna play a first person shooter game and so you get introduced through some game server and now the two of your computers are directly connected. There's no server middleman. The only latency whatsoever is how long it takes to go along your ISP router from your ISP to his ISP and vice versa and connect the two together.

[00:02:32] And now you can have latency approaching anywhere from 15 to 50 milliseconds. So whereas we were talking about 50 to 100 milliseconds with near real time web sockets, we're now down to the 15, 20, 30, 40-millisecond range on average between peer-to-peer connections. Now that's not true if you're peer-to-peer connected to somebody from Minnesota and somebody in Australia, okay?

[00:03:00] You're gonna be more like 40, 50, 60, 70 milliseconds best case. Just because the speed of light from here to there is gonna take 40 milliseconds. So there's gonna be some variance here but in best case scenarios you're playing with somebody in a different city that's a couple hundred miles away.

[00:03:16] You might be able to have sub 20 millisecond response times in terms of messaging back and forth. And that really opens up a whole new tier of things that wouldn't have been possible with 100 millisecond latency but now that we might get 20, it starts to become a lot more realistic.

[00:03:31] More real time games, More real time collaborations, definitely lots more security. So peer to peer web and this WebRTC based thing is really GONNA push the capabilities of the web into a whole new realm. So [COUGH] WebRTC.org, it's a website that talks about this technology. And there's really kind of, the main pillars of WebRTC started out with, we need some sort of stream of data to send along, and that's where we got the get user media.

[00:04:00] So the very first incantations of a peer to peer web had to do with sharing your video, your webcam stream, and your microphone stream with another person. In other words doing that meet me conferencing sort of thing. Kind of like we've been able to do for years through proprietary protocols like Skype but all those others.

[00:04:18] Now we can do this directly browser to browser. And about this time last year, we saw the very first case where a call was made, literally a call a video and audio call was made from a Firefox browser to a Chrome browser. The Mozilla team called the Chrome team and they were able to have a conference between their two browsers.

[00:04:41] And what we saw was something very interesting because this is the first time in the history of the web platform that two different browsers, not two different like Firefox 18 and 19 but two entirely different browser vendors had to communicate directly with each other. It's the first time it's ever happened because it's the first technology that is ever directly connected to browser instances.

[00:05:05] We've always had that we agree upon standards in terms of how we communicate with the server and we've always relate information through some middleman. This is the first time that Chrome and Firefox had to agree on the protocol for how they were gonna encrypt their data as they're talking to each other.

[00:05:20] That's never happened before and that was a huge milestone in and of itself that we got to the point where we could see these two talking to each other. So that happened about a year ago, we saw that. There's even more stuff that's been happening with web RTC recently which I'll speak to in just a moment.

[00:05:36] But I wanna give you just a real quick 50,000 foot level view of how this WebRTC thing works. So you understand some of the players that are involved in it. So [COUGH] the first thing that we need to understand is the most likely candidate that you're gonna see the most, visible demo that you're gonna be able to show off to a boss is this idea of capturing your webcam and sharing webcams with each other.

[00:05:58] So this is my highly technical, architecture document for how a web cam capture is gonna work. You got a camera inside of your browser and you're sitting there in front of your screen and that camera pops up and it says, do you wanna allow the page to access your webcam?

[00:06:13] You've all probably seen that pops up at the top. So you say yes, I wanna allow it to access my web camera. And then you get that stream and you did that by using the getUserMedia that I showed you earlier today. And you call that API, you get the user media stream.

[00:06:29] You're probably gonna wanna attach it to some video element so that you can show it to the user. But you don't have to attach it to a video element. You have the stream element. You could just transport it elsewhere and never even tell them. The only way they would know is that they could see the little green or yellow light on.

[00:06:47] I'm sure many of you have probably seen demos out there where it's not just I'm looking at my webcam but maybe they're doing something funky like a grey scale or the sepia tone. Or they've broken it up or they're putting your video onto a 3D cube in a web GL or something.

[00:07:01] And you may have wondered, how is it that they're doing that? How is it they're in real time, they're modifying the video stream? And it turns out they're not. They're not actually modifying the video stream and this is the most important thing to learn about capturing the video.

[00:07:15] When you have a stream into a video element, the video element doesn't have to be visible. It can be a hidden video element, that's just simply retrieving the stream. And so if your webcam is updating 30 times a second roughly then it's gonna be updating your video player, that live feed into the video element 30 times a second.

[00:07:38] Well the video element fires an event every time it gets a new frame of data from the stream, it fires an event, so you can listen to that event. And capture the image data out of the video element and write it to a canvas. So that's what we're showing here, it's called drawImage.

[00:07:56] You would say, the canvas.drawImage and you would give it as its source a video element. So I'd say, capture whatever is currently in that frame of video and draw it out to a canvass. In fact, in the process of drawing it, I can add extra stuff to it so I can superimpose watermarks and do a gray tone on the pixels and any other kind of transformation that I wanna do to the video in real time as that video frame is updated 30 times a second.

[00:08:22] I can draw the pixel data out, make my transformations to it and draw it to a visible canvas. I can even resize it as I've implemented here. So I can crop it, and I can scale it down or blow it up or whatever I wanna do with it.

[00:08:36] Now, once I have an element on a canvas, sometimes people wanna be able to capture a picture from the webcam. And be able to save off that image. So the way we do that is kind of the reverse. We take from a canvas and we call what's called a two data URL.

[00:08:52] So we get a data URL representation of the image data that's currently in that canvas at that exact moment. And once we have that data URL, we can put it into an <img> tag and now that person can right click and save the image off as a file on their server, I mean on their desktop.

[00:09:08] Or we have a data URL that we could ship off to the server and save that file, upload that file into our server or something like that. So we do this dance between canvas and image tags and canvas and video tags. Once you start working with this sort of technology, you end up doing that back and forth quite a bit.

[00:09:27] So you get really familiar with those APIs of drawImage and two data URL. You go back and forth quite a bit. What's your question?
>> Speaker 2: I was just gonna mention that in the media API's workshop, there's actually an exercise where you add filter effects to video of canvas.

[00:09:41] [INAUDIBLE]
>> Kyle Simpson: So go into that latest workshop and practice that exercise. It'll show you exactly how it's doing it. He probably has better diagrams than I do for that.
>> Speaker 2: Yeah it's actually live code, so it's cool.
>> Kyle Simpson: All right [COUGH] so that's how we would do at a very high level, that's how we do our webcam captures.