This course has been updated! We now recommend you take the Complete Intro to Real-Time course.
Transcript from the "Node.js Observations" Lesson
[00:00:00]
>> [MUSIC]
[00:00:04]
>> Kyle Simpson: All right. So that's it for our survey of HTML 5 technologies the next thing that we're going to do is start to get our hands a little bit dirty with node. Well we're going to move into some code there but I wanted to take just a few more minutes to kind of the last of my lecturing if you will for the day before we get into our code.
[00:00:24] I want to just kind of talk to you about where node really sits in the whole scheme of things that we're dealing with technology wise. This is firmly in the realm of Kyle's own personal opinions and my own take. I in no way shape or form do I represent any official stance and moreover I'm not even very liked in the node standards community so they probably would disagree with me.
[00:00:50] I'm just sharing with you my own personal opinions and where I see nodes fitting in not only now but in the future. Because there is a tendency and I'm one of those people that would have this tendency cuz if you can't tell. I like JavaScript a lot. There's a tendency to think that Atwood's Law which says anything that can be written in JavaScript will be written in JavaScript does that that means we should just JavaScript all the things.
[00:01:13] We should write everything in JavaScript throw away all the Python, all the Ruby, all the .NET everything that's ever been written and anything but JavaScript throw it away and rewrite it in JavaScript. That would be super exciting to me but it also would be a really terrible shitty world.
[00:01:28] Cuz JavaScript is not right for everything. There's a whole bunch of stuff that JavaScript is right for that we haven't tapped its potential yet, it's a bright future. I say always bet on JavaScript. So I see a bright future and it's only getting brighter for JavaScript. But that doesn't mean that everything should be written.
[00:01:45] I heard I don't even know who it was that I heard it say this but I was at a recent conference that I was at, one of the panelists said, you know [COUGH] theoretically you could write code for my pacemaker in JavaScript but I don't want you to stick JavaScript in my heart right?
[00:02:00] So think about it in that respect I probably agree with you. I don't want JavaScript running my medical devices probably. So just because you could do it in JavaScript doesn't mean you shouldn't that adage goes for a lot of things. And so just because node.js makes it possible for you to write everything that you could conceive of in JavaScript on the server or in robots or light bulbs or wherever else it's been embedded, just because you could doesn't mean that that's what node is really for?
[00:02:26] It doesn't mean that that's where node sweet spot is. So, if it's not zero and it's not infinity, where is the middle ground? What does node do well? The first thing and this is the reason why we're spending most of our time today thinking about it from the perspective of communication technology.
[00:02:42] It's really super efficient at communication. It's really really good at doing highly efficient communication. So this concept of web sockets when we talk about signalling for peer to peer you'll see the same kind of concept it's really really good at low latency high throughput communications. Not so good at static serving of gigantic big files.
[00:03:04] I probably wouldn't write a CDN server that serves up giant video files based upon node. It's not so good at that. It's not that it can't do that, but that's just not its most efficient sweet spot. It's really good at lots and lots of tiny things happening all at the same time and taking advantage of the event loop and things like that.
[00:03:21] So,
>> Kyle Simpson: [COUGH] Communications is a great sweet spot, any time you're going to think anything at all about web sockets or any of the kind of highly efficient communication that's necessary. Nodes should pop to the top of your head because it's going to be every bit as efficient.
[00:03:39] It really runs very comparably to the most highly optimized systems on engineX and other things like that. It really can. And in some cases out perform them. They've been cases where people have been able to show that it outperforms natively compiled C code and that's kind of a shocking thing to consider compiled JavaScript program out performing up optimized C but it's possible.
[00:04:02] So there are some really good things that JavaScript on node.Js could do for us. But a couple of years ago when node was kind of first coming out and in fact literally while node was being invented. I was wrestling with this concept of how does JavaScript on the server fit into the jobs that I was working at.
[00:04:23] Remember I've been in this industry, have worked in lots of jobs I only recently in the last couple of years became a teacher. So I got 13 years of working in all the crappy, corporate jobs that all of you work in. And I know what it's like when you get really excited about something but the reality is that it doesn't fit with where you're working.
[00:04:41] And I had this frustration over and over and over again, where I wanted to use JavaScript to do some task on the server. I thought it makes a lot of sense that I could write it better in JavaScript anyway, so let me do it in JavaScript. But back then we had no node and so the only option was install the Java stack and use Rhino.
[00:05:00] Or something like that, which I don't love Java so I didn't want to do that, and so I was exploring this idea, what does it mean to use JavaScript on the server? How does that fit in with the overall web stack? And what I'm gonna do is to share with you some ideas that I've had them out there for a while, I read a whole bunch of blog posts about this.
[00:05:16] If you're interested more in this topic, you can go to middleend.com. And it redirects to a bunch of blog posts that I've written over the last several years about this topic. But I started thinking about a different way of approaching architecture that might leverage JavaScript on the server.
[00:05:31] And remember this predates node so I'm not even really talking about node pursuit per se I'm talking about JavaScript on the server. I started thinking, if I'm at a company that already has 20,000 lines of PHP code, or Java, or .NET or whatever, if I got really excited and I went to some conference or I went to a workshop like you are here.
[00:05:50] And I got really excited about JavaScript on the server and I saw node, and I was like, my God, I gotta do this, and I go back to my company. And on Monday morning I walk into the standup and in front of my boss I say to everybody here's my thought we need to throw away all 20,000 lines of our code base and rewrite everything in JavaScript.
[00:06:08] What is the likely reaction going to be for my team just in general? If it's not all ready in JavaScript the likely reaction is GTFO not interested. We're not, It's not practical. Even if that's exciting, it's not practical. And many times there's a strong gut roll reaction. No, I definitely don't want to rewrite it in JavaScript.
[00:06:28] Your server side dot net guys are not interested. All right. So, where does that leave me? If I'm excited about JavaScript but it's not practical for us to go and write our application in JavaScript and we don't have the ability to do greenfield development from scratch, then where does that leave me, am I stuck?
[00:06:44] Do I have no options in terms of JavaScript? So what I just want to share with you of the next few minutes is my take on where server side JavaScript fits in to the overall scheme of things in the real world. Okay, we have to set aside the cool fun stuff like the stuff that we're gonna explore today.
[00:07:03] Set that kind of stuff aside. When you go back on Monday morning and you try to think about, how could I use JavaScript at my job? If it's not a reality that you're gonna rewrite everything Is there any chance? And then these are my thoughts. So I started thinking I was at this job, it was a PHP stack.
[00:07:20] And I started thinking [COUGH] what would it take for me to convince my team, not that we need to rewrite the entire application in JavaScript, but there was one particular pain point that I really really hated. And it was the Smarty templating engine. Because we're using PHP. I don't know if anybody had any experience with this, was many, many years back but we were using Smarty and I really, really hated some things about Smarty.
[00:07:42] Didn't like how it was working and so I was trying to convince them or trying to come up with a way to convince my boss. And my team that we should stop doing the templating part of our presentation layer using PHP and using Smarty. And I thought we could write a really cool simple templating engine in JavaScript.
[00:08:02] This predates a lot of the current JavaScript templating engines but I thought we could build a templating engine in JavaScript. And if we could run that JavaScript on the server we can do exactly what we do now where we present things and it didn't even need to be a single page app.
[00:08:16] It could still be server driven app but we could let JavaScript run that presentation layer. And so I started playing around in my mind with what that architecture would look like and PHP is as many of you know it is a per request synchronous model. When you make a request to the server it spins up a thread with Apache and PHP in it.
[00:08:38] PHP runs the completion, it sends out a response and it shuts itself down. There's no you know persistent memory mothership thing like there is in Java where it can manage stuff in memory. It's just, it's a per request sort of thing and I thought, well, then I probably need a Java Script engine that can be spun up per request.
[00:08:57] So when PHP comes in and sort of the CGI model, I could spin up the Java Script engine, run some code to generate a template, shutdown the Java Script engine and take the string and ship it back out to the browser. So I started exploring. What would that look like?
[00:09:10] And a little known fact at the same time that Ryan Dahl was inventing the really awesome no JS. I was writing a server side JavaScript engine called byte chain to do exactly that task. Could I spin it up on a per request basis run some of my server code in JavaScript and then hand back control over to PHP, or .NET Java or whatever.
[00:09:32] This actually spanned over a couple of jobs that I worked on this idea because then I went to a .NET shop. And then I went to a Grail shop and I saw everything under the sun and every one of them the same pain points were there were these things that I wanted to have control over in doing JavaScript.
[00:09:46] And I could not do them in JavaScript because it wasn't an acceptable architecture pattern. And so, I came up with this idea of the middle end, I called it CBC client view controller architecture. And this idea of the middle end is that, I observed that in every one of those disparate stacks that we talk about whether it's .Net or Java or whatever.
[00:10:07] There's a certain core set of tasks that always happens. And most of the time these core tasks are things that a front end developer needs to be aware of. And needs to have control over but equally likely those tasks are deeply embedded inside the guts of the back end of their app.
[00:10:24] Things like templating, things like routing of URL's, formatting your data, validating your data. How many of you have written validation rules two, three, four, five times inside of different parts of your stack? You know exactly what I'm talking about just five right. No big deal just five. Because we have to write it once in JavaScript on the front end because we wanna be responsively reacting to the user.
[00:10:49] And then we of course we can't trust what comes to the browser so we write it again in our application layer. And then the database guys flip us off and say well I don't even trust the application layer so I'm gonna rewrite it in a stored procedure Inside of the database and then who knows what else?
[00:11:03] I mean there's all these multiple layers of people that feel like they need to do the task of data validation cuz I can't trust the other guy. So there's a maximum computer science for this that says anytime there's more than one copy of something one copy is always wrong.
[00:11:18] So the more times we repeat our data validation code the more chances there are that something's out of sync, something's not right and that's a source of untold amounts of problems. The same kinds of problems could be identified with templating, with URL writing, with data formatting, internationalization, all these things.
[00:11:37] These are things that I call middle end tasks. And they're usually embedded deeply within your back end and that means that your front end developers who are the ones that need to care the most about those things. We're are the ones that care the most about how information gets transferred back and forth through JSON.
[00:11:53] We're the ones that care the most about how things can be templated and presented in the front end. The backend guys don't care about that stuff that's why they write servlets to handle it cuz they're not interested in markup but we are, as front end people. We're interested in that stuff and yet this unfortunate paradox is that the stuff that we're most interested in is the stuff that's furthest away and most abstracted away from us in the stack.
[00:12:16] And I think, that's an architectural problem that we need to solve. So the ideas of middle ender coud we surface, these middle end tasks into a middle end tier. These things like data validation and routing and templating and so forth, could we put them in a middle end tier?
[00:12:30] And then I went so far as to say what if we implemented the middle end tier using server side JavaScript? So I'm not talking about rewriting your entire app. In fact 90% of your app we wouldn't even touch. I'm only talking about this small 10% of your backend, that your backend doesn't even really wanna do anyway and it's not very good at.
[00:12:51] Just having it stop doing stuff like templating and data validation and put those things into the middle end. And when we implement them in JavaScript, we get this huge win because now we write code that literally the same line of code can run both on the server and in the browser.
[00:13:06] So we finally get to that euphoria of I can write my data validation rule once and only once and use the exact same code in both places. So these were these ideas about middle end using server side JavaScript and it predates the concept of node.JS. Now, I've long since abandoned Bikechain because clearly, Node.js won and it gave us a really awesome server side JavaScript environment.
[00:13:29] But Node.js is not the only thing that's ever been done in server side JavaScript, and it's not the end answer. There will be other server side JavaScript things. So I'm really talking bigger than Node here to say that it happens to be a really great option right now for running your JavaScript.
[00:13:46] And I would suggest to you if you're going back to your jobs on Monday morning and you're wondering to yourself, how could I ever convince my boss to let me do server side JavaScript? One way to do that is to tell him to flip him a big middle finger and say I want to rewrite our entire code base and then you're probably going to be looking for a new job.
[00:14:05] Another perhaps more likely scenario is you go to him and say hey I've got this crazy idea, what if you let me rewrite all of our data validation rules in a Node.js and little tiny middle end tier and all I do is the data validation rules. That's it I'm gonna rewrite them in JavaScript.
[00:14:23] We'll take them out of the database take them out of the application layer take them out of the crappy JavaScript and jQuery that we've already written. And we'll write them once in a properly designed way we'll audit the rules to make sure that they're valid and they're sane and that they're solid.
[00:14:36] And then we'll run it inside in Node.js and we'll use the exact same code inside of the browser and now we have one location. A source of authority for our data validation rules. That to me seems like a somewhat more plausible conversation to have with a team. I don't know.
[00:14:51] You may work at a job where they'd still laugh at you but it seems like a little bit more plausible of a way to go about it and see you do that you prove it. You say let me prove it. Let's try it for six months and see if it works.
[00:15:00] And then once you've proven that, then you can come back to him and say, okay, why don't we talk about data formatting and our internationalization rules? How about we put that in the middle end? And then how about we take all that templating crap that you've been doing and we put that in the middle end?
[00:15:13] And then how about we talk about URL routing and we put that in the middle end? And pretty soon, you have a nice well formed architecture where the front-end and the back-end don't have to worry about each other. Because the buffer zone, the middle end is where all of that important stuff happens.
[00:15:26] Your front end developers that know JavaScript can do everything they need to do with the middle end tasks in their native language. And we get the benefits of dry reusable coding. That is what I see the future of Node.js and server side JavaScript. That I think is a much more plausible story for how JavaScript continues to take over.
[00:15:46] Rather than the exception cases like when Netflix comes out and says, we're rewriting our entire app and node. And, everybody cheers and says that's awesome. There were for every one of those there's a 100,000 companies that say, hell no. We're not rewriting our app. Those people still can benefit from server side JavaScript.
[00:16:05] Your company can benefit from server side JavaScript. It can benefit from Node, and I think one of the ways it might do that is this middle end task concept. So that's it for my lecture on middle end. Let's actually jump into some code if you're interested at all in any more of those ideas or reading about it or talking more about it again you might check out middleend.com.