This course contains good reference material, but does not reflect our current quality standards.
Transcript from the "Middle-End Architecture" Lesson
[00:00:00]
>> [MUSIC]
[00:00:03]
>> Kyle Simpson: We're going to try to switch gears a little bit and focus on applying JavaScript to some stuff and this particular workshop is not exactly a formal workshop. This is more, we are going to basically just go through some code that I wrote for the purposes of being able to illustrate ideas that I have kind of espoused in various talks over the last several years.
[00:00:29] So, [COUGH] just as a brief little history lesson that no body would care about, but since you're listening to me you have to listen to me anyway, [LAUGH]. Way back in 2009 kind of towards actually the early parts of my speaking career, I was working at a PHP shop.
[00:00:52] And I wanted to be able to do JavaScript on the server because there were things that I wanted to do that were awkward or harder to do with PHP. And I also hated the fact that I was essentially repeating code a lot. I don't know if anybody else has had this frustration of like repeating code before, but I was definitely having this frustration.
[00:01:15] My particular pain-point was data validation rules. And I remembered back to my computer science schooling years before that one of the principles, one of the maximums that I learned in my first programming 101 course. Is what we now call the dry principle, but back then what they said in computer science school was anytime there's more than one copy of something, one copy is always wrong.
[00:01:42] Which nowadays we just refer to as dry, we say don't repeat, right? If it's possible to not repeat yourself, it's better to not repeat yourself. It's better to reuse something than to repeat it over and over again. Obviously those are not hard and fast principles, but they're guiding principles in software design.
[00:02:01] So, [COUGH] it struck me that I was repeating this data validation rule set in PHP and I was turning around a writing the same code in JavaScript. And I had done that actually many times cuz I had written many many sorts of PHP applications. Years earlier than that, way back in 2002, I’d written a time sheet application for a company.
[00:02:25] And I’d written all the back-end stuff in PHP inside all my classes and stuff like that in PHP. And then I had to turn around and write most of that stuff in JavaScript as well. So I was repeating some of that same logic over and over again. And any time I needed to refactor something had to go back and refactor it in both places.
[00:02:41] And over the years, I worked in a lot of different shops, I saw a lot of different possible solutions to those sorts of problems. I worked in a Java shop, and I saw, I don't even remember the name of it. But there was some framework that probably Java people know right off the bat, but essentially, they expressed their data validation rules in Java.
[00:03:01] And it automatically generated JavaScript for them, it automatically wrote the JavaScript rules. They had no control over it, it just injected it in the page. That seemed like theoretically it was some solution to the problem because now the JavaScript gets pushed to page, except we ran across a problem where there was one particular sort of check that was happening in Java.
[00:03:20] That the equivalent in JavaScript ended up giving a different result. There was no way to exactly get the same check result in JavaScript that they were getting out of Java. It was close, but we ran across this one case, and it was annoying that I had no capability to go in and change the JavaScript.
[00:03:38] Because it was auto generated on the fly and I had no control over the code that they were generating. So we had to work around that by writing a different set of validation rules and reformatting the way the data was coming in. It was sort of the tail wagging the dog problem, like we wanted to fix the problem but we couldn't so we fixed this other entirely different thing so that the problem would go away.
[00:03:57] So as you can tell, I've had lots of different exposures to this in different things, I've seen it in Python shops. I worked in a Python shop one time and a Django and thankfully I didn't have to do a lot of the Python work, but I did have to do a little bit of the Python and Django work.
[00:04:13] And in particular the problem that I had was whatever framework they were using on top of Django, whatever their application framework was. It could not accept arrays of values from a JavaScript, like if you posted an array of values you couldn't post an array and have it automatically translate into a Python array.
[00:04:35] It required you to have an individually named thing which is great until you start doing UIs where they want to have a big set of radio buttons, for example, or check boxes or whatever. You don't wanna have to uniquely name those things, that's annoying. You wanna just pass an array of all the check boxes that have been checked with true and false in them.
[00:04:52] But we couldn't do that, we just flat out couldn't do it. And the back-end was sort of our restriction at this point, because JavaScript clearly knew how to do it. But we pass the array in and it would work and so I'm just just running into this problem over and over where code was being repeated or things are being too difficult to do bridging this gap between JavaScript and the server.
[00:05:11] And then there were times where I wanted to configure something about the server. I wanted to have some control over something about the server like the way it was putting together the HTTP headers. I wanted to change that to optimize some performance or I wanted to have more control over the way it's packaged up files or whatever.
[00:05:27] And over and over again from Python shops to .NET to Java shops and everything in between Ruby and Rail and everything. I worked at all of them and in every one of those jobs that I worked at, essentially, the back-end ruled, and if you didn't know the back-end language, you just couldn't do what you wanted to do.
[00:05:45] And I was a JavaScript person, or wanting to be a JavaScript person, rather exclusively and this dirty secret was I had no control over the back-end. So in 2009, I had started to get this conclusion like the problem is not that I haven't learned every single back-end of every single job that I've ever worked at because that's not practical.
[00:06:03] It's not practical for me to be hired as a JavaScript developer, but have to become a Grails developer to do something and that literally happened to me, I'm not making it up. I went into a job, very first day, hired as a pure front-end JavaScript that was hired to build an indelible JavaScript widget.
[00:06:21] Totally JavaScript work, I wasn't hired to do anything back-end at all. And I said to them on the very first day, the guy that was kind of training me, I said, all right so what I need is, I need an empty HTML file and an empty JavaScript file.
[00:06:35] You need to be able to serve those up and that will be the basis for my widget. And so just tell me where to put those files and then I'll go from there, I'm good. Just tell me where in whatever your system is, where do I put these files?
[00:06:46] And I was imagining there was some special folder somewhere in this Grails application, I could just drop a couple of files and bam it magically worked. And he said, yeah, you can't really do that. And he literally handed me this huge book on Grails and he said you're gonna have to read Chapters 4, 5, and 6 of this book.
[00:07:03] Figure out how to set up a route, a static routing forward and do this and that and he completely lost me, and I flipped, I was like are you kidding me? That's what I have to do to put an empty HTML and empty JavaScript file out on the web.
[00:07:16] This is ridiculous and so it was kind of this long boil that I had gotten to and I kind of flipped out. I was like, I don't wanna do all of that crap to put up some JavaScript files or two. I want control over at least the part of the server that I need to have control of.
[00:07:33] So in 2009, I started to formulate this alternate way of thinking about web architecture, and I eventually came to call it middle-end architecture and to this day. If you go to middlend.com, it redirects to a series of blog post that I wrote about that concept. Where essentially, I was suggesting there's a set of tasks that every application architecture needs to do which I wanted to be able to do in JavaScript.
[00:07:57] Because many of those tasks, the majority of them in fact are things that you need to do in both locations. For example, data validation, data formatting, routing, templating, caching, headers, those sorts of things. Those are all things that every application does, but they're also all things that you end up having to do a lot of on both sides of the fence.
[00:08:21] The parsing of those things and so forth. So I wanted to be able to implement that in JavaScript and reuse the code in both places. And my only option was to suggest some sort of architecture that would allow me to write the JavaScript and then use it in both places.
[00:08:35] And so I developed this idea of middle-end architecture, it's a tongue in cheek idea. What comes between the front-end and the back-end, of course, the middle-end. And I defined it as the top 10% of what's happening on the server and the bottom 10% of what was happening in the browser and everything in between.
[00:08:52] And I called that the middle-end and it involved all of those tasks that I was talking about. So at the time there was no such thing as Node.js, we had a couple of things. Like, in the Java world, we had Rhino, which still exists today and it's being rewritten, I think to a different engine or whatever.
[00:09:09] But we had Rhino, which was a way to do it. But that only worked if you happen to be in a Java shop. You wouldn't run a Rhino instance just Java all by itself. So that didn't seem to work out for us, especially since I wa, at the time I had moved onto a job where I was in PHP and they definitely didn't wanna have a Java tool in the stack.
[00:09:28] So what could I do? If I couldn't run something with Rhino or whatever, what could I do. And I had read that you could take the JavaScript engine out of the Chrome Browser, the V8 JavaScript engine, that you could write some C++ bindings around it and you could create a JavaScript environment somewhere.
[00:09:48] And I thought to myself, I'll just do that. I'll just write a server-side JavaScript engine if I've already got V8, all I've got to do is write some bindings. I just got to make it accessible to the to the file system and a few other things like that and that's all I'll need to do.
[00:10:03] So I said about to write a server-side JavaScript engine and I in fact did it I write an engine called Bikechain. And it's still up there on GitHub, not that anybody would ever use it, but it's still there if you wanted to go and look at it. I wrote this engine in JavaScript and this was all contemporaneous before and contemporaneous with the launch of my Nodejs.
[00:10:23] Cuz if you fast forward to November of 2009, and I was at JavaScript Con VU in Berlin. I was over there speaking and sat and listened to a talk by a relative unknown in the JavaScript industry Ryan Dahl, stood up and in his dry wit way announced this new thing that he had been creating called Node.js.
[00:10:45] And everybody flipped, I mean we all went nuts. It was standing ovation, I mean, it was crazy cuz everybody's so excited about this concept of getting to write JavaScript on a server. And I'm like yeah that what I've been working on for six months, we need this. Its an inevitability that we wanna put JavaScript on a server.
[00:11:01] Within three months of Node.js coming out and is hugely popular as it took off, I abandoned Bikechain cuz there was absolutely no reason to keep trying to do something a different way. But the process of putting JavaScript on a server regardless of how it actually done the process is what I wanted.
[00:11:18] I wanted to be able to run JavaScript on a server, and to this day that's how I feel about JavaScript. I don't actually care that much about Node, I'm not even all that emotionally wrapped up in this whole forking from Note to IO thing that's happened. And whether they're gonna come back together cuz for me Node is not a trademark.
[00:11:35] Node is not a command line, Node is a category. It's a category of applications that bring JavaScript out of the browser and put them in the place where I want them which happens to be the server. It's a tool and I don't think that Node and IO are the end of this tool chain.
[00:11:54] They are the first modern step that we've had, by the way, they weren't the original. Do you know what the original server site JavaScript was? 1997, LiveScript Server from Netscape. Actually 96 is when it was really kind of coming out. Brand new after JavaScript was out had LiveScript server.
[00:12:11] Netscape was the original, the OG of server side JavaScript. Of course, it's pretty awful and nobody used it, but nonetheless concept of putting JavaScript on a server is not new. But just in our modern age we kind of think of Node as the big first step to modernizing server site JavaScript.
[00:12:30] But it's not the last in my prediction. It's just the first big one that we have seen in a while and we gonna see something that comes after it, might be a year or two or three years from now. But there will be some new idea that comes along, I'm anxious to see things like Firefox OS, which is JavaScript at the kernel operating system level.
[00:12:47] I am anxious to see those things take over where we don't need a user land process like Node to run, we can just run natively JavaScript any part of our application if we want. So I think we're gonna continue to see that evolve. And I don't get so wrapped up in exactly what is Node doing or not doing, or whatever.
[00:13:07] Good question here, somebody just putting a quote in. [COUGH] So that's the little brief history lesson that gives you a perspective on how I approach server site JavaScript. It's a tool that to help me solve problems. The problems that I like to solve are sharing code, not repeating myself, being able to do things in a buffered way.
[00:13:29] In other words, I wanna be able to do my front-end completely separate from how I do the back-end. And that middle-end architecture I referred to, the back-end essentially becomes irrelevant to the front-end. The front-end never talks to the back-end, it talks to the middle-end and the back-end never talks to the front-end.
[00:13:43] It has no clue what the front-end is doing. Could be a mobile app, could be a website, could be another API server, it doesn't care. The back-end simply becomes a headless black box state machine. It manages the session, it knows what state it's in, you give it some data, it gives you some data back.
[00:14:01] And that was attractive to me, cuz I didn't wanna have to relearn every single back-end system to figure out how to manage the headers, and the cookies, the HTML files and all that stuff. I just want you to give me some data, I'll figure out the presentation layer, that's the job of the middle-end.
[00:14:16] So even though today I'm not going to espouse specifically this middle-end architecture, I just want you to have that as the backdrop for what I'm talking about when we go into this code.