Deep JavaScript Foundations Understanding Scope

[00:00:00]
>> Kyle Simpson: First, what you need to know, definition. Scope means where to look for things. It's the set of rules that helps us figure out where stuff is found, how to resolve something. So that definition should ask, or should bring up at least two questions in your mind. The first question that should come to your mind is, what are we looking for?

[00:00:21] Great question. The thing we're looking for is any variable, any identifier, regardless of how it's used in our program. It turns out there's only two ways that something can be used. A variable can either show up in a position where we're retrieving the value from it, or it can show up in a position where we're adding the value to it, or we're setting a value into it.

[00:00:41] Those are the only two places a variable can show up, but wherever it shows up, that variable belongs to some bucket. I like to think visually of our program as a division where every variable is a colored marble, and this is my visual way of thinking about it.

[00:00:55] It's like sorting all the marbles, so I put all the red marbles in the red bucket and all the blue marbles in the blue bucket, and the green marbles in the green bucket. Scope is the set of rules by which we sort all those marbles, and when we want the red marble, we know what bucket to go and get it out of, okay?

[00:01:13] So that's the first question, what are we looking for? We're looking for the colored marbles. We're looking for those identifiers, but there's another more subtle question. And I wanna take a few moments to talk about scope from the perspective of what's going on inside of the JavaScript engine.

[00:01:29] I've already said this, so I already kinda gave it away, but many of you have heard JavaScript is an interpreted language, it's a dynamic language. So if I were to assert to you, JavaScript is absolutely, positively a compiled language, that might fly against some of your intuition. You've always been told before it's an interpreted language, and I will tell you absolutely, positively, JavaScript is a compiled language.

[00:01:54] So I wanna dig into that for a moment, because the second question this should ask is, who's doing the looking? Who's doing the looking? Is it just a JavaScript interpreter trying to find dynamically allocated memory or something? Or is there something deeper going on? And this is really at the heart of this whole unit of discussion, is that scope is a compile time process.

[00:02:20] That our program does go through a compile step, and it is at the compile time that all of these scope decisions will happen. We like to think of it being more like a runtime thing, that JavaScript is a single pass interpreted program language, but it really isn't that way.

[00:02:38] And I'm gonna point out multiple points throughout the rest of our discussion where the difference can be very dramatic. The difference where we say, well, I think that it's in a single pass, and I think the variable should be this value, and it turns out it's something entirely different.

[00:02:58] And you're like, I don't understand where my thinking is wrong, and that root thinking problem is that you're thinking about JavaScript as a single pass, okay? So JavaScript is absolutely compiled. [COUGH] Let me dig into that for a moment.
>> Kyle Simpson: When people say JavaScript is not a compiled language, it's a dynamic interpreted language, and they compare it to what is more traditionally thought of as a compiled language, like a C++ or a Java.

[00:03:26] What they're usually thinking of is the distribution model for the end result of the compilation. They're usually thinking, well, a compiled language, I run it through a compiler and I produce some executable code on the outside, and then I take that executable code and I distribute it. So you think, well, I do that with Java, I produce byte code, I do that with C++ and I produce the binary 1s and 0s.

[00:03:51] But with JavaScript, I don't compile ahead of time and produce some executable code, I ship the source code. So we think our paradigm is, what is the distribution mode for the program? Do we distribute it as the executable code or as the source code? I remember in CS, a really long time ago now, but I remember sitting in CS courses and having an instructor essentially intimate that that is the major difference between the languages.

[00:04:23] But I think that's wrong. I think while that may or may not be a true validation, there's nuance there, even to pick a part there. But while that may or may not be true, I don't think that's at all the most relevant difference between a compiled language and an interpreted language.

[00:04:40] Let me prove to you for just a moment that Java Script is compiled. There's multiple ways I could do that. We could open up the source code for V8 and we'd find that there's actually a compiler there, source code for Mozilla SpiderMonkey, there's a compiler. But let me prove it to you more pragmatically.

[00:04:56] Have any of you ever had a JavaScript bug, anybody? Okay, good, we're all among friends, we've all written bugs before. Have any of your bugs ever been something where you syntactically did something wrong, like you were missing a comma, or you left off a curly brace, or some kind of syntax error that occurred?

[00:05:15] Okay, in that scenario, since we're all nodding our heads, in that scenario, if you've ever created a syntax error in a program, did you ever think to yourself, well, gee, that syntax error was on line ten. But when I went to run the program, it did not run lines one through nine first and then give me the error on line ten.

[00:05:37] When did it give me the error? Before it ever ran line one, right? Well, how do you suppose it did that? How do you suppose JavaScript knew in advance that there was gonna be a syntax error on line ten before it had ever run line one? There's a very simple answer to that question, because the JavaScript engine compiled your code first.

[00:06:00] And the compilation validated the syntax long before it ever handed off to some part of the engine to execute. So at a minimum, I need to adjust your thinking about how you process JavaScript programs. From thinking about this as a single pass, we go all the way through the program top to bottom, to thinking about this, at minimum, as a dual pass.

[00:06:24]
>> Speaker 2: [CROSSTALK] Are you using that interchangeably with parsing? I mean, it's really just parsing it, right? Is that the same thing as compiling?
>> Kyle Simpson: Well, okay, that's a great question. So the question was, am I using compilation interchangably with parsing? Yes and no. The important part of what we're talking about is the fact that parsing does need to occur, cuz obviously, parsing needs to happen for compilation to occur.

[00:06:51] But compilation, there's multiple stages to what we mean by a compiler, and so I'm gonna digress a little bit into compiler theory. I'm one of those people that actually enjoyed that class. That was my favorite class in CS, I loved compiler theory. I wrote compilers for fun now.

[00:07:09] I understand many people may not be like that, but my brain seems to like this, right? So there are multiple stages to the compiler. The last step of compilation is code generation and linking to create your executable code. So does that happen with JavaScript? Or is it just the first stages, where we have lexing, tokenization, and parsing into an abstract syntax tree?

[00:07:36] Which of those two is true? There's two ways to answer this question. First off, at the most base level, the JavaScript engine absolutely does code generation. It absolutely does produce an intermediary representation from the source code to the thing that it hands off in the JavaScript engine, which is actually, literally in some cases, called the JavaScript virtual machine.

[00:07:58] That's actually akin to a byte code, not terribly dissimilar from what's handed off from Java when you compile a Java program, you produce a byte code. Now, that byte code that's handed out to the JavaScript virtual machine is not portable, it's not platform independent like it is with Java, but it is akin to a byte code, okay?

[00:08:17] A set of instructions that are the binary representations. So it's not literally handing off, say, an abstract syntax tree, it's actually a binary intermediate representation. But modern JavaScript engines are actually fantastically more complex than what I'm boiling down now. Because there's all kinds of things like jitting and other stuff like that, JIT is J-I-T, Just-in-Time compilation.

[00:08:41] And as a matter of fact, and this part I don't even fully understand, even with my knowledge of CS. I don't fully understand how you can partially JIT compile a piece of code. To me it's an all or nothing. You either compile it or you don't compile it.

[00:08:53] But actually, modern engines are going through three or four different JIT passes. They have a fast JIT, and then a slower JIT, and an even slower one. They go through it multiple times and they figure out more about the program as they go through it, and then they trace, well, with runtime.

[00:09:09] And then the engine is recompiling stuff based upon what it knows. As it sees stuff happen in runtime, it'll throw away a set of guesses and recompile another set, and hot swap it in. There's all kinds of complexities going on. So it's actually not even a two pass.

[00:09:23] It's very much more complex. But if we just simplify it down to the two pass, there's a single step that compiles, which does the parsing, produces some representation of that program that an execution knows what to do with. I'm gonna call that an IR, an intermediary representation, it's like a byte code.

[00:09:40] And then there's a second pass that executes it. So when I say compile, I mean JavaScript goes through those two passes. That help answer that question?
>> Speaker 2: Yeah, yeah.
>> Kyle Simpson: Okay, so what I'm asserting is that that first pass has to occur because parsing has to happen. Otherwise, we'd never know about a syntax error on line ten if we were on line one.

[00:10:02] As a matter of fact, there's more proof that the parsing is happening, because JavaScript has a set of things which are enforced in the parsing stage of the grammar. They're enforced as what are called early errors. For example, in strict mode, if you have a JavaScript function that has two different parameters with the same lexical name.

[00:10:23] Like you have function foo, and then you say a comma a in the function signature, that's not a syntactic error. That's totally syntactically valid. But that is still thrown as an early static error. In other words, before the program is ever run, an error is thrown to tell you, you're not allowed to have two parameters of the same name.

[00:10:44] That could only happen if there was first a full pass through the program to do all the parsing and validation of the code, and know everything that they need to know to verify these parameters are the same, right? So there has to be that first pass that occurs.

[00:11:00] So all I really want you to do is to just reset your thinking from the one pass to the two pass. First, think about there's this pass that happens to validate my program, and it's during that pass that lexical scope gets set up. It's during that first pass that scoping happens.

[00:11:19] It is not during the second pass, which is what most of us have sort of loosely come to believe, okay? So JavaScript is absolutely a compile language. As a matter of fact, I would make the case that that compiler architecture that I just quickly alluded to there is vastly more sophisticated than any compiler architecture for any other language in existence.

[00:11:42] Because we're demanding so much more of JavaScript, and they've come up with all these different techniques that were never even relevant to other things. For the C++ compiler to be a JIT compiler wouldn't even make sense, right? Cuz you do all AOT, it's all ahead-of-time compilation. So JavaScript has to do a whole lot more, it's a lot more sophisticated.

[00:12:03] Now, [COUGH] I'm gonna make this statement, there is an asterisk here when I say JavaScript organizes its scopes by functions. That's not entirely true. We are gonna talk about block skipping later in this unit. But for now, we're gonna simplify things and say the individual unit of scope is the function, okay?

[00:12:28] So here's a piece of code that is deceptively simple. This piece of code doesn't even look like it does much, cuz it just makes some variables. There's no logic or branching or looping or any of that stuff. So you'd be inclined to just sort of skip over it and say, well, this isn't all that interesting.

[00:12:48] I'm probably gonna have this slide up for 20 or 30 minutes, cuz there's way more to this than you think there is. For us to fully unpack what it means for this to be a two pass system, to start to rewire our brains to start to think a little bit more like the JavaScript engine.

[00:13:04] The vast majority of the complexity of compiler theory is not stuff we need to concern ourselves with. People like me like to obsess about it, but it's not necessary that you'd be a compiler theorist to understand JavaScript. But there are a few things that you should be at least moderately aware of, okay?

[00:13:23] So I will use a few different terminology, a few different concepts. But for the most part, the only thing we're really concerned with is in that first pass when we compile the code, we wanna look for all the variable declarations and all the scopes that they get added to.

[00:13:37] That's the important part, don't worry about tokenization or abstract syntax trees or any of that other stuff. All we wanna care about is how do those buckets get created, and how do we sort those marbles into those different buckets? Make sense?
>> Kyle Simpson: So to do that, I've got a little trick that I do when I teach this material, and it's gonna feel silly and a little bit strange.

[00:14:04] If you're listening live or if you're watching this video later, even if you're not actually physically here, I'm going to encourage you to do this, even if it seems silly. I'm gonna take you through a little exercise here where we have an out loud conversation. And the things that we say are gonna sound a little bit ridiculous, but it's okay because we're just doing this all together, all right?

[00:14:29] So we're all gonna sound ridiculous together. It's a little teacher trick that we use to help something that's a bit abstract get stuck virally in your head, okay? So I'm just being completely transparent about it. It's silly on purpose because it helps something that you otherwise wouldn't really think about that much.

[00:14:49] It helps it get stuck, and the important part is not that you'll ever repeat this exercise. You'll never actually say these things ever again. But the concept will get stuck virally in your head, okay? So we're gonna go through a conversation that the different parts of the JavaScript engine have.

[00:15:07] Fancy word, we're gonna anthropomorphize. That means to take something that's not a human and treat it as if it is. We're gonna anthropomorphize the parts of the JavaScript engine that are relevant, and eavesdrop on a conversation that's occurring as this code is being processed. And I style it as a conversation so that we understand the different responsibilities, and how they're divided.

[00:15:30] So that's why we talk as if these are two different people having a conversation, and we're gonna go through this in two passes. So the first pass through this code, we're going to be doing that compiling step, that looking for lexical scope step. And [COUGH] we're going to be having a conversation or listening in or pretending to have a conversation between these two entities.

[00:15:55] The first one would be the compiler itself, it's actually processing the code. And the second one is gonna be the scope manager, that's the kid at the playground that manages the buckets, and puts the marbles into the buckets, okay? These two are gonna have a conversation as this code is processed, and the goal is to set up those buckets and properly sort the marbles.

[00:16:18] And that conversation is gonna start a little something like this. We're gonna start on line 1, we're gonna be looking for formal declarations, formal declarations of variables, and formal declarations of functions. Line 1 is a formal variable declaration, because we see var keyword there. Line 3 is a formal function declaration cuz, we see the function keyword there.

[00:16:39] There are informal declarations, which we do not care about, we're only looking for formal ones, okay? So we get to line 1, we're the compiler, and we're gonna be talking to the scope manager. We get to line 1, we're gonna say, I'm the compiler speaking, hey global scope manager, because I'm in the global scope, so I'm talking to the global scope manager.

[00:17:00] Hey global scope manager, I found a formal declaration for foo, and then I'm gonna ask this question. I'm gonna say, have you ever heard of him? Now, the question might seem irrelevant, but actually, the question is really important, because the question lets you in on that division of responsibility, and that's important to keep string.

[00:17:21] So the compiler is saying, I don't know whether this variable exists, who's the only one that can answer is this marble's ever been found? The scope manager. So hey, scope manager, you ever seen this red marble before? I have a formal declaration for a variable called foo, have you ever heard of him?

[00:17:36] And there's only two answers, yes, or no, okay? In this case, what's the answer gonna be?
>> Speaker 3: No.
>> Kyle Simpson: No, never heard of him, but now that you mention it, I'll go ahead and put that marble into the red bucket. So now we have a red bucket with a red marble in it.

[00:17:53] The variable foo has been registered into the global scope, okay? You notice we didn't pay any attention to what was going on on the right-hand side, the equal sign, that's all executable code, we'll get to that later. All we care about now is these formal decorations and sorting out the marbles.