Introduction to Node.js, v2 File System

[00:00:00]
>> Let's move on to the file system where we do have a couple exercises that you'll be going through. And this is where you're going to start to see just how strong node JS is compared to or how strong an OS level up OS level language is compared to like a browser based Language and all the things you get access to.

[00:00:18] So, file system. Until node JS there really wasn't a great way to access the file system in JavaScript. I'm sure there was, but it precedes me. A lot of time I started learning to program there was no JS. It was early but it was around. So I don't know how you would have access the file system with JavaScript before no GS because JavaScript didn't really run outside of the browser.

[00:00:41] You can maybe click a button and upload a file if the user selected it, but you couldn't go query someone's file system in the browser that would be a security risk. So I don't think there was a really good way to access someone's file system or a machines file system with JavaScript before node JS.

[00:00:56] And that's really powerful. But now that no GS is here, just like any other OS level language, we do get access to the file system. And it's crazy because like when I first started learning about that I was like okay, cool, so I can do stuff with a file.

[00:01:11] I don't have any interesting files on my computer a lot. Honestly, why this is a big deal. But if you think about all the tools that you use, especially anything as far as front end tooling. It's all just, implementation or an abstraction around the file system. And that's how you get like some of those magical workflows.

[00:01:30] Some of those things you can't live without, they just like, I'm so glad that that did that for me to clean this up. It's shortened this and concatenated that it joined this. Those are file system projects and it's a very powerful tool to have. So what we're going to do is we're going to create a simple templating solution with the file system.

[00:01:50] We're going to like render some html on the page, but not until we process it first. Which is very similar to something you would do on Webpack or Babel, or any other tool that you use to build your react or whatever front end language that you have. Feel free to follow along.

[00:02:05] If you don't want follow along I do have all the code we would be using you can copy and paste it to recon from the cuorse tha I have here. So first we want to understand FS module. So I'm gonna go back to the project here. And am going to just going to leave this staff outside here.

[00:02:25] And I`m gonna rename this to MJS cuz like I said, I'm going to use MJS going forward. If you don't wanna use MJS that's fine. Just know that all my exercises are going to use MJS. I'm gonna use the latest modules. If you like common JS, so if you want to do it, go ahead and do it.

[00:02:39] I just won't be doing that going forward. So first thing I want to do is understand the file system. So we import FS from FS, which stands for file system. We get access to this super awesome package here. Let's log this out right quick. Oops!. I can spell con after that log, there we go.

[00:03:03] You can see, we get a bunch of stuff here. A bunch of really cool methods, always like to log stuff. You can see, we have things like a pin file, a pin file, sync access, access sync. As shown or I don't know how you actually pronounce that, but always just always pronounce it like, chown or whatever.

[00:03:22] But it's a file manipulation permission handler thing. Close, copy, exists all this stuff. And you can see there's always like a sync version of it. That's really interesting. And that really just comes down to like how no MJS works. So If you don't use the sync version of a file system method, it's going to be asynchronous.

[00:03:40] As in, it's going to be set aside in the event loop to be processed on the next tick at some point. Because it's a synchronous, we're going to continue to accept input throughout this node execution while this thing is being processed in the background. And the node will notify us via callback when it's done.

[00:03:57] So that's the asynchronous version. If you prefer to block the process of node as in, hey, do not do not move forward, do not process any other code. Do not accept any other input until this is done, then you would use the sync version. And in most cases you probably would never use the sink version of anything and node JS.

[00:04:17] There are some rare ones where you literally don't want anything to happen. Typically like on a server startup, or some initialization, where you just want everything to be done before the next thing happens because it only happens once. That's when you would do it. But if you're already like executing a server or something like that, you probably don't want one request coming in and blocking your entire server.

[00:04:39] So no one else can actually hit your server because you're trying to upload an image. So you probably don't wanna do that. So you typically will opt for the async methods. Have these file systems, these filesystem methods, which are the ones that don't have the word sync after them.

[00:04:55] So like I said, there's a lot of interesting ones here. The ones that we're going to be interested in are going to be the read file and a write file. And we are going to use the async ones. And actually Fs was just updated to include not only do they have the async ones, but they actually gave us this nice path to the FS module that has the a sync ones that are already wrapped with promises.

[00:05:18] Because before they were actually, you have to use them with callbacks, which is cool. But promises are better because we can a sync await them, which is amazing. So we're actually going to import from Fs/promises. That way we get the permissive ide versions of the methods that we want.

[00:05:34] So now that we got that, the next thing we need to do is let's just get our template file. So we're gonna copy this html here. Looks like standard html, but it's a little different. And I'm gonna tell you why I will make a new file. And I'm gonna call it template.html.

[00:05:49] And I'm gonna paste it in here. So it looks like an HTML file, except if you go look at, line 9 and Line 10. You can see if you use any type of JavaScript framework, you notice that this looks like some type of interpolation. This looks like what JSX would have in the brackets.

[00:06:04] This is where you would insert some JavaScript. That's exactly right, because that's exactly what we're about to do. We're about to process this file, and we're going to replace these placeholder variables with actual JavaScript values using node JS. So very similar to what you would have in JSX, or Angular, or view, or svelte, or whatever framework you want.

[00:06:24] We're gonna interpolate these values via node.js without actually executing it in a browser. So in order to do that, we first need to import that will not import. We need to read the file, we need to get the string representation of that HTML file. That way we can find the variables that we want to replace and replace them.

[00:06:45] So let's do that. We're going to use the read file method from Fs. And I'm just going to go ahead and destructor up here because it's a named import and I'm going to call it Read file like that. So you can do that. And then what I'm gonna do is I'm going to read the file.

[00:07:02] So I'm gonna say const file or let's call it template equals. And I'm gonna await this because read file is a promise returning function and the latest version of node we do have access to async await. And I can actually use it on a route like this without having to wrap it in a function which is also really cool.

[00:07:22] So I can say await, read file. And then read file takes in. In this case, we're gonna pass in two arguments. The first one needs to be the absolute path to the file that you want to read. Typically what you would see in a common JS implementation is you will see someone import path from path.

[00:07:43] And then they will do something like path.join. And then they would say I wanna join from this directory name over to the template.html. And this this will totally work back in comma js. So we try to execute this now, you'll see. It's not gonna work. And that's because __dirname is not defined.

[00:08:05] And I know it's said that __dirname is a global, and it is. I didn't lie to you. It definitely is. But only when you use CommonJS, you actually don't get access to __dirname or filename inside of model.js. I found this out a couple months ago, and it's caused a really big bug in my app.

[00:08:20] This is why I said yeah, just switching over to njs, and just think that it's gonna work, it's not really. You gotta go in and fix all this stuff on a case by case basis, because you get stuff like this, such a big variable that doesn't exist anymore.

[00:08:32] Yeah, you're probably gonna have to make some changes. And the change that we're gonna make is we're going to use, fortunately for us, readFile also takes in a URL object. And we could construct a absolute path using the new URL implementation. So we can say a new URL like this.

[00:08:53] And what this is gonna take is two arguments. The first one is gonna be the basic, the relative. It's basically gonna be the file name that we want. So it's gonna be what do we want? We want template.html, so we wanna go there. But then we wanna set the base, which is going to be we're gonna grab something from import.

[00:09:15] So it's gonna be import.meta.url. That's just basically if we log this, I'll show you what this is. Little bit of that, so let's say console.log. That, and then we might as well also just, yeah, we'll just log that, so you can see what that is. And you can see it's literally the path to this file.

[00:09:37] It's the absolute path to this file, so that's gonna be the base of where we want to load from. And then we're gonna pass in template.html here, so it's gonna be relative to that file. And now when we run this, I'm going to get rid of that log, and I'm gonna console.log template.

[00:09:55] We should see some strange output here probably, exactly. So we see this thing called a buffer. A buffer, it's like the little individual bits of, in this case, a file. It's all the little bits that make up a file. It's not a string, it's implementation. It's when you want to stream something.

[00:10:16] And in that case, imagine that this file was a gigabyte. If we read this file in Node.js and loaded it up, we would have to load up a gigabyte in memory. And I don't know if that's something you wanna do, especially if you have multiple files. So it's better to just stream them bit by bit.

[00:10:33] And that's what a buffer is, it's literally streaming the file. But this isn't useful if we need to do some processing, unless you wanna do a process on a buffer, which I don't. So we need to convert this to a string. And then we can actually look at those variables, those placeholders, and we can interpolate them with actual values.

[00:10:50] So that's what we'll do. We got a couple ways to do this. We could just call toString on a template. And now that I actually just converted it to a string, because buffers have a toString method, and that works. And you can see we get our HTML back just fine.

[00:11:03] The other way we can do it is we can just go over to the second argument of readFile. And we can pass in the format that we want here, and I'm just gonna pass in utf-8 like this. I'm sorry, I said format, the encoding that we wanna pass in here.

[00:11:17] And utf-8 just basically means turning that into a string. And now I don't need the toString. And it still logs it, the HTML here. So step one, we read the file. We have it loaded up. It's a string now. It's inside of our Node.js file, so it was an HTML file on disk.

[00:11:35] Now it's a string in memory, and now we can process it. So the next strategy is to figure out how to replace these with some actual values. In a real implementation, you would probably use some other templating language. You would use some regex based thing. We're gonna do it the cheap way, and we're just gonna loop over this and do a string.replace and keep it simple.

[00:11:56] So first thing is let's actually get the data that we want to swap out with these variables. Let's make an object here, and we'll call it data, and we'll have a title. And we'll call this learn Node.js. That's the title. And then I think the other one was a body.

[00:12:14] Yep, so we'll have a body. And a body is this is the final HTML. So we got our data now. And then the next thing we wanna do is we can just loop over this data. And then we can just replace every single instance of the placeholders that match the key inside this, because the key is title and body, matched with the variables title and body.

[00:12:41] So we'll just match those using the replace method. We could say for(const). Let's just say key value, key value of Object.entries. If you've never heard of Object.entries, when you call Object.entries and pass in an object to it, it's basically gonna return an array of key value pairs. So tuples, so it'd be an array of arrays.

[00:13:11] And each array in the big array is gonna have a key as its first value, and a value as its second value. So in this case, I'm getting the key and the value of each key value pair, and I'm looping over them. So for each one of those, what I wanna do is I just wanna replace the placeholder.

[00:13:29] So I'm actually just gonna turn this into a let, and I'm gonna say template = template.replace. And if you look at the placeholders, they start with this bracket, and they close with this bracket, and then there's a variable name in the middle. So that's what we'll look for.

[00:13:44] So we'll say replace these brackets, followed by the key here. And I wanna replace it with the value, and that should be okay. So I'm replacing everywhere where there's a key bracket with the actual value. So I'm gonna replace that. I'm updating the template all along. And then I'm just gonna log, The template to see if it actually works.

[00:14:09] So let's do that. And you can see we actually replaced those variables. We replaced where there was a bracket title with Learn Node.js. And we replaced the paragraph tag with this is the final HTML. Don't worry about these little arrows. That's just my terminal. It prints these arrows out.

[00:14:24] I think it's the font that I'm using, I don't know. I kinda like it. But no, it doesn't actually look like this. You won't have an arrow here. It's kinda confusing, but yeah, this totally works. So we're almost done. The next thing we need to do is we need to write this file back to the disk, because we do want this file to be loaded in the browser.

[00:14:42] We wanna see it. We want it to be served by a server. We're gonna deploy this somewhere, right? So we have to write this file back to the disk. Otherwise, what was the point of this? So the thing that we wanna do here is we wanna import another file called writeFile, which is pretty much the opposite of readFile.

[00:14:59] So we're gonna say await writeFile like this, and then it pretty much takes the same arguments as readFile does. So we're going to say new URL. Oops, nope, not that, this one. And we're gonna pass in the name of the file that we want. In this case, let's just call it index.html, like that.

[00:15:24] And then imports.meta.url, which, by the way, I think there's a lot of different approaches to solve the fact that we don't have __dirname and filename inside of MJS. This is just the way that I've settled on. This is the way that I've seen other libraries and frameworks approach this problem, and it's something that I've decided on.

[00:15:44] But if you were to Google well, how do we get around no __dirname or filename in MJS files? You will probably find a lot of different solutions. This is the one that has been consistently great for me. So we're gonna write the file. And then we want to, as a second argument, pass in the file that we wanna write.

[00:15:59] In this case, it's the template string. So we're gonna await and we're gonna write that file. And we should see it pop up here on the left when we execute this code. So let's give it a try. So I'm gonna do that. There was no output here because I didn't log anything.

[00:16:13] Because the output was exactly this index that HTML being written to the disk. So if I go check it out, you can see we're all good here. And just to check to make sure it's valid HTML, I can open with, not that, we're gonna open, where is this?

[00:16:30] I wanna, you're revealing finder. Then I'mma a double click that. And you can see it's a real HTML website. We just process some HTML similar to what a build tool that you've probably been using for quite some time does. So that's really powerful. Obviously web pack is more than this, TypeScript is more than this, Babel is more than this.

[00:16:49] But at its foundation at its core, it's reading your files as strings in memory and it's processing them, complicated plug in systems and architectures and an edge case protections and all types of crazy things. At the end of the day, that's what it's doing to the file system, it's reading it and then it's writing it.

[00:17:05] Like you've ever use web pack or babel or Typescript and use a seal line is like give us the input file that you want to start with. Then give us an output directory to output too, they're writing to it, they're reading from it, that's exactly why they need those arguments.

[00:17:20] So that's a really powerful feeling. I know when I started reading or writing files, one of the first things that I built was a like starter kit like I always built at the time I built like everything in like mango express Angular and node so like the mean stack.

[00:17:38] So I built a little system that would basically scaffold out the start of how I like to build an app. So it's like a little scaffolding tool. And that was like the first thing that I built with the file system and I thought that that was really powerful.

[00:17:51] So, any questions on the file system or anything that I just covered? That's a good question. So, await, let's talk about that. Let me go back to the code. So what is await? And what is it doing here? Good question. So, we have to understand how a synchronous code works in JavaScript.

[00:18:10] So I'm not gonna go deep into that because there are so many amazing courses on front end masters that can talk about that much better than I can in this course. But the short answer is read file and write file are asynchronous tasks as in, they actually execute in the background.

[00:18:27] So what what I'm saying is we'll get to await. Let's say I just remove this await. What's actually gonna happen is when JavaScript looks at this file on line three of this line on line three, it's going to initiate this function. But it's not going to wait around for it to be finished.

[00:18:42] It's going to task it to happen later on in the background. And it's gonna continue to move on. Now, that's gonna cause problems because as you can see on line 11, I expect there to be something called template available. But if I don't wait for it, this will actually just be undefined and it's my code is not gonna behave the way that I intended it to.

[00:19:01] So there are different ways to get around that and node JS or JavaScript in general, the standard way is to use a callback. So if we didn't use FS promises, and we decided to use FS, I would actually pass in a callback here like this. That would say, like error result.

[00:19:21] And this callback would be called when the read file was done. And then I would have to move all of this code into this callback because I don't want that code to be executed and so refile is done. And as you can see, that starts to get sloppy.

[00:19:37] You have callbacks inside of callbacks inside of callbacks. So we solve that with something called promises. So now if you have a promise, what you can do, instead of doing a callback is you could, you can actually just do a something called dot then right after this, which kind of looks like a callback, but it's not.

[00:19:55] And then I would just move everything inside of this dot then which it doesn't start, it doesn't expand out this way it stays flat but it's still not the best looking way, it still doesn't read top from bottom like when we look at code, we read it from top to bottom.

[00:20:10] And when it stops behaving that way, it can get very confusing and that then keeps it closer from top to bottom. But it's still not quite there. You still have these call backs inside the dot then. So we built another abstraction on top of that and that's called a sync await.

[00:20:25] So now if you have a function that does return a promise, which these functions do, they do return promises. You could prefix the promise with a keyword called await. Which will stop JavaScript from moving on until this asynchronous piece of code was done. It is the equivalent of moving all the code into the call back like I just showed you.

[00:20:48] Instead, visit usually, we don't have to do that and we could feel confident that this code won't execute until this is done or aired out.
>> Yeah, do we not have to provide the async keyword or is it implicit?
>> Gotcha. Yeah. So that is a new feature in the latest version of node, you do not have to apply the async keyword anymore inside of a function.

[00:21:09] Is your good to use it flat globally now it's pretty amazing. Yeah, if it's top level like I have here, you don't need an async because there's nowhere to add it. But if you do have a function, like for instance, if I have a function called, get users like this oops, go away.

[00:21:29] I can't just say await, something like this. I can't say await db.getusers. I can't do that, right? That's not gonna happen, because you have to put async in front of this, like that. So yeah, it's only the top level, which is something relatively new and extremely powerful. But yeah, if you're inside of a function, you do have to put the async keyword before your parameters on that function.