Real-Time Web with Node.js Real-Time Web with Node.js

Piping Streams

Kyle Simpson
You Don't Know JS
Check out a free preview of the full Real-Time Web with Node.js course:
The "Piping Streams" Lesson is part of the full, Real-Time Web with Node.js course featured in this preview video. Here's what you'd learn in this lesson:

The output from one file stream can be piped to the input of another file stream. Piping typically uses both the read-stream and write-stream APIs. Kyle also demonstrate a great streams resource -

Get Unlimited Access Now

Transcript from the "Piping Streams" Lesson

>> [MUSIC]

>> Kyle Simpson: The last thing I want to show you about streams is that streams are pipeable. Which means, just in the same way that you think about on the Unix command line, on our command lines we can take the output of one command and pipe it into the input of another command, and then pipe the output to another input.

[00:00:21] We can do the same thing with streams, and that turns out to be a super powerful way of doing things. My simple little illustration of that is gonna seem kind of silly, but this is really powerful, the ability to pipe things into a chain. Very much in the same vein as how we're expressing our asynchronous flow control chains, you can express a series of pipes.

[00:00:42] So what I'm gonna do is I'm gonna say stream.pipe. Remember this is my read stream. I'm gonna pipe that stream to a file write stream. Remember we've got a create read stream, I'm going to pipe it to a file write stream. So I'm going to say, fs.createWriteStream, and I'm going to give it a different file name, which is filename+ a .backup extension.

>> Kyle Simpson: I'm not gonna change anything at all about lines 8 through 13 where I'm listening for the data events to happen, so those things I'm gonna continue to passively listen about. But I'm also saying in addition to that, I want you to pipe whatever you read in from this file, I want you to pipe it out to another file.

[00:01:33] In a very highly efficient manner, without me having to do any of that work, I just want you pipe it. Now, realistically, you probably aren't gonna do this. You're gonna read in a file, and pipe it to, you're going to read in a gzipped file and pipe it to a gunzipper, and then pipe those contents into another file.

[00:01:51] Or even read in a CSV file and pipe it into a CSV parser, and then pipe that into a zipper and then pipe that back to a file, or whatever. You're gonna read a file from an HTTP request and post it somewhere else using these streams. So, it just goes by taking a stream name and saying .pipe, and giving it the name, the target of another stream to pipe it into.

[00:02:17] And for the most part, it just handles those details. The devil's in the details, there's more to it than that. You sometimes have to understand which pipe you're dealing about and where your events are happening. For example, if I wanted to know when did it finish writing to the file output, I can't use .end because this is my read in.

[00:02:38] I would actually need to attach an event to the file writing thing to listen, I think it's for end or close, or something like that. So, you have to be a little bit cognizant of which stream you attach your events to, cuz it can get a little confusing when you start piping, which stream am I even talking about?

[00:02:57] But other than those sorts of implementation details, piping of streams is a really highly efficient way of transferring large amounts of data around without doing stuff like we did on line 8, where we're grabbing all the stuff into a string.
>> Kyle Simpson: All right, any more questions about streams?

[00:03:22] Let's test it. So I put that code in there, let's prove that it actually works. We know that we already have a test3.txt, but we don't have a backup yet. If I do node 3.js --file=test3.txt, we're gonna get the exact same behavior printing out our whole thing, but now when we look at our file system, we have an exact duplicate here of test3.txt.backup.

>> Kyle Simpson: Lots of fantastic blog posts out there about streams. I'm gonna show you one other tool that's really interesting. I just found it recently, but it's by a guy I'm sure none of you have ever heard of him, but his name is John Resig. And he recently wrote a blog post back last fall about it.

[00:04:20] But he put together this thing called the Node Stream Playground. And I just was fascinated playing around with this thing, so But it allows you to kind of visually see how you can plug streams together and what the code looks like to do those things. So it's sort of like a visual builder that helps you learn how streams and pipes work.

[00:04:40] So he provides a couple of options here. You can get a file from a file system, like we were doing, or you can get a file that's coming in from a request, like a POST request. Let's do it from the file system, so I'm gonna read a file in from the file system.

[00:04:54] And it shows you the code right there, it shows you what code to do, fs.createReadStream, just like we already did. And it shows you the output that you're gonna get, which is the contents of this people.csv file. Now, what if I wanted to pipe that into a write file, like we just did?

[00:05:11] If did addStream piping it into the write file, you'll notice that he did exactly what I did, fs.createReadStream.pipe to fs.createWriteStream. So you see the exact same code. But, let's say, maybe I don't want to pipe it to that. Maybe I want to pipe it to an unzipper. So now we show how to create a Gunzip stream using the built in zlib library, and that will unzip it on the fly.

[00:05:40] And then you can add more of these in here after you've Gunzipped it, now you want to parse the JSON or whatever. Of course the file that's coming in isn't actually JSON, so now we're starting to see errors, but you see how this tool helps you build your pipes of your streams, and shows you where they go and what to attach to what to.

[00:05:56] So I've found this to be a fantastic tool for learning streams. There's lots of other great books and blog posts out there, but streams are really probably top three most important concepts to understand in Node. I'm really just starting to get my head wrapped around them, honestly, myself.

[00:06:13] But definitely, make sure you do some more reading about streams, because the future of Node is definitely very stream based.