The Hard Parts of Servers & Node.js Setting Up the Stream
Transcript from the "Setting Up the Stream" Lesson
>> Will Sentance: We have our tweets.json here. People, we're almost there. We have our tweets.json here in the file system. We're now gonna get super smart about how we go about accessing this running code to clean the tweet that we get out of it. Let's go everybody. Line one, what are we doing?
>> Will Sentance: What are we doing in line one there?
>> Virginia: We are declaring a label called cleantweets.
>> Will Sentance: Cleanedtweets.
>> Virginia: Setting it equal to.
>> Will Sentance: Big old empty string, excellent. Next line, Saruse?
>> Saruse: So we are declaring a function cleantweets.
>> Will Sentance: Cleantweets, exactly. Cleantweets, is a whole function definition.
[00:00:44] We're just saving code to probably be auto run, no, we're gonna run that one later on. Okay, next line William?
>> William: Declaring a function called do our new batch.
>> Will Sentance: A-hah. We're never gonna be the one's running this function, this looks like it's gonna be our auto run function.
[00:01:02] Do our new batch excellent, thank you William. Rich, next function next line sorry. Left hand side.
>> William: Declaring a constant called Access tweets archive.
>> Will Sentance: Okay people. Now I want you to all, to access tweets archive, AccessTweetsArchive. Let's make sure our call stacks set up. There it is.
[00:01:28] Okay, AccessTweetAchieve. And the right hand side says to use which node C++ feature, Zeph?
>> Zeph: FS
>> Will Sentance: FS, exactly. And specifically, make a connection to tweets.json. So, access tweets archives run fs dot, this create,
[00:02:22] The create read stream, do you think with this piece of information about where to go look? What's it roughly gonna do in node, do you think?
>> Charlie: Tell the file system to load that file.
>> Will Sentance: Fantastic. Exactly. Actually it's gonna say to libuv, to set up a dedicated thread to go and get this data and start pulling it.
>> Sam: Object with methods.
>> Will Sentance: That
>> Sam: That will tell node to do certain stuff.
[00:03:30] I should said actually. FS was set up to go look for.
>> Will Sentance: Thanks everyone for staying so attentive on what. Hopefully now it's starting to feel boring, in a sense because Matt nodded fortunately just before I said the word boring though it wasn't actually a response perhaps. So, he decided to feel boring because it is just the same thing again and again at this point but all of node is this again, okay.
[00:04:06] That's the only thing to me that's not this model of trigger, background feature, set up auto run function, have it auto run when the background feature finishes work from the file system or whatever or the inbound message. Have auto inserted data from node, again, and again, and again, and again.
[00:04:23] Okay, good. All good, now we an auto run function, and we need it to run on a certain of event, certain event. An event that's trigger every 64K batch, 64,000 characters from the Json file. And that, is gonna happen in the very next line, we gonna use access tweets Archive which is the object full of of methods that give us access to this background feature let's make sure that's clear.
[00:04:54] Just a little star there to show that the accesses is, this particular background feature of FS not feature sorry. This particular instance of using FS in node. The one that's set up to access this file. And we know that, we should say that fsreadstream does this automatically at 64 kilobytes.
[00:05:15] Shout out, what event, Zeph?
>> Zeph: Data.
>> Will Sentance: Data, there it is. That's what we're gonna write, that's done by fsreadstream by default. We can change that, by the way. They call it the high water mark. We can change that. After the docs, but you change it by passing two read stream as second argument, or the first argument that it will ignore, there's nothing there.
[00:05:37] Which is an object with a property you can put high watermark is and some value, which will set up what the batches are, okay. To optimize, make sure you know you're optimizing. When you run the function when the data then gets screened. Okay, but now let's use AccessTweetsArchive, AccessTweetsArchive.on, and determine what function we want node auto run when that data event at 64 kilobytes gets triggered, or gets screened, sorry, emitted, emitted, there it is, technical term, not screened, emitted, or broadcast, they're both good terms for it.
[00:06:18] And then pass what function, what function do we want to have auto run, can you spot?
>> Sam: Do on new batch.
>> Will Sentance: Do on new batch, thank you, man. Do on new batch, we pass all of it's data in. By the way, just as a side note, everybody who actually happens is we pass a reference, this is a link.
[00:06:35] Back to where this function was saved, just in case anyone thinks we're copying all the codes into here, we're just actually saying, make a link up to this function, have it run just so we know, just a little side note. Okay, in it comes, so on the data event being screened auto run do on new batch, there it is.