Setting Up the Stream

[00:00:00]
>> Will Sentance: We have our tweets.json here. People, we're almost there. We have our tweets.json here in the file system. We're now gonna get super smart about how we go about accessing this running code to clean the tweet that we get out of it. Let's go everybody. Line one, what are we doing?

[00:00:20]
Virginia.
>> Will Sentance: What are we doing in line one there?
>> Virginia: We are declaring a label called cleantweets.
>> Will Sentance: Cleanedtweets.
>> Virginia: Setting it equal to.
>> Will Sentance: Big old empty string, excellent. Next line, Saruse?
>> Saruse: So we are declaring a function cleantweets.
>> Will Sentance: Cleantweets, exactly. Cleantweets, is a whole function definition.

[00:00:44]
We're just saving code to probably be auto run, no, we're gonna run that one later on. Okay, next line William?
>> William: Declaring a function called do our new batch.
>> Will Sentance: A-hah. We're never gonna be the one's running this function, this looks like it's gonna be our auto run function.

[00:01:02]
Do our new batch excellent, thank you William. Rich, next function next line sorry. Left hand side.
>> William: Declaring a constant called Access tweets archive.
>> Will Sentance: Okay people. Now I want you to all, to access tweets archive, AccessTweetsArchive. Let's make sure our call stacks set up. There it is.

[00:01:28]
Okay, AccessTweetAchieve. And the right hand side says to use which node C++ feature, Zeph?
>> Zeph: FS
>> Will Sentance: FS, exactly. And specifically, make a connection to tweets.json. So, access tweets archives run fs dot, this create,
>> Will Sentance: Read stream. There it is. And pass it, tweetsArchive.json. Just to bring a character at this point to javascript, but you know it means something.

[00:02:08]
Okay, this is gonna do one thing in JavaScript and one thing a node. Raise your hand if you want to tell me, roughly what it's gonna do in node? Charlie what's it rough you gonna do? There you go, you didn't even raise your hand. Charlie, what it's roughly gonna do in node?

[00:02:22]
The create read stream, do you think with this piece of information about where to go look? What's it roughly gonna do in node, do you think?
>> Charlie: Tell the file system to load that file.
>> Will Sentance: Fantastic. Exactly. Actually it's gonna say to libuv, to set up a dedicated thread to go and get this data and start pulling it.

[00:02:41]
>> Will Sentance: .Okay, what's he gonna do in JavaScript? Do you remember that thing we dwell on for quite a while? It doesn't JavaScript. Raise your hand if you can tell me, what's it gonna to return now in JavaScript immediately? Yes. Sam.
>> Sam: Object with methods.
>> Will Sentance: That
>> Sam: That will tell node to do certain stuff.

[00:03:01]
>> Will Sentance: Yeah, to edit what we just set up. Fantastic very nice from Sam. Yeah, that's a real tricky one. That's honestly really really tricky that bit. I think yeah so it's gonna mean JavaScript return out in JavaScript in to accessories archives an object, full of functions, methods that have a connection to this particular open stream of data or open access point of this particular file.

[00:03:30]
I should said actually. FS was set up to go look for.
>> Will Sentance: Thanks everyone for staying so attentive on what. Hopefully now it's starting to feel boring, in a sense because Matt nodded fortunately just before I said the word boring though it wasn't actually a response perhaps. So, he decided to feel boring because it is just the same thing again and again at this point but all of node is this again, okay.

[00:03:52]
The only thing that's not like this in node, is that you can technically split up with your JavaScript code into different files. And then bring them all back together using the require and the module pattern, we're not gonna get to cover today, it's pretty clear at this point, but we will cover our future talk.

[00:04:06]
That's the only thing to me that's not this model of trigger, background feature, set up auto run function, have it auto run when the background feature finishes work from the file system or whatever or the inbound message. Have auto inserted data from node, again, and again, and again, and again.

[00:04:23]
Okay, good. All good, now we an auto run function, and we need it to run on a certain of event, certain event. An event that's trigger every 64K batch, 64,000 characters from the Json file. And that, is gonna happen in the very next line, we gonna use access tweets Archive which is the object full of of methods that give us access to this background feature let's make sure that's clear.

[00:04:54]
Just a little star there to show that the accesses is, this particular background feature of FS not feature sorry. This particular instance of using FS in node. The one that's set up to access this file. And we know that, we should say that fsreadstream does this automatically at 64 kilobytes.

[00:05:15]
Shout out, what event, Zeph?
>> Zeph: Data.
>> Will Sentance: Data, there it is. That's what we're gonna write, that's done by fsreadstream by default. We can change that, by the way. They call it the high water mark. We can change that. After the docs, but you change it by passing two read stream as second argument, or the first argument that it will ignore, there's nothing there.

[00:05:37]
Which is an object with a property you can put high watermark is and some value, which will set up what the batches are, okay. To optimize, make sure you know you're optimizing. When you run the function when the data then gets screened. Okay, but now let's use AccessTweetsArchive, AccessTweetsArchive.on, and determine what function we want node auto run when that data event at 64 kilobytes gets triggered, or gets screened, sorry, emitted, emitted, there it is, technical term, not screened, emitted, or broadcast, they're both good terms for it.

[00:06:18]
And then pass what function, what function do we want to have auto run, can you spot?
>> Sam: Do on new batch.
>> Will Sentance: Do on new batch, thank you, man. Do on new batch, we pass all of it's data in. By the way, just as a side note, everybody who actually happens is we pass a reference, this is a link.

[00:06:35]
Back to where this function was saved, just in case anyone thinks we're copying all the codes into here, we're just actually saying, make a link up to this function, have it run just so we know, just a little side note. Okay, in it comes, so on the data event being screened auto run do on new batch, there it is.

[00:07:01]
Okay.