Loading the Model & Webcam

[00:00:00]
>> The first thing that we're gonna do, because we're gonna write kind of most of the transfer learning logic ourselves, we can start by in the index.html file, we're going to comment out the three lines that were part of part 1. So what's just under this comment part 1, we're gonna comment that out.

[00:00:18]
And we also will not need the two script tags at the top. So we're commenting out the teachable machine library and sensor flow. We'll import them separately, and we are going to comment in what's under part two. So as you can see these, there's a little bit more elements.

[00:00:37]
But because you know the way that we were looking at teachable machine, they were like buttons to start recording and stop recording. So we're gonna have to have that ourselves. For example here, I kind of hard-coded it to left and right to keep my theme of left tilting, left tilting, right.

[00:00:54]
There's gonna be a button that we're gonna press to record left samples, a button that we're gonna press to have right samples. Polls, and here this Div is just going to tell us the amount of examples we've trained so far. And here, as well, one important thing as well is change part 1 to part 2, so that we can write our code separately in the part2.js file.

[00:01:14]
So just to recap as well, comment out part 1, the script tags and comment, In part 2. So if we have a look at what's, let's try to run this. So if you go back to your terminal, it should still be the same comment. So if we go to local host, 1, 2, 3, 4, yes was yes, so that's normal.

[00:01:38]
So at the moment, we didn't write any logic. So that's normal, that it's just saying loading model, that's kinda the default state. And when it's loaded we'll just remove that div. Okay, so let's start, it's gonna be a bit more complicated because we're moving from kinda like the magical thing of like, as you use a tool that does everything for me, and now we're actually going to do it, and it's gonna be hopefully a nice segue onto building our custom model later on in project 3.

[00:02:07]
As we were doing in, mostly in project 1, we're gonna start by importing everything as tf from TensorFlow js. So from @tensorflow/tfjs. And for this one, when you're doing transfer learning, kind of implementing it from scratch you also need to import something called TensorFlow.js data, and for everything as tfd from, and then the package @tensorflow/, don't seem to have that.

[00:02:39]
Well, yeah, tensorFlow.js tfjs data. So the packages are important. And because again in this UI we're gonna have to have interaction with our buttons. And then let's start to create our variables that are gonna hold the reference to the better limit dependent elements. So at first in the index.html I have, yeah.

[00:03:03]
So I have a buttons container and instead of adding an onclick event to each separate button, I'm just going to get the whole container and then use event delegation. So we're gonna do record buttons, okay? I'm going to get document of, cannot type document dot get element by class name.

[00:03:27]
And why is that correct? Let me check. Okay, no, I did in an environment to spread it. Okay, that's fine, let's keep it that way. So each button has a record button class. So we're just gonna get both of them here. And then yes, I do have a buttons container that I also need because you'll see it a little bit later.

[00:03:55]
But if you want to go a bit faster, that's gonna be by ID, and that's gonna be buttons container, okay, so we have each button, we have the container, then we also have a button to run the training process. So we're gonna train button document, the get element by that's an ID of train and we also have a button to start the prediction, so we're going to call predict button.

[00:04:30]
Predict, and we have a status element to kind of give us an idea of like is the model, I think it is the model loaded or In the next step of the training process that is happening, so let's have a status element variable. And the ID for this one is just status, okay?

[00:04:53]
So this is I think all what we're going to need in terms of getting the elements from the index to HTML file. And now we're gonna start declaring some variables that we'll use along the way. So as before, because we are going to use the webcam, we're gonna have a variable called webcam.

[00:05:12]
And because transfer learning is creating a new model on top of an existing one, we're gonna have a variable called initial model. And because we're gonna track a mouse down event on the button to keep recording, we're gonna have a mouse down variable that we're gonna set to true and false to know when to stop recording.

[00:05:34]
And we're also going to have a new model variable that's gonna kind of keep the reference to the model that we create. So that hopefully should still make sense and we're gonna slowly go into the more complicated things. So at first, because we are here only training two different labels, we are going to start by creating a variable called totals.

[00:06:00]
And it's which is gonna be an array of numbers and 00 because when we start, we have not recorded any examples. So as we go, we're going to increment these numbers. It's just to be able to like display them on the screen. So if you refactor this later, you might not need this variable but at the moment in the way that I built, it made it a little bit easier to increment over time.

[00:06:23]
Then we also have our labels. And in my particular example, I used left and right. We can start with that, and then you can change it as you want. But that will be the labels that we're going to train. If you wanted it later on, you could have an input field in your HTML and let the users customize it in the UI.

[00:06:40]
But that would just be faster to do it that way here. And then we're going to start creating variables that are related to the way that the parameters of our model. So first we're going to create a variable called learning rate. And this is maybe going to sound weird, but we're going to set it to 0.0001.

[00:07:00]
And that's just the value I've been using in my project. But the learning rate is how frequently the model's weights are changed during training. So, as we go, if we have time or if you want to do that later, you can experiment with sending it to 0.1 or 0.5 and see how it actually makes it more accurate or not, depending on the amount of samples and all of that.

[00:07:24]
But for me, for my particular project here, that works, and we can play around with it later. Then, I was talking about batch sizes before, so we're gonna have a batch size here in the way that we're gonna pass it in wherever it needs to be used later, it has a fraction.

[00:07:41]
So we're gonna use 0.4, which is basically we're gonna yeah, a batch size is like, instead of saying 40, we're just gonna use 0.4. We could just divide it by 100 later, but that's just probably easier here. And in terms of Ebox, again, I used here 30, and that 30 steps to train the model.

[00:08:03]
You could have less steps, it will go faster, but it might not be as accurate, or you could have more steps and it might be more accurate, but it will also take longer. I picked 30, again, sometimes people say that when you tune these parameters they're more of an art than the science.

[00:08:19]
So let's say that if you use these exact parameters, but with more samples or less, you might actually get different results. So we get start with that, and then tweak it. And then we have a variable called dense units, and that is the number of outputs of the layer.

[00:08:37]
So every time that you have a layer in a machine learning model, there's an output that comes out of each layer that is then fed into another one and that's called dense units. So it's not maybe not super clear the way that certain terms are used in machine learning.

[00:08:52]
Hopefully when we actually use it later. Yeah, it will make a bit more sense. But let's say that in the model that we're gonna create, I believe there's only two or three layers, and the final layer will need a different type of dense units but we'll talk about that when we write it.

[00:09:08]
And then just a variable to set the stage of if it's currently training or not. So I just have like a simple is training that is the initial value will be false because when you load your app, it's not currently training and then is predicting, I think that's just to update the UI.

[00:09:28]
You can set it to false or just leave it. So let's see, I was like a initial step of our application. We imported the models, we get some elements from the page. We declare some variables that we're gonna need later, and so it's gonna be a bit easier to tweak if we instead of not declaring these variables and using them in place, we get to actually put a name on them and understand what they're used for and we can tweak them.

[00:09:50]
It makes it easier to tweak them later. So, as we've done throughout these projects, we need to start by loading the model. So, we are gonna have a load model function that's gonna be an async function. And the first thing that we're gonna do is, we are going to have to go and get the initial model that we need.

[00:10:12]
So that model is going to be MobileNet. So before we use COCO-SSD for transfer learning, seems like they're using MobileNet. So we're gonna keep it that way. So if we do, actually for this one, just to make sure that people get the URL, right, because we're getting it from in a remote storage.

[00:10:31]
We are going to cheat a little bit and we're going to go into the solutions folder. So if you go back to your terminal I'm gonna stop my server running. But if you go back to the roots of the project, I added a solutions folder, and inside this solutions folder there's the three projects that we're working on.

[00:10:49]
But if you could not look at it in advance, to not spoil everything, that would be great. But if you're a bit behind or anything, if you're worried that you're missing out, then it's fine. So we're gonna go into project 2. And if I open this, so that should be the solutions to project-2.

[00:11:12]
And all I want you to copy for now is this here from line 24 to 27. It's a URL to the Mobilenet Moodle stored, you know, in like Google Storage thing. So I didn't want to have to like, you know, dictate like the long URL. So let's just copy this and then the rest we can read it ourselves.

[00:11:32]
So just again to repeat solutions folder, a project two and then part two, line 24 to 26. And then you can Close that let's not look at the solutions in events. But again, it's fine if you do. Here, you can see that we're not putting load. We're putting load layers model.

[00:11:51]
And I think that this is a method that allows us to then extract the different layers of the machine learning model. So we're not loading the model itself. When we're loading, we're gonna load a specific layer of ID that we're gonna reuse to train our own model. And the way to do that is we're going to declare a variable called layer.

[00:12:10]
And specifically, they seem to be extracting some mobile net dot get layer and the one that we want is called conv underscore pwu, I'm not sure what pw will use for, and then 13 and relu. So I do not have the details about how Mobilenet was actually created.

[00:12:43]
And it might be the end layer of the whole training process that we're extracting to then train our own. I'm not sure about that one, but what we're gonna do then with that layer is we're going to return tf.model. And I think that's creating almost kind of like a new layer, new model out of that layer, right?

[00:13:05]
But right now I don't remember completely. So what we need, we have inputs, that's gonna be mobile net dot inputs, and then we have outputs. That's gonna be the layer.output. Okay, so for this, in terms of loading the initial model, that's it. So we loaded in mobilenet, we extracted a layer, and outside of that, we created kind of like a new model that we can work with when we're gonna create our own model.

[00:13:39]
So you'll see later, but this is going to end up being the first layer of our custom model. So we kind of build on top of Mobilenet. So for loading the model, that's it. And then we can have, as we did before, an init function. That's gonna be an async function.

[00:13:59]
And the first thing that we need, again, we're working with images. So we're going to get the webcam. And what's interesting here, so we did declare a webcam at the top, so we can just write it like this. So the webcam we're going to await, and this time it's going to be tfd.webcam because apparently here we imported tfts data as tfd and they have a method available called webcam as well.

[00:14:31]
And what you need to pass in is the webcam element from the index.html page, so what we have here so our video stream here that has the ID webcam. We actually need to pass it in there as well. So document.getelementbyid webcam. Okay, so at this point, we are starting to get the webcam and we are going to load our model onto a variable as well.

[00:15:01]
So we had our initial model variable that we declared before, and what we're gonna do is call or await our load model function so that now the initial model is actually stored in the correct variable. So at this point, that is done, and if you look at the here you know, at first I had like loading model to say that the page is not quite ready to do things.

[00:15:25]
Once the model has been loaded, then we can get the status element and we are just gonna do some simple style display is not. So it means that as soon as the model is loaded onto the page, we're just hiding that status message to kind of say, to the user like you're ready to go, basically.

[00:15:43]
And here what we can do as well is that we're going to be able to then at first here this div is displayed on. So we're hiding the kind of the elements that we're gonna need. And now we start to actually show them as well because the model is ready to go.

[00:16:03]
And we started loading the webcam. So if we just get that element, as we have been getting the others, we could have declared it outside as well but we're just going to need it here. Controller and style, display and block, so again, if you wanted to extract that in like a separate function to make it easier, that's that would be obviously possible as well.

[00:16:27]
But in terms of the, let's say initial state, that is pretty much it. So if you wanted to just call it here, we're not done with the logic, but if you just wanted to call it here so now that we loaded the model and we have the webcam.

[00:16:42]
Let's see if it works because at first we have the loading status of the model. And now if it works well but the model should be loaded, this kind of like status should disappear and we should see the webcam on screen with the rest of the of the divs that are there select train model and things like that.

[00:16:58]
So if I reopen, oops. I forgot which, oops. Okay, so 1, 2, 3, 4, yes, okay. So what's happening here, if you see if I refresh again, loading model, then it disappears. And then I have my camera here, it's maybe zoomed in a little bit, and at the moment we don't have the rest of the logic.

[00:17:20]
We don't have any handler for adding left, right, and right, starting or training, but that's kind of like the basic UI that you need. We're gonna add samples, then we're gonna train the model live and then we're gonna be able to actually start predictions.