Using a Custom Trained Model

[00:00:00]
>> So kind of the same thing as project-1, where I have a starter HTML file, and we're going to kind of fill in the blanks and write. Except that this time there's no utils file, we're gonna write a lot of this from scratch. So there is a part 2 as well to this project.

[00:00:16]
And part 2 is where instead of teaching the model through teachable machine, we're going to do everything in our own application. But to get started, as more of the quick start, we just have three divs here. There's a Start button that's gonna kinda init everything. And then we have a webcam-container and a label-container so pretty, just small divs.

[00:00:42]
So if we open, let me just check that. Verify that here, the first file that's imported is part 1. It should be if you have not modified these files. So we're going to start writing our code. Okay, so if you have run npm install, you should now be able to run, npm run start.

[00:01:03]
And yes, okay, no error, that's cool. Okay, so here we just have a Start button, at the moment it does nothing, we have no interaction. Yeah, no interaction that we implemented. So the point of this example is, again, we're gonna start the webcam. I'm gonna do a little bit differently, because here, if you look at the index.html, there is actually a teachable machine image library that you can have in a script tag.

[00:01:32]
So import into your application and they have their own syntax in terms of toggling the webcam and taking a screen capture, so we're just gonna use what they're offering us, basically. So here, I'm gonna change the syntax a little bit, we're not importing here. We have the script tags there just to show you that you can do it in two different ways, but there's just the minified TensorFlow.js library and the minified teachable machine image.

[00:01:58]
Because, as I showed before, there's also one with sound and one with pose. Okay, so first of all, to be able to use our model, we also have to load it. So the first thing that we're gonna do is go and get the path to the file. So let's say, or maybe, yeah, I can call it path.

[00:02:17]
And then, for me here, I can see I called it my_model". So here, it's just going to be the folder that has the three different files here. And then we also want to be able to add interaction with a startButton, so we're going to get that button, I think it has idea of start.

[00:02:37]
So at the moment, we have the path to the folder, we have a startButton, and we want to be able to add an onclick event. So as we did before, we have startButton.onclick, we have arrow function and we are going to call init that we have not created yet, but we will.

[00:03:00]
Again, we're just kinda setting up the project. And now let's create the actual init function. So, we have, oops, init, and probably will actually be an asynchronous function, cuz I think, yes, it needs to be asynchronous function. Okay, as was mentioned very recently, in the my_model folder, there's three different files.

[00:03:29]
So here in the model.json file, what this has is more the description of the different layers of the model that was actually downloaded. So for example, here you have class name is sequential. So in terms of types of models that you can create with TensorFlow.js, there is sequential models and functional models.

[00:03:48]
I have not used functional models before, but sequential just means that you have layers that are in a sequence. So you start by feeding a lot of data to your first layer, and then it has an output that is fed to the second, and to the third, and how many layers that you have.

[00:04:02]
So here, oops, here, for example, you have layers that's an array in here, so it's a description of how your model was actually built in terms of its layers. And you have different parameters like here, Conv1, that must be a convolutional, like a Conv2D layer. So usually, it's what's working with images, and 2D means x and y, two-dimensional layer, and you have some parameters like number of filters.

[00:04:32]
I wouldn't know too much more about this, cuz obviously we did not write this model ourselves. It's a combination of what we're using the background of teachable machine, which is the MobileNet model, and we're adding our content on top of that. That might make a little bit more sense once we do part 2, where we're writing everything from scratch, but the model.json by itself is not everything.

[00:04:54]
What you also need is the metadata.json. And this is more of like it also has some parameters where it tells you the version of TensorFlow.js used, the package that it might need to actually use with and the labels that we were training. So right and left, so that then when you have the prediction from the model, it will know it's either right or left, it cannot predict something that it doesn't know.

[00:05:19]
And also, it was trained with an image size of 224. So that's kind of important, because when you're feeding images to the model, it will expect an image size of that number of pixels. And then we have the weights file here, but this is a binary file. So even if I was opening it with a X editor or something, that's not really readable, but this weights file is essential.

[00:05:44]
So if you thought, that's useless, I'm gonna remove it, you actually can't. It won't work, because I think it's in the, oop, not here. Yes, so here in your model.json file, it's describing how the model was built. And it will need this weights stat bin file, because that will tell it a little bit how the model was tweaked during the training process.

[00:06:09]
So as I was mentioning earlier on in the course, the weights are just adjusted as the model is being retrained. So even though we can't see what's in this weight span or if you were doing maybe binary reverse engineering or something like that, it still needs it. So even we're gonna import just the model.json file, but in the background, this file is going to kind of require the details that are in the weights file.

[00:06:34]
So now that we have these three files, though, we need to just specify the ones that we're gonna import in our application. So in our init function, we can have, oops, a variable called const modelPath. We called it path before, so let's just keep it like that. And that would be either path and model.json.

[00:06:59]
So because we're going into the, sorry, by the way, at the beginning, I forgot the end slash here. So if you've been following along, don't forget to pass in the slash after the name of the folder. So the path to the model is the path + the name, model.json.

[00:07:16]
If you decide to rename that file, don't forget to rename it here as well, and we're going to import the metadataPath as well, and that would be the same thing path, but two metadata.json file. Okay, so at the moment, it's just stored in variables, we haven't actually loaded anything in the browser yet.

[00:07:37]
Let's declare a model variable as well outside of the function, cuz we're gonna use it in a few different places. So we have a model variable, and here we're going to actually assign it the model being loaded. See it's an await function, and here, we're writing tmImage and that's coming from the teachable machine library that's impoted in the index.html file.

[00:08:01]
So we have tmImage.load, because it's always load. And then we are gonna pass it the two parameters. So the path to the model, so modelPath and the path to the metadata file as well. So at this point here, in the model variable should be our accessible like the actual model loaded in the browser so you can actually use it.

[00:08:26]
Then what we can do, there's a function that you can call on that model that's called getTotalClasses to kind of see that the total classes that we trained is two. In our case, we know the number of classes, because we trained the model. So I'm gonna write it anyway in case sometimes someone sends you a model that you have to use and you don't know how many classes there is even though you could see it from here, but let's just imagine that we want to write that anyway.

[00:08:55]
So we have maxPredictions or could also be, no, it could be maxPredictions, cuz it could either be one of the two label, so that would be model.getTotalClasses. So that's also a method that's available on the model when we're using tmImage. And then at this point, the model is loaded and we just have the number of total classes.

[00:09:24]
Okay, so the next step that we have to do is to get the webcam, because we're going to have to take pictures from it. So where you declared the model variable, we can also have webcam variable here. And we're going to assign it, webcam =. Again, it's going to be method that you can call from tmImage, so we're gonna instantiate a new webcam object, so a new tmImage.webcam with a capital M.

[00:09:57]
Don't forget that, otherwise it will not know what it is. And in that webcam, we are going to pass it, oops, a question?
>> Just real simple one, how tmImage is in scope because of the CDN calls additional?
>> Yes.
>> Okay.
>> Yeah, exactly, I didn't import it the way that I did before just to show that you could do it in multiple ways, you could definitely also have a package that you import, yeah.

[00:10:28]
But yes, yes, that's right. I think that when you import this, it adds, yes, it makes available a tmImage variable. Okay, so now that we kind of declared a new webcam, there's actually parameters that you need to pass and that will be the dimension of the webcam. So here in my particular example had 200 and 200 and true is flipping of, I mean, there's a third value, sorry.

[00:10:54]
I have it true, but I'm wondering if we should have true or false. It's like flipping the webcam. And in the previous example, we had it false, so let's start with false to keep it consistent. And if we had an issue because we just did right and left, maybe flip is going to, let's give it a try.

[00:11:12]
So I'll have to remember later that if there's an issue, it might be because of that. But for now, let's have it false, and we need to actually call a new function, that's setup. So I think that's going to, actually, this, it's an await. So await webcam.setup, and that's going to do things in the background that I'm actually, it's going to request access to the camera.

[00:11:37]
So the same way that we had it before, but more manually. This setup is making sure that it's requesting you for access, and you'll have to allow it if it asks you. If it doesn't ask you, you might have at some point allowed it for all websites or something like that.

[00:11:51]
But at this point, it's not actually rendering the webcam in the page. So what we're gonna do is await webcam.play, and that will actually start playing the webcam feed in the browser. And then after that, so even though you have the webcam set up and it will start streaming, we need to actually do something with that.

[00:12:14]
We need to start kind of continuously calling a function so that it will run continuously the prediction on the webcam output. So what we're gonna do, we just have a window.requestAnimationFrame, and we are gonna call it, I'm gonna try to find a word that's maybe, okay. We're gonna call it loop just to be maybe clear that it's looping.

[00:12:37]
But we have not declared that function yet, so we will. First, we need to append the camera onto the page, cuz at the moment, we instantiated on object. We are setting it up and playing it, but we're not appending the image coming back from the webcam to the page.

[00:12:58]
So here, if you look at the index.html file, this is what the webcam-container is. So we're going to do document.getElementById("webcam-container"), and we're going to append as a child, not just the webcam, but webcam.canvas, webcam.canvas. Okay, and at this point, that should be it for the beginning. So we have webcam, it plays, it's rendered.

[00:13:33]
But we have not declared our looping function.