Predictions from Live Webcam

[00:00:00]
>> So now if we think in terms of what's happening in the UI, it means that we've clicked on our train model, we see a little bit of output. And what we want to do is actually start predicting the live things that are happening on screen. So, we have a predict button that I just showed, so we're going to add an onClick event on it.

[00:00:20]
So predictButton.onclick, that's gonna be an async function as well. And, oops, not buttons, button, sorry. Okay, so predictButton.onclick, we're gonna start by having our pre, that is predicting set to true. And while we are predicting, because again, we wanna do it on the fly, we don't want to predict only one snapshot of the camera.

[00:00:51]
It's like, as we are moving, we want to get each frame and predict them. So we're gonna have a while loop and whiloPredicting, what we're going to do is reuse our getImage function as what that we declared before. So we can recreate a variable code image, and we can call again, our getImage function because it's going to do the same thing.

[00:01:12]
It's going to take live snapshots, but instead of storing them in memory to add examples, we're actually going to predict it right away. So first of all, we are going to make a prediction, first through Mobilenet. So what we're doing is we can call it initial model prediction or something like that.

[00:01:38]
And we're going to call initialModel.predict, because, if we remember, I mean, we haven't used mobile nets in the other example, we use Coco SSD, but it would be either predict or detect. But we're gonna pass our image in here, and if you remember the way that we're loading the mobile net, we're not loading the model itself, we're loading a layer.

[00:02:02]
So what's coming back here is not necessarily a label of like, left and right, it's called embeddings, and I can probably talk more about that tomorrow. But once we do a first prediction with our layer of mobile net, we can now pass it to the model that we just created before.

[00:02:19]
So here we're actually going to have access to our predictions, where we have our new model and we can call predict as well. And we are going to pass in the first kind of, what's coming back from the first pass in mobile net. So at this point, if we just log that, actually, there's a bit more code.

[00:02:43]
So to make it better, maybe let's not log it, let's keep going a little bit. And we're gonna want to extract the class of the predicted gesture. So we can do predictedClass, this is going to be coming from predictions. In the previous example, when I did some kind of calculation of like, math, max, and stuff, ao there's actually another way to do it, and I just wanted to show you different ways.

[00:03:15]
And you can call an as one-dimensional function, that is something that's available on predictions coming from a model in TensorFlow.js, and then you can also call argMax. So these are not like, native JavaScript methods, it's what can be called on predictions coming back from the model. So in actual new model, the custom way to do it, but there is there is a shorter way to do it, as well.

[00:03:38]
So this is going to return the predicted class of the argument that has the maximum probability. And then what I want to do is we're going to get maybe the ID of the class, I forgot why I wanted to do this. I think we're gonna log it. Okay, so let's just try, so we're gonna do classid, and what you can have is the predicted class.

[00:04:11]
So we're gonna do, wait, predictedClass. Yeah, there's a method called data, and we can look later about what that returns. But on this, we can get the first element, which should be, because predictedClass is the maximum, I think that function might reorder by probability. And here we're getting the first one because we're interested in the one with the maximum probability.

[00:04:35]
So what we can do it now Is then we're gonna log our labels, our array of left and right, and we're going to call classId. So this is going to be like index, I could have just called the index as well. So it's gonna be the index of the label with the highest probability, and we're getting it from our labels array.

[00:04:55]
And just to avoid keeping too much stuff, we're gonna call dispose again, img.dispose. And we also should call the function that we called previously, nextPrint. So again, every time it's kind of like waiting for requests admission frame to finish and then it's like running the rest. So at this point, I think that we're almost done, but now is gonna be the time where we wrote a bunch of code, I'm gonna try it and see if it actually works.

[00:05:28]
And if it doesn't, that's gonna be fun. So now at this point, if we refresh our project, I'm just going to open the console log over the console, so we can see the logs when they are coming. But what you should be able to do is, when you press, I'm gonna tilt my head left, I'm gonna press the button.

[00:05:51]
It's not working. I knew it. Okay, when coding with two arguments, the first argument must be a string. Okay, there's multiple errors, which is not fun. Okay, so I have an error with inner text. So there is somewhere where I am setting inner text, maybe an element that doesn't exist.

[00:06:08]
Is that the issue? Yes, so why does total not exist?
>> I think you need a dash.
>> You're right, thank you. Okay, I forgot the dash, so, if you followed as I wrote, yes, add the dash. Otherwise, if you use the solutions, then it should already be working for you.

[00:06:28]
And here, this one, tf.tidy, okay, so somewhere I used the tidy function and I must have, I know what it is. So here, the labels length is supposed to be the second argument to the one hot tensor here. I think that's it. So if I refresh, and now if I tilt my head left.

[00:06:53]
Yay, okay, so I'm gonna 54 examples. And if I do it right, Okay, and now if I train the model, so at this point, we should have images and labels kinda kept in memory, waiting to be fit into the model. So when we click on Train, Okay, it took a little bit of time, but you can see the loss and well, it went quickly, but it started as 0.8, which is a bit high, it's too close to 1.

[00:07:23]
And as it's going into 0.000 10, it means that the probability is much better. And because it stopped updating, it means that our model has finished training. So now if I click on Start Prediction, okay, I can see a label, so I have right, and then I have left, and right, and left, and that's it.

[00:07:44]
So basically, instead of using a teachable machine, where obviously it's much faster, all of this is actually, it's kind of basically very similar code, but in like a nice UI. You can actually create an application where you're letting your users do it right into your app as well.

[00:08:05]
So what's happening here is like, when I refresh them, the modal is gone, you'd have to retrain it. So there is, I think there is a way but I haven't done it right now. So I wouldn't be able to really show it to you, but I think there is a function called saveModel or maybe, no, I swear I wrote it.

[00:08:24]
Okay, or maybe it's called save. And you can save your newly trained model into some kind of local storage in the browser. And I think that then if you refresh your app, you can maybe fetch that model directly and have your app still running with the samples that you created without having to ask the user to retrain it every time.

[00:08:43]
So, when I was playing with transfer learning, that wasn't really a thing. So that seems to be, I don't know how new, but it seems like you are able to maybe do something like this. So in terms of calling the save method, we will do that later in the workshop, when we actually create our model, and we're gonna actually generate files that we're gonna be able to use.

[00:09:07]
But I think that's also possible to do when we do transfer learning.