Recording Examples

[00:00:00]
>> So to record samples, we're going to go back to the code, I mean, we can leave it in there and write it below, that's fine. So we had our buttonsContainer. So here we have buttonsContainer, and we gonna have an on mouse down event, because we're going to steal, we going want to keep running the logic while the mouse is down.

[00:00:22]
So that we don't have to do click record, click record that would be annoying. So if we do on mouse down and then we have an event object and the way that I did it here pretty simply is that if the target of the event is the first of our record buttons.

[00:00:43]
Then if we look at the html file, the first is the left sample. So again, that's not very dynamic, that's kind of hard coded, but you could update the UI later. And here what we're going to do is we're going to call a function we haven't written yet, but that's called handleAddExample and we're passing it 0 as kind of like our label for the left gesture.

[00:01:08]
And if it's not, if the button that is pressed is not the first one, it's the second one, then we can handleAddExample1. So sometimes as well, your labels instead of being strings, they're being transformed into a number. So you have 0, 1, 2 and things like that, depending on the index in the array of labels that you have here.

[00:01:33]
So at the moment, when we press the button is going to call this function, but we have not read enough function yet. Before we do that, actually not before we are going to create our handle AddExample now. So if we do const handleAddExample, this is going to be an async function and we pass it the label index, so label index.

[00:02:01]
So what this is going to do, first of all, we had a mouse down variable. We're gonna set it to true because while we're clicking or mouse downing, we're gonna set it to true because then we're gonna have a loop lock. So as long as you're clicking on the button, then we're gonna addExamples.

[00:02:20]
But first, we had a total, basically in the UI, we want to update to this total as we're adding examples as well. So we're gonna have just a total variable and what we're going to do is, Yes, okay? So let me just check something, I forgot what I was doing here.

[00:02:50]
So document.getElementsById. And here at the beginning we declared our labels variable with left and right. So we're just gonna do labels and then the label index because if it's 0, then it will be left and we're just doing that. So it's just a dynamic way of saying we're getting this right total instead of left total.

[00:03:14]
What you could do instead is to get these elements separately and say, this is the left example, right example, and you change them over time just to be a bit more dynamic. We're getting the labels and we're getting the idea of the element based on that. It's just maybe a bit more overcomplicated, but you can rewrite it if you want later.

[00:03:35]
So we were saying that while we're clicking on the mouse, we want to add example. So we gonna have while mouseDown. We're gonna call the function we have not written yet either. But here we're gonna no, careful with auto complete we're not calling handleAddExample again, but addExample, and we're passing it the index as well.

[00:03:57]
And we want to update our total variable to be able to be one example, two examples, three examples, for as long as we're pressing on the button. So we have our div, and we are changing the inner text, so total.innerText, and we're going to ++ our totals that was declared at the top, the array with 0, 0, and we are increasing the total that was in this array.

[00:04:25]
And we're putting that into our variable here. So what that means is that when you're gonna click is going to look at that total array of 0, 0 is gonna increment the one that I'm currently selecting. And, adding that number into the variable. And something that you that is available with TensorFlow just as well is something called nextFrame.

[00:04:46]
And that basically returns a promise when request animation frame is computed. So that it's kind of maybe allows it to not just kind of block your browser. Okay, so we have the first handler of the AddExample, but we created here a function called addExample that we have not yet created.

[00:05:11]
And that might start to become a little bit more complicated. So let's see what we can do with this. So we create the addExample function here, and that's also going to be an async function. And as before, we passed it the label index, I'm just gonna call it index.

[00:05:30]
And when we're adding example, what this means is that we're taking a snapshot of the camera and feeding it into the model. So the first thing that we need to do is get this snapshot of the camera. So I just created a separate function called getImage. And maybe before we keep writing what's in the addExample, let's create that getImage so that we kind of have an idea of what it does.

[00:06:00]
So we have const, getImage to be an async function as well. And here, if we were looking again at the way we declared the camera or the webcam, here it was with tfd.webcam. So there is a method available on it that's called capture. So the image here is, we're going to await, webcam.capture, and that's going to be the image captured from the webcam feed.

[00:06:31]
So at this point, we have an image, but usually what you have to do is you might do a little bit of resizing or process the image. So what we can do is const processed image or resized image or anything. And so the way that it's done in my example, is I'm introducing a new function that I have not talked about yet that's called a tf.tidy and it's basically a way to make sure that you're not keeping too many things in memory.

[00:07:03]
So it's kinda cleaning up after it's done, because when we're taking the image, we're fitting it to the model, but we don't need to keep that image in memory cuz what we're gonna need is the model. We don't need to keep all the image. So tf.tidy on its own it actually takes a function.

[00:07:19]
So we can do this and we are going to call a few different methods that I maybe can explain more tomorrow, where we have the image. And we need to expand it's dimensions because the model might have been trained with four dimensions and maybe this image has only one.

[00:07:39]
So in terms of tensor, sometimes you also have to give it the dimension of the tensor. So if you have a model that was trained with the dimension like three numbers and you actually give it something with four, it's not going to be able to run with it.

[00:07:52]
So I can go further into that tomorrow cuz we're gonna go a little bit deeper into it. But then we called, toFloat. So whatever is going to be returned from this, we want to transform it into float numbers. And then we divide by 127 and I think that it's a transformation thing on the image and then I had .sub1.

[00:08:19]
And so I can get back into this kind of thing, we're gonna write something kind of similar to Mura as well. But these are all methods that are used to transform the original image that we have. What you can do later, if you want, is to try to see what happens when you don't use these methods.

[00:08:34]
And it's probably going to return an error because the model was expecting a certain shape, and you need to figure out what that is and transform your current image into that. So what we are doing here then is that we are going to need the processed image, but we don't need to keep the image created from the capture in memory.

[00:08:51]
So we do is dispose. So we are disposing here of the original image and we're just returning the processed image. So the image that has been resized and optimized to work with our model. So this is our getImage function. So now that we've written it here, we can go back into our addExample function here.

[00:09:15]
So once we get an image from the webcam. We are starting by feeding it to the original model to get kind of an example label, so we can have example here as a variable, and we are gonna call predict on our initial model that we created above. So we do predict, and we pass it the image that we just got from the webcam.

[00:09:42]
And at this point, this example, variable is going to be kind of kept in memory and concatenated with subsequent examples until where it's actually ready to be used to create a totally new model. So maybe as I'm talking about it right now, it doesn't make the most sense, but as we're gonna put it together, hopefully it will a little bit more.

[00:10:06]
Just before I add the example function, what we can do is create, let's say, I wanted to call it maybe examples and labels, but that might be confusing or that might be conflicting with other variables later on. So the way that they sometimes shrink it is to just like xs and xy, but that is basically going to be, xs is going to be the actual example data and xy is going to be the labels attached to these pieces of data.

[00:10:40]
So we can do it like this and then at first, we're going to create a y variable maybe y isn't optimized I'm not quite sure how to call it right now. Let's start with y and then maybe you can rename it later. But again, we're gonna write a function inside the tf.tidy function to again clean it up once we're done with what we're doing here.

[00:11:08]
We're gonna create what's called a one-hot tensor and that is basically turning yeah, categorical data into numerical data. So instead of having to feed it like right and left is just gonna be 0 and 1 and numbers starting from 0. So we have one-hot, it's called one-hot tensor, and you have to declare your tensor inside it as well.

[00:11:35]
So we have tf.tensor1d and here it just means that the data is written as a one-dimensional array. We don't need to actually create, we can have tensor 2D, 3D, 4D, just depending on how complex your data is. But here for the image, we're just transforming the image into a very long array of just one-dimensional data.

[00:11:56]
And what we need to pass in it is the index because we're creating one- dimensional tensor of just the index for that image. I'm not quite sure, there might be different ways to do this as well cuz that kind of seems maybe over computed. So maybe I'll see tomorrow if we can simplify a little bit but for now, it starts with this, and you have to call toInteger on it.

[00:12:24]
Sorry, the toInteger is after the oneHot, so it's the oneHot tensor that we are converting to integers. And then we're passing labels.length. So what's gonna end up happening here is that I think as a result, we should get an array of maybe it's gonna be something like 0 and then labels length will be 2 and things like that, okay?

[00:12:51]
So what do we do with that variable? We'll see that, shortly. But we had an xs and xy here. So first of all, if these are empty, because they are not set at first, oops, so if they are null, we have not yet added any example to create our model.

[00:13:16]
So the first thing we're gonna do, we're gonna set a value, to xs and it's going to be tf.keep the example. So here the example that we had is a prediction from the model. I mean, .keep is kind of a way to keep it in memory and we're going to concatenate everything in the else statement.

[00:13:36]
But at first access is set to the prediction from the example and xy is going to be the label. So we're going to keep the label that we just declared above. And in the else statement, so if one example has already been added, then we are going to concatenate our examples all together before training the model.

[00:13:58]
So here we can have a previous x, and that will be equal to access. And we're going to redefine x or add a new value to it, and that's going to be our tf.keep again, but we're going to concatenate so we're using our previous version. My notes are annoying, okay?

[00:14:19]
Our previous version of sorry, previous x, sorry. And we're going to concat in it, our new example because while we're at this point, we have the example here. That's a prediction again, but we're going into the else statement this time. And another one of this is a 0, I forgot what this second value is.

[00:14:43]
Let's move on for now and we're doing the same for the y value as well. So the labels, so we can kind of copy but change it from xs to xy, and then xy here. And then instead of previous x, we're concatenating the labels. And instead of example is going to be y.

[00:15:07]
So in the end, xs is going to be a big concatenation of all the examples, and then xy is going to be all the concatenation from the labels. And again, we don't want to keep too much stuff in memory, so we're going to dispose of them. So previousX.dispose, and then previousY.dispose and then we can also dispose of y here.

[00:15:39]
Oops, dispose and we can also dispose of the image, okay? So at this point, this is the main function that's adding all the examples together into some kind of data that we're gonna be able to use later on.