Introduction

[00:00:00]
>> Welcome to my workshop on machine learning in JavaScript. My name's Charlie Gerard, I'm a staff engineer at Socket and overall, I'm a self-proclaimed creative technologist. So what that means is that I like to push the boundaries of what can be done with technology and more specifically with JavaScript.

[00:00:16]
So I've been doing front-end development for the past ten years. And on my personal time, I like to explore and see what else I can do with JavaScript. So on your day-to-day, if you're currently working as a front-end dev, you're probably building business minded applications. But there's actually a lot more things that you can do with JavaScript, and one of the things is machine learning.

[00:00:36]
So I'm really excited to be here today with you and to show you kind of what I've been doing with TensorFlow.js, over the past six years now. I think the first time that I actually built a little TensorFlow.js application was in 2018. So before we get started, I kind of wanted to maybe show you a few things that I've built to kind of give you an idea of what is possible.

[00:00:54]
So I think maybe on the kind of more useful side, you can build a dark mode clap extension. So you can use a machine learning model and train it with the sound of you clapping. So if you do clap clap and then it just turns on, [LAUGH] the dark mode and light mode, I was doing this when I was working at Netlify.

[00:01:12]
So it was an extension that was customized to just work on the Netlify website, but it's something that you could do as well. Another one of this is maybe also on the useful side is an application, a little mobile, a website that you that was kind of optimized for mobile.

[00:01:27]
Where you could take pictures of things around you and kind of tell you which bin you should throw it into, [LAUGH] like recycle. But that depends on the regulations of wherever you're living, but it was using image detection and then you can find public resources of your city and tells you where you should throw certain things.

[00:01:47]
And then another one on the more kind of creative side is to actually build a custom machine learning model with your own body movements using a piece of hardware. And just recreate a prototype of a street fighter game, so you play air street fighters, you're punching the air and then it makes your character do things.

[00:02:04]
So it is a prototype, it's not like a real street fighter. There's only one character that reflects my movements. I could push it further and also make it like a real fighting game. But it's also something that you can do with TensorFlow.js. You gather live data from movements and then you create your own model, and you use the output to control an interface as well.

[00:02:23]
And in the same kind of in a more pre-trained model thing, it's like a clone of Fruit Ninja that uses body movements to play as well. So it uses Three.js as well, so it's all in client side JavaScript where you track your body movements and then you actually apply that to a 3D world.

[00:02:42]
And you can see I'm not really good at it in this demo, but you can see that as I'm moving, there's a bit of a delay. But it's a game where you can play Fruit Ninja instead of having to exploit on a screen, you can play it with your hands in front of your webcam.

[00:02:55]
And maybe just like one more before we move on, it's also experimenting with building UIs with body movement as well. So Figma allows you to create some plugins and with a bit of a hacky way, you can actually have it as well. Use TensorFlow.js to track, let's say the movements of your body on the camera, and then you apply that to layers on Figma.

[00:03:17]
And I mean, obviously, it's still a prototype as well. But I like to experiment with seeing what else you could do, then just type on the keyboard and use the mouse to be able to create experiences. And this is not a website, this is also on Figma, so it's using JavaScript to do things that are even outside of the browser.

[00:03:36]
So hopefully by the end of this workshop, you'll understand better how to build things like this. You don't have to be doing creative things at all, you could do more useful things for your job, but it's just understanding what TensorFlow can offer you. And how to actually in practice put that together and then however creative you wanna be will be up to you.

[00:03:58]
In terms of prerequisites, it's just hopefully things that you're kind of working with every day, a git, Node, a code editor, I use VS code, and your browser, I use Chrome, but there is hopefully nothing more that you will need. Everything will be in the repository, we'll just run npm install and then write some code.

[00:04:17]
So throughout this workshop, we're going to learn about three main features for TensorFlow.js. So there is using pre-trained models, then doing transfer learning. I think at the moment with more of a AI movement, sometimes it's called RAG for retrieval augmented generation. I'm not sure about why we call it RAG now, but it feels so similar to transfer learning.

[00:04:40]
In the context of Tensor.js, we use transfer, well, I'm gonna call it transfer learning. And it's basically using a pre-trained model, but adding custom input. So if you're using an image recognition model, but it's not trained with the images I want to predict, then you can kinda merge these together and create an application as well.

[00:04:58]
And at the end of this workshop, we're going to create our own custom model. So from scratch, we're going to create an object, or not object detection, but image detection model. So we're gonna gather our data, split it between the training set and the testing set. We're gonna write a model, talk about layers and things like that.

[00:05:16]
And in the end, we'll have an application in the browser where we can draw certain things and feed that to a machine learning model and get a label out of it. And then we'll talk about where you can go from there as well.