Human in the Loop

[00:00:00]
>> Scott Moss:human-in-the-loop. This means a lot when it comes to AI. It's not just about what we're going to be talking about, but a lot of different things. So, human-in-the-loop is different on the context. So when it comes to like training and fine-tuning, you might see something like RLHF. Anybody seen that before? Reinforcement learning, human feedback. It is the technique on which we train LLMs today, reinforcement learning, human feedback.

[00:00:27]
So, reinforcement learning is essentially the process of like, I'm going to give this model all of this labeled data, right? It's kind of like our emails. It's like, here's some inputs on some data and then we understand the outputs of it. So what we do, because somebody labeled it, a group of people, an army of people labeled this data essentially, what we're going to do is we have this model that is raw.

[00:00:51]
It's neural networks have not been tuned, the layers have not been tuned, the weights have not been adjusted, it's just basically raw. We have to adjust the weights. A weight is a fork in the road in which a path can be taken and how you adjust it between 0 and 1 determines what other node it goes to and eventually those nodes lead to some statistical outcome that gives you a token, right? So, the reinforcement learning is exactly what it sounds like.

[00:01:20]
We want to reinforce something with you, so we're trying to train you on how to learn what a picture of a dog is, we'll feed you pictures of a dog over and over and we'll ask you like, was this a dog? Yes or no, and if you get it right, we'll reward you. How do we reward you? By not changing your weights. We won't adjust them, but if you get it wrong, we will punish you, right? And how do we punish you?

[00:01:53]
We'll change your weights, right? And the model will keep doing that over and over and over until statistically it will get better at predicting the thing that you're asking it to predict, that's reinforcement learning, reinforcement learning. The human feedback part is exactly what it sounds like because you need a human involved to give the AI feedback as far as like the reward and the punishment, right?

[00:02:19]
So only a human could get involved whereas like actually technically I guess like right now they are using LLMs to do a lot of this stuff too with like, don't get me started on that, but like they're using LLMs as like the human-in-the-loop at this point, but you might hear human-in-the-loop in that regard, that is not what I'm talking about. You can read more about reinforcement learning, human feedback.

[00:02:41]
Evaluation and quality control, you will hear humans in the loop when it comes to evals. I talked about this earlier, where you can have like a human eval, where it's like, oh, a human, a subject matter expert needs to look at this because we have no idea if this is good or not. You might, that's another place you might hear someone say human-in-the-loop. Active learning. This is when a model encounters uncertain predictions, it routes those cases to a human for labeling.

[00:03:05]
So if you have a very specific model, this might be outside the context of an LLM, but traditional AI that's like trained to spot, you know, I don't know, Alzheimer's in brain scans, and then you fed it a bone, a kneecap X-ray, it might be like, yeah, I'm going to ask a human what this is. I've only looked at things that look like, you know, brain scans. What is this? So in that case, a human might be brought up to label what this is and then feed it back to the AI like, oh, this right here, this is a knee joint and the AI is like, oh, OK, cool.

[00:03:46]
Thank you. Now I know that that's a knee joint, right? The one that we're going to be doing is runtime approvals. So this is just like agents requesting human approval before executing a certain action. So in the case of us, like things like deleting a file or I don't know, I guess running any terminal command ever, like you need my approval, right? So, and I don't want the agent to decide when that approval happens.

[00:04:13]
I want it to be deterministic every single time. So that's what we're going to do. Why do approvals matter? I hope I don't have to tell you why they matter. They clearly matter, but one that you might not know is, I didn't put it on here, is true freedom. I don't like, I think LLMs have the power to free us from monotonous, time-consuming tasks that we don't want to do. But because of this issue right here, trust, because we are a generation of people who knows what it was like before LLMs, the new generation trusts LLMs entirely like they just, it's just like we trusted Google, they trust LLMs.

[00:04:57]
So they will grow to not, but right now they do. We don't trust it because we know what the world was like before LLMs, so because of that, we aren't actually free from these tasks, we've just offloaded it to watching someone else do it. It's not actual freedom, it's just like, I went from typing on my keyboard to watching somebody else do it and then like typing on my keyboard sometimes. Well, if there was some way that I can guarantee nothing bad would happen if I wasn't sitting at my computer or bound to some device, that would free me up a lot to do other things.

[00:05:33]
Other work or go watch TV, something else, like anything else. So I feel like having those approvals really matter. Like it's great that like LLMs are like supplementing us and making us 20X engineers. But I'm still sitting in my editor watching cursor work just so I can like see what's going on, like it'd be really great if this thing could just work in the background and I can get an approval. Oh, turns out that's how we work as engineers anyway.

[00:06:00]
We have PRs and we have, you know, code reviews. Someone else other than you working on the same codebase as you is doing something on their own time with some objective that you probably know about because you're all on the same team, and then at some point randomly that you don't know about, they'll ask for a code review and then you'll get to it when you get to it, and then you'll review it and you'll say yes or no and fix this or don't fix that and then eventually that process happens over time and then it gets approved and it gets merged in.

[00:06:31]
What if everything that you needed to do was like that, right? Without the burden of like this person's really waiting on me, I feel bad. No, the agent's not going to feel bad, OK? That's where I think human approvals really matter. It actually is the next level of unlocking the productivity that we're being promised that LLMs will give us, but we're still bound to chat interfaces that require us to sit here and wait for tokens to stream just to be like, why'd you do that?

[00:07:01]
Right? That's not true freedom, that's not true productivity gains, we're just trading. Like the ratio goes from like one, you know, for every one thing that we did, we're doing like 0.75 things. It's still pretty high. I would rather it be like 0.25 things, right? 25% of what we were doing, we're no longer doing. Now it's like we're still doing 75% of it. It's just a different type of work. So human-in-the-loop, in my opinion, is one of the things that we need to do to get us to that next level of true productivity gains.