Applying the Data Converter

[00:00:00]
>> Will Sentance: Okay, let's double check, and then we're gonna start to learn some of the fancy terms for all these pieces.z But let's double check our converter on some more sample data. So actually, from our sample where we know both the background information. And the label, let's get a bit we didn't actually use to come up with the converter to check whether the converter is working.

[00:00:29]
Because this is going to be a crucial part of developing our converters, making sure we don't get the converter to work perfectly for our sample. We literally tweaked it. We watched Maria go, shoot, it didn't quite work. Let's change it. Now, that might be capturing the underlying pattern of the overall population, the underlying rules implicitly of the overall population for converting background info into refund decision or it might be getting it perfect for the sample, but then doesn't work very well for the rest of population.

[00:01:10]
So we wanna check that we didn't use our sample only to get perfect conversion for our sample, but not for our overall population that may not be exact what it shouldn't be. If we only have 100% conversion on our sample, then unless our population is 100% identical to our sample, it is not going to necessarily be a great converter for our population.

[00:01:36]
We'll see more of this to come, but let's then for now, double check it on a new refund request, the way we do know what the customer service rep decided. So I think we're gonna have me. Yay. Me and unlike Joe. I did get my refund request. How very classes of me.

[00:02:06]
I'm very sorry, Joe. I certainly should have been the one who didn't. But there it is, refund one. Let's now go ahead and validate our converter. Validation. Let's put in, well, let's say I had two years, small refund and only one account on the IP and therefore the customer service rep, well, for maybe many reasons we don't know, said one.

[00:02:37]
Let's apply our conversion tool. Let's apply our converter. I'll just use this little blue. This is our converter we're applying. Nick, account longer than 0.9 years for Will? Yes. Dollars less than 200.
>> Ben: Yep.
>> Will Sentance: Yes. Account for IP less than three
>> Ben: Yes
>> Will Sentance: Yes. Do I get a refund?

[00:03:05]
>> Ben: Yes,
>> Will Sentance: Yes, and we know the customer service rep said so too. And so the accuracy on our validation data is one out of one. So for our converter trying to capture the patterns the data that is associated with giving a refund or not in our sample, we are doing rather nicely with our validation, with our checking of that on a new piece of data, a new instance, fancy word for bit, in this case, request of our sample, which we did not use to come up with the rules.

[00:03:48]
And we check it, and it does nicely. Now, in practice, people, we have 100,000 elements in our sample. We'd put 80,000 of them towards coming up with the right conversion values and maybe 20,000, 20% in the checking. Now, people, we can apply our converter to new refund requests where we do not have the customer service reps decision.

[00:04:15]
We do have a label and we can make a prediction. All right, let's do it. Let's do It. Who are we doing? We're doing Paul. Okay, pool. Paul is rather clever. Paul has their account for one year. Okay, looks good. How much are they spending? Refunding just to keep it under the threshold, Paul, what do we think?

[00:04:49]
1.99. 1.99, Paul thinks like a true criminal.
>> [LAUGH]
>> Will Sentance: How many accounts on Paul's IP? I mean, too many to count. Paul's running a full-on bot farm for refunds but we don't know whether to give a refund or not here. We're going to use our automated decision.

[00:05:14]
Each year goes by, my crouching on the whiteboard gets harder. All right, what decision will we get? Paul, greater than 0.9 years account history? You bet. Under 200? You bet. Accounts on IP less than three? Absolutely not. But with our converter two out of three is true, do we give poorly refund?

[00:05:46]
You bet we do. And this is the 150th refund Paul's got that day of a $199. Dear, we're experiencing here immediately a key challenge with any, we'll see the fancy name for our converter in a moment, but a key challenge with any of these, which is something called drift.

[00:06:10]
The notion of drift, our sample that we use to get a perfect conversion, perfectly captured the nature of our, so we even validated it. I know there are only four elements, and of course, in practice we'd use many many more and hopefully catch the nature of Paul's behavior, but we didn't, and so along comes Paul and our population is not reflective of our sample.

[00:06:36]
Paul's character, Paul's pattern of behavior has drifted away from our sample's characteristics. And actually this is a key part of deploying a model in production is maintaining or minimizing what's called concept drift, a movement away of the definition of the problem you're even looking at here or certainly a moving away, in this case of the characteristics of the population from the sample.

[00:07:05]
Or you could just say our sample is way too small and doesn't capture but definitionally, this is our sample. It is a match. Something is up Paul's. And by the way, this is a big part of fraud detection models. If you have people like Paul, who decide my goal is to work out almost pen test, penetration test, what these boundaries might be that I can get away with coming back again to vote manipulation, then you'll be looking for what are the maximums you can get to before you get shadow banned on Twitter or wherever, on X or whatever it is.

[00:07:48]
Okay, so people, we're gonna have thumbs on this because in a moment we're gonna go through and start to give the fancy names for each of these pieces. And then talk about how this plays into the discipline of machine learning. Wait, thumbs on taking our sample with our matching data for each instance, each person, each request, and identifying the rules to do the matching in our sample, then checking that that matching is a good set of rules on our validation data.

[00:08:27]
In fact, we check it on 20,000 we checked on one, and then being able to use it on a new future refund request of Paul's. And I know it's kind of lame that it then didn't work. It did work. It made a prediction. Is it a good one?

[00:08:41]
No, but it did make a prediction, very bad one. Perhaps because Paul is brilliant and pushed and drifted away the population nature. And actually, by the way, it's a huge part of production engineering for this space, because maintaining the similarity even just going from this being done with the customer service rep to this being automated is a shift in the nature of a refund request from I have to chat with someone to do it to I can press a button and get it.

[00:09:11]
That immediately leads half of the population to drift apart. But let's have thumbs on you lost me, I'm clear, I want clarifications, woo-woo, no clarifications, people are Paul, Nick, go ahead, Nick, please give us a clarification.
>> Ben: Is this always, like, a yes or no? Like, is there, like, weighted averages or anything applied to this kind of stuff?

[00:09:33]
>> Will Sentance: Yeah, absolutely. We can definitely, and we'll see, especially when we implement alternative models or alternative we'll see the name for them in a moment, alternative converters that we absolutely can get to some distribution of probabilities, some threshold amount, some levels of confidence. Absolutely, we'll see that play out in a moment.

[00:09:58]
Okay, how do we identify the boundaries? We actually already started pointing this with Ben. Ben, how did we go about identifying these boundaries with which part of your, actually, this is not a classy question. With which part of your face. Okay, that's not good. How did you go about identifying these boundaries, Ben?

[00:10:17]
>> Ben: Just looking at the lowest value.
>> Will Sentance: Yeah, okay.
>> Ben: That was accepted.
>> Will Sentance: Yeah, sorry to talk over you yeah, that's exactly right. Like looking at with one's eyes, with our eyes, that's possible for example four with well, four rules. Is that possible for 100,000 in the sample, we might have 100,000 data values attached to every single request?

[00:10:51]
What if I have a time series that's data over a period of time for every dollar spent by that customer. What if I have detailed information on their interaction with the app in the moments leading up to the request? These are all profoundly large data sets where every single one of them might have a boundary that is pertinent, might have a boundary where if they've spent this much money yesterday on the day before for this much less, that might be the important factor.

[00:11:25]
If they took these sorts of pattern of actions before they made the refund request, that might be the important factor. There's no way we can eyeball that. We're gonna need some techniques to do so.