Metrics Exercise

[00:00:00]
>> Okay, metrics, let's do an exercise. So, for each of the following, identify what metrics you would track as part of that project. So just give me a list. I try and put at least one or two. Generally, not more than like four or five for a project, otherwise you're like kind of spidering off into some interesting things.

[00:00:25]
But just like top level things, here's ways that we can measure that this is getting engaged with, that this is worth supporting, that this is something that we wanna keep doing, or is this something that we wanna roll back and not do anymore, right? Those are the metrics that I generally try and seek.

[00:00:40]
So as part of those checked metrics, what company wide or organizational wide metrics would you affect for those all up companies? Okay, so building a better designed Add to Cart process for Nike where recommends related products. So I'm sure many of you've seen, you add this thing to your cart and then it recommends this other thing, right?

[00:01:08]
That's kinda the thing I'm getting at there. A product to improve music recommendations by Spotify. When I say improve, I mostly mean that I listen to this one song, the song ends, and then you know how it keeps playing another song if you run out of things to listen to.

[00:01:28]
Making that recommendation more closely match what we think the user actually wants to listen to. That's what I mean by that. Okay, detecting stored credit cards expiring and preemptively asking users for a new one for Delta Airlines. This is definitely not something that just happened to me [LAUGH].

[00:01:48]
Adding a share to Facebook button for completing a bike ride on Strava if you're not familiar with Strava, it's like an exercise tracker, right? So you can track your run or track your bike ride or something like that. They used to be our neighbors when I worked at Reddit, fun fact.

[00:02:03]
And adding the ability to sign into a PlayStation, any PlayStation, by with just a QR code on your phone. So, let's go through these one by one and see what you all thought about this. Certainly, one of the things that I would track as an e-commerce website, having worked on e-commerce websites is average order size.

[00:02:27]
Generally you wanna sell more items, right? And so that would be one that I would expect this a ladder up to and then just revenue in general, right? As part of it, you would track how successful this was, are people adding it and I would also check abandon rate, right?

[00:02:45]
Because whenever you add extra steps to a checkout process people abandon, it's just always how it works, right? And so if you're having more cart abandonment and too much cart abandonment, then you would want to abandon this project. But if you're raising your average order size or average amount of items and just seeing successfully, we recommended this, they added this, you might wanna keep it.

[00:03:09]
You can certainly check the customer satisfaction of this as well. How annoying am I when I asked you, do you wanna add this also to your cart? I hate when Uber Eats does it, right? It's like, please take that away from me. Let me have some sort of thing to take that off.

[00:03:25]
The day it does definitely lower my satisfaction with Uber Eats. Yeah, and I said revenue. You might be recommending things that are more profitable to you. I don't really like that. I think I'd rather recommend to someone what I actually think that they want. In general on the person says, I want to match my user's intent.

[00:03:45]
My user wants this, I want to try and give it to them rather than trying to say, hey, your intent is this, but my intent is this. I try and put theirs above mine. I'm sure there are people out there that would disagree with that, but they can be wrong, that's fine.

[00:04:02]
A product to improve music recommendations by Spotify. I'm sure one of their top level metrics is engagement, right? How long are people listening to music? So that's what I would expect them to be tracking with this is if they're listening to a song and then it just let it keep playing.

[00:04:22]
Are they adding more time or are they just been listening to something else?. Anyway, I'd be tracking that. I probably be tracking. I'm sure they have some metric around like A time where they suspect someone is not actually there listening to it. If I left my car playing it for some reason and left and no one's listening to it, I probably wouldn't be trying to drive those up.

[00:04:44]
I don't know how you would track that, but if you were Spotify, you could probably try and figure that out. Netflix actually worked very hard to figure out that sort of thing. Because if they're listening to those music, Spotify is paying royalties on it, right? So you don't wanna drive things up unnecessarily.

[00:05:00]
User retention, right? Actually a big reason why I stick with Spotify over Apple Music personally is that Spotify gives me much better recommendations and I end up finding out about level a lot of music through Spotify recommendations because they're just better at it in my opinion. So retention would be one that I would look at on that.

[00:05:16]
And I think a strong metric that you could look at to say that we're doing betters. They're playing this song, we recommend this song, and then if they follow some path based off of that song, that they went and listened to another artist buy it, another song that was in a similar vein.

[00:05:34]
Whatever way we can attribute it is, we showed them this, and we got some measure from them that they liked it, right? That would probably be my project metric of, I'm looking to see if people like what I'm recommending to them. Because that will correlate as well to their satisfaction, right?

[00:05:52]
Okay, detecting store credit cards. I think my top level metrics I'd be looking for here is just success, successful checkouts, that would be one that Delta would track as a whole company. And then one that I would be looking for for the project is successfully charged stored credit cards that came from previously expired ones, right?

[00:06:13]
I book so many flights, but every time I go to Delta, I just expect them to have the right credit card and it's gonna work that time. And so I got really frustrated. Why is this not working? He's it has an old credit card. So that was my rant on that.

[00:06:28]
But yeah, revenue lifetime value of a customer that's another one that you'd probably look at as well. And then maybe some sort of abandonment around like we asked you to open the email and then do nothing about it, probably means we're annoying you but who knows? Share to Facebook button for Strava, I'd be looking for referrals and specifically social referrals.

[00:06:56]
So how many people are tracking back to something like this? If they're a Strava user, I'd probably be looking for high fives. Or you can encourage your users, or comment or something like that to say, hey, we're driving social engagement on the platform, which makes the platform more sticky, right?

[00:07:13]
So that would be one if they're signed in and if they're signed out, how many sign-ups are we getting from these referrals to Facebook? Maybe some impressions. I mean, I guess impressions are useful as well, so I'd probably be really interested in those because I imagine I've told you, some of you today probably found out about Strava for the first time, right?

[00:07:33]
So they're not a totally ubiquitous brand. So adding a share button to Facebook from Strava, the things that I'd probably be most interested in this would be social referral traffic, right? People that are actually clicking on it, just to prove that we're not just spamming Facebook, right? I'd also be looking at how many people are actually clicking the share to Facebook button, right?

[00:07:57]
Cuz if no one's clicking it, then what's the point of that feature? Maintaining social integrations is a pain in the butt as someone that's had to do it. If you don't have to, you don't want to. So that would be one. I would be looking for people that are referred back to, are they signing up if they're logged out?

[00:08:12]
Are they interacting in some other way if they've signed in, so some sort of like high fives or commenting. That would be good just for social engagement which makes our platform more sticky. And then impressions are also very interesting as well because that drives brand awareness. Not everyone is aware of what Strava is as a brand, so driving people to the site is gonna drive awareness and eventually that might lead to them signing up.

[00:08:40]
Okay, and last one, ability to sign into a PlayStation. I would just be looking for time to sign in, right? Cuz I imagine a lot of people don't wanna put their username and password in with their controller. So successful signings would be good and then time to sign in so you can correlate, we have more signings, and people are successfully doing it and spending less time doing it because it's not one of the more fun parts about it.

[00:09:07]
If I was the PM of that, the other thing I might be more interested in is driving as like a bet that I go to my friend's house, my friend has FIFA and we're gonna play FIFA together. Typically I would just pick up a controller and not sign in because I don't wanna be bothered with that whole sign-in process.

[00:09:26]
But if you made it really easy for me to just scan a QR code, I'd be much more apt as like, all right, let me bring all of my stats over with me. And then let's make this personal that we can track our record against each other, right?

[00:09:41]
And so I would scan that with the hypothesis as the product managers, having data on where people are traveling is probably useful and knowing who's playing couch co-op against each other would be useful. And you can end up with some really interesting recommendations like, hey, we saw you play PlayStation with this friend.

[00:09:58]
Do you know that you two might like playing? I don't know FIFA street together or something like that. And so you get this new vector of useful data to talk to your customers about where you can offer them useful recommendations. I'm always looking for those kind of synergistic overlaps between me and my customers.

[00:10:16]
I want you to do this. You also wanna do something that's similar. Let's find the Venn diagram of what's good for you and what's good for my company, right? Those are the things I get to feel really awesome about the job I'm doing as opposed to, how can I trick you into doing things that are good for the company that's not super detrimental to you.

[00:10:33]
I don't like those kinds of overlaps.
>> What we've discussed here is essentially a blend of data that I can collect on people who are using my platform or service that they may or may not be aware of and then information that they can provide me. Is there a way you think about the balance or which type of information should go after?

[00:10:57]
And there's plenty of these ethically dubious circumstances where a little bit of privacy invasion which would probably be a great benefit to users but there's other concerns over those data being collected.
>> Yeah, I mean, it's something I think a lot about. Let's start with just the absolute base level, you got to comply with probably with GDPR, right?

[00:11:22]
Your users have to be able to know what's being collected on them, and they need to be able to be forgotten, right? That's just like the baseline legal aspect of it. My personal philosophy on this is, one, data is a liability a lot of the time. So what you know about a data can also be exfiltrated about that data, and so you wanna be cautious to keep things that are directly pertinent to things that you intend to use, right?

[00:11:50]
So don't collect. Nike doesn't need to know my age, for example, when I'm buying shoes. So they should not collect my age, right? Things of that nature where don't collect data that you don't need for some particular reason. I don't know, maybe Nike does need my age, but hopefully you're drawing the conclusion there of like, maybe McDonald's doesn't need to know my age when I order a cheeseburger.

[00:12:14]
So that's number one. Number two, I generally try to keep to informed consent that you know that you are giving me this data and at the very least of like strongly implicit consent, right? So if I told you it's like, hey, I took this data that you wouldn't be particularly surprised by that.

[00:12:34]
Let me try to give you two examples of this. So using that PlayStation 1, where you log in with a QR code on your friend's PlayStation, you are probably unsurprised that PlayStation correlate is like two PlayStation accounts played together, right? That is not gonna be super surprising and I was like, hey, we think that you two might know each other if you are on the same couch playing the same PlayStation.

[00:12:58]
Now you might not have said, I'm okay with you gathering that data, but you're not particularly surprised, it's data you actively gave by scanning the QR code. I feel okay about that. I think it's okay to gather that sort of data. Now how would you feel about, you went and played on that PlayStation and then later a different friend when played on the same PlayStation without you there and all sudden.

[00:13:19]
I can still correlate, you were there and this different person was there at a different time. That feels a little ethically dubious to me to say, we're gonna correlate that. We think that you also probably know each other just based on that fact alone, right? Maybe it's a bit less so of like, you played with this person and I played with that person.

[00:13:36]
We can probably correlate that there might be some sorta social graph there. Just like LinkedIn recommends to use, hey, you probably know these people because you were there at the same time and you have all the same connections. That feels okay to me as well. Let me draw one too that I found particularly disturbing.

[00:13:53]
I'm a Gmail user, like probably most of you. And I found a screen within Gmail, is like here's everything you've ever bought. And I was like whoa, I did not feel okay with you reading all of my emails and pulling out all of the things that I've bought.

[00:14:12]
And then I found out they'll show you, here's all the ads we recommended to based on the things that are in your inbox. And I was like, I'm very okay with you not or not okay with you providing that email to a third party, right? Or even to your ad server, right?

[00:14:29]
Now, I posted it on Twitter and obviously got very much both sides of why you're using a free product, you are the product haha, I hate that, by the way. Never told me that because then I won't be your friend. Yes, I know that free products or whatever.

[00:14:46]
But I still think it's on the provider of the service to have informed consent tonight. That was a thing that when I found out about it, I was very surprised by it and I would not have consented to that, right? So that's where I draw that line of, don't gather that data if that user would have been surprised that you have gathered that data.

[00:15:06]
Another example is, if you go to Nike and you buy shin guards for soccer, It's probably okay that Nike buckets me into a soccer mailing list, assuming that I gave permission to be on a mailing list. It makes sense that you buy soccer gear, you probably wanna know about soccer things.

[00:15:23]
That kind of derivative based on information, or actually, sorry, I'm probably going too long on this, but last example here. When I was at Reddit, the way that we advertised to you on Reddit is, what subreddits you have subscribed to, right? So if you subscribe to the soccer subreddit, you're gonna see soccer ads.

[00:15:42]
That felt really okay to me because you have given Reddit that information and we're not pulling that from third parties or anything like that. We're just using it based on what information you have chosen to give us. So a lot of ethics tied up in this. A lot of law tied up in this.

[00:16:01]
I'm going to invite you to try and act both legally and ethically. It's a hard problem.