Team Metrics

[00:00:00]
>> So now that you're armed with a little bit of stats, a little bit of graphs, let's talk about metrics. And metrics are central to being a good products manager because it's evidence. It's evidence that you are going in the right direction. It's evidence you're going in the wrong direction.

[00:00:16]
It's a massive tool for you to describe success, to describe failure. It's you measuring and being a little bit more scientific about your approach to your product. So first of all, what is success and what is failure? I hope that your organization and your team has some metrics to describe, we are trying to affect these metrics.

[00:00:46]
These are core to our mission, and here's how we are measuring that. And we are hoping to affect these other metrics that we are hoping they go down, or up, or whatever those are. So let's talk about that. But generally all of your actions, and I think that's fair to say, all of your actions should affect your metrics in some capacity.

[00:01:09]
And if they are not affecting your metrics, you better justify that to yourself. And that usually means, I need to change what metrics I'm looking at if I'm doing things that are not affecting my metrics. Because obviously I care about something that's not measured, and I should measure the things I care about.

[00:01:24]
So let's talk a little bit about team and organization metrics. Again, I've said this several times, but this is actually now the most important thing I'm about to teach in this class is, if your company and your team don't have North Star metrics or most important metrics, which a lot of people call North Star because it's the guiding star, right?

[00:01:48]
If they don't have top-level metrics, go make them. If you do nothing else, that'll be the most important thing that you've done for your company, period. And the way you figure out is, what are my North Star metrics? The first thing you look at is, what's your mission?

[00:02:04]
What's your charter? What's your team's vision? If you have any sort of documents or pages that describe, my company cares about this. Usually that's gonna be something like, hey, we're trying to, the Frontend Masters mantra is to educate more people more deeply, am I close? I'm close, right?

[00:02:28]
So you hear in that statement right there, more people, that's sign ups, that's retention of users, right? Those two things together, and then more deeply is depth of engagement. How much are you watching this? How much are you engaging with the platform? How much are you engaging with other people in your community?

[00:02:46]
There I just defined at least some of the KPIs that Frontend Masters cares about, right? That's a fairly straightforward exercise for you to think. It's like, okay, what does my company actually care about? And if your mission statement doesn't describe something like that, you probably need to change your mission statement.

[00:03:05]
Because it should have, here's the change I'm hoping to affect in society, right? Here's the demographic of people that I'm trying to get to, and here's what we're trying to offer them. So this is an interesting kind of exercise to pick your favorite company and go over what you think that their core metrics would be.

[00:03:31]
And when I say core metrics, or top-level metrics, or North star metrics, synonyms of the same thing. So yeah, let's look at Netflix. What do you think Netflix's top metrics are? Sign ups, just more people on the platform is goodness. Retention, are they staying there month-over-month? Revenue, things that you can get people from lower tiers of Netflix and the ad-supported ones into the higher level ones generally is a good thing.

[00:04:05]
Watch hours, or depth of engagements, how much are they watching the shows? And that's probably pretty close to most of their what? I would imagine they probably have one around, actually, I know exactly what they have because I used to work there. But they have one where they prefer Netflix shows over other shows, right?

[00:04:27]
Cuz they wanna justify their investment in personal content that keeps people on Netflix, right? You sign up for Netflix to watch, Orange is the New Black or Stranger Things, right? And that keeps you from signing up for Hulu because you can't get that stuff on Hulu. But if you're signing up to watch Friends, that might move tomorrow to Hulu, right?

[00:04:48]
So that's kind of why they track that as well. But generally, these are the things. And I imagine now they have games, right? So they're probably gonna have something around games. And they have stuff for ads, right? That's new to me. I don't know how they handle that as one of their top level metrics.

[00:05:05]
But probably people watching ads successfully is probably another metric in there as well. Again, you could probably just roll that up into revenue though, so. And gaming could probably just be rolled up into retention. So these are the metrics that every team inside of Netflix should be contributing to in some fashion.

[00:05:32]
So I worked on the sign up process across web and TV, and Windows Phone, fun fact. What do you think that team's charter metrics are? Sign ups, that's the North Star metric of that team cuz it's the sign up process. Time to sign up, generally if a user lands on the site with the intent to sign up, we wanna get them from zero to sign up as quickly as possible, and as usefully as possible.

[00:06:04]
Bounce rate, this is a funnel, right? So if you go on this page and you get to the third page and you bounce off of that page, that's not good. And then 30 day retention. We don't wanna sign up people that are actually just gonna bounce off, right?

[00:06:18]
So that was another thing that we tracked pretty closely. But notice all of these, 100% of these ladder up, when I say ladder up, I just basically mean that they feed into, right? All of these ladder up into organizational metrics. That should always be the case. If your team is working on something that doesn't ladder up into your organization's goals, means you're going this way and they're going that way, and why, right?

[00:06:48]
Now you might say, well, I'm working on a product that's a skunk works product or project, or it's a big bet, or blah, blah, blah. You should figure out a way that that fits into your company's organization somehow. Either your organization is going to add a new top-level metric that your skunk works project is gonna feed into eventually.

[00:07:07]
Or you should figure out, okay, eventually I'm doing this and it's gonna make our project more compelling so more people will sign up for it or will retain users better, blah, blah, blah, things like that. Okay, any questions about team or organizational metrics?
>> What's your opinion on depth of metric measurement?

[00:07:32]
And sometimes this is said above you, but, [COUGH] my information science background said, I don't say if I were advising the C suite, I would say these metrics are incomplete because we don't have things like characterization of the first 30 days, which would be trivially accessible on a platform.

[00:07:51]
You could be kind of a great example. Did you sign up to watch the new show, binge it, and then not sign up, right? In which case we know why, or was it you poked around and petered out? Those kinds of things might be useful, but that is really what does the organization wanna hear?

[00:08:08]
So how do you, and I know product would be very much involved in these decisions, how do you decide where that lower bound of information collection needs to?
>> Right, no, I think it's a good question. So, first of all, Netflix very much cares about your first 30 days on Netflix because it basically determines what your lifetime value is to Netflix.

[00:08:30]
What you do in the first 30 days characterizes down, how long you're gonna stay on the platform, your churn rate, all those kinda things. So there are some organizations that would say, we're gonna put that as a top level metric and every team should care about this. I, at least at the time, we basically said that that just is retention.

[00:08:55]
And if you have a great first 30 days, you're gonna affect both retention and revenue. I think lifetime value actually was a top-level metric on some of these as well. But arguably that's retention and revenue kind of just combined, right? So the way that I think about it is, all of these kind of more advanced metrics which I care about, we should definitely measure, and the teams working on those things should absolutely have those as team metrics.

[00:09:25]
But they should all somehow ladder up into one or multiple of these, right? So it should just basically be like, if I affect what people do in the first 30 days, I'm definitely gonna affect these other metrics, right? And so that's sort of correlation where there's I can justify this by this just being an advanced metric of this, or a derivative of this, or this is just this further down the line.

[00:09:49]
And if I'm doing something that this is important, and it does not affect one of these metrics, you should change your organizational metrics. That's, that opinion, and I think other people would have different approaches to that. Then I'm right. [LAUGH] I don't know. That conceptually makes sense to me.

[00:10:11]
That's the way that I've worked through my career.