Testing Performance

[00:00:00]
>> Todd Gardner: Next, we're gonna talk about tests and tools. So how to actually test for these metrics, how to gather this data and how to interpret it as part of that, we're gonna need to talk a little bit about statistics. Now, this isn't gonna be like a seventh grade math course.

[00:00:14]
We're just gonna do just enough to get us by so don't feel too bad you didn't do your homework. So we're gonna talk about testing methods like, how do we go and look at this data? Then we're gonna look at some common tools that we use, and then we're gonna talk briefly about real user monitoring, which is a classification of tool, that can help you gather a bunch of interesting data.

[00:00:38]
So first, testing performance. How do we go about thinking about capturing data? Where do we get the data from? So there's a couple of different approaches to this, and each has a little bit different drawbacks. So there's a couple of different ways we can test performance. It's important to think about this, because how we measure performance?

[00:00:59]
Where we choose to measure from can give us drastically different results. Fundamentally, what we're doing is we have a host somewhere. This could be a server, it could be your cloud, it could be the hosting service you provide, and that has all your content, and then you're delivering that content across the Internet to your users and rendering it on their device.

[00:01:22]
And so there's a couple of different places we could test that web performance from, the first place, and the most common place that most people do it is with what's called lab data. Is they create a test device and it's located usually somewhere really close to the host.

[00:01:40]
Close in terms of like network or connectivity or whatever for example, when I'm running our sticker store locally, I'm literally testing the performance from the same place. It's not needing to cross any networks or do anything that would be lab data. So we have a test device, we render it and we gather some metrics about it.

[00:02:04]
Another place that's really common place to test is what's called synthetic data, as we have some robots some service out in the internet somewhere that we say, hey go to that website and tell me how fast is. Now that's a little bit better, it's a little bit more accurate because it crosses a real network in order to test it.

[00:02:22]
But generally, these test devices are just a server like another high-end machine on a high-end network somewhere out on the Internet that is rendering your page. We're not really, neither of these two approaches are testing what the users are seeing. We're both just taking like we're cutting out a bunch of steps and creating a simulation of what we think the site is gonna be doing.

[00:02:50]
The third way that we can capture data is called field data. And that's what we were talking about a little bit this morning, is we can capture those performance metrics from the real users using your site and send that data off to a reporting service. Now this is the most accurate way because it is reality.

[00:03:12]
It's these are the people that visit your site and this is the performance experience they had. Everything else was a diagnostic, was a test. But this sort of field data is real. It's gathering this real data. Now there's huge differences in this. And it's important to note that there's a sample size involved.

[00:03:33]
So this is the same, this is performance data from the exact same website. On the left is a Chrome Lighthouse report, which is a tool that we'll talk about. And a Chrome Lighthouse report is lab data. It's, I am running a test from my laptop against a server that's probably close to where my laptop is maybe hosted from my laptop.

[00:03:58]
And it generated a performance score of 100, which means that time, that one time performance was great. On the right are the core web vital data from the real users that visit that site and notice that there's not one score there. There's thousands of scores representing every single user who visited.

[00:04:20]
Now this is two different views of the same website, and on the left we have lab data saying you got a perfect performance score. And on the right we have field data saying, hey, there's a lot of people that have a crappy experience on your website. Now, it's not saying they all do because some people.

[00:04:42]
Some people had a great experience. It does happen sometimes, but a lot of people had a really bad experience. And so we have to think about how we look at data. When we're talking about lab data, we have one sample. You run a test, you get one performance score back, and it says something, it says 100, it says 95, it says 80, it says some metric that you're tracking and it gives you one data point.

[00:05:12]
But when we're looking at field data, we get lots of data points. You have one score, one value for every single user who visits the page over whatever time period you're looking at. And we have to interpret this data because these could be performance data from the same website, but how do we think about this?