Testing Pyramid & Development Model

[00:00:00]
>> So let's talk about different kinds of tests. So a lot of time people talk about this testing pyramid. And the idea of a testing pyramid is that you have two axis, you have execution time, and you have number of tests, right? And so what people are saying is that primarily, you should have unit tests And when the unit test really check, is that they check whether your code does what it's supposed to do.

[00:00:26]
To put it differently, it checks, when you write your code, you have if statements inside of it to make different decision based on different circumstances, right? Unit tests really are all about checking the if statements, right, that do right things happen with the right input. Functional tests, or sometimes known as integration tests, really bring multiple units together.

[00:00:49]
And there, you want to test not necessarily whether individual pieces do the right thing on their own, although you can, but more you're interested in, are the things compatible with each other? Like if I, for example, had a DBSCAN, which was the clustering algorithm, I had to format my data in the correct way for the DBSCAN to consume it, and then DBSCAN produce something, which I had to further format in the way that my clustering component could consume, right?

[00:01:21]
All these conversions, are they making the right assumptions? And this is the purpose of an integration test, right? You're not really questioning whether the DBSCAN itself is doing the right thing, you're just questioning of whether the conversion code to get it to state what DBSCAN can use, and then the conversion code to get it back out so that I can use, does that make sense, right?

[00:01:43]
And so, the functional test or integration test really are more into having a bunch of things together to see if you get the correct output. Now, it's not always that they're just interested in communication, sometimes you can have unit level tests as well, but it's primarily what they're interested in.

[00:02:04]
Now typically, functional tests will be slower because there's more code to run, but also because your functional tests are more likely to depend on I/O. If you think about it, modern CPUs are extremely fast, it's really difficult to keep a modern CPU busy with just code. Usually when you have to wait for things a long time, you're waiting for IO.

[00:02:28]
If we had our unit test where we were talking to GitHub and I was saying, look, that's a slow test, it takes a third of a second, the issue there wasn't that the CPU wasn't fast enough, the issue there was that we were doing IO to the outside world.

[00:02:41]
And so, as you go up in this existing pyramid, you're gonna discover that you're gonna do more and more IO, and it's the IO that makes the test fundamentally slow. And finally, the scenario test, or like the famous E2E test, which is essentially the Playwright, Playwright is gonna be even slower because you are bringing up a full on browser, you're pretending to be a user, right?

[00:03:06]
Every single click might be a round trip to a server, which renders a new page, which navigates to a new place and things of that sort, so these tests as you go up, becomes slower. And this is why the recommendation is that when you develop software, you wanna focus on unit tests on the bottom.

[00:03:25]
If for example, I have a list of orders and the list of orders are incorrectly showing in the incorrect order, chances are I've probably messed up the sorting algorithm somewhere. So there should be a function somewhere in the code base that does the sorting algorithm. And I should be able to just invoke the function as a unit test, rather than go into an end-to-end test and try to order a bunch of try to input a whole bunch of orders through the UI, and then verify that the output does the right thing.

[00:03:55]
That's way too high level for this particular thing, right? We're just trying to assert that individual order forms are in the correct order, so we wanna make sure that our comparator function does the right stuff, right? So that's why you wanna focus mostly on unit tests. Now, this is kind of the ideal world, doesn't mean that that's actually what happens in real world, especially when you have a legacy code base, right?

[00:04:19]
If you have a legacy codebase, this will probably be inverted, because when you have a legacy codebase, about the only thing that you can do is to do end-to-end testing. And slowly over time you can go and see if you can refactor things and introduce your mocks into the system so that you can carve out smaller and smaller chunks for unit testing, makes sense?

[00:04:47]
Question?
>> So do the functional integration tests have the same testing harness, or is there a way to test a gateway like a database call? If I mark the database I don't get the actual result, which is what I'm most concerned with.
>> Right, I think the kind of tooling you use I think is orthogonal, for example, you can use Playwright with Storybook.

[00:05:15]
So, because you're using Playwright, is that an end-to-end test, or because it's a storybook, it's Integration test, right? So like, to some degree, these are not written in stone in terms of what they are, it's more like a guide to kind of talk about. When you wanna test something like a database, there are a couple of strategies.

[00:05:36]
One is you can mark it out, but the problem is, maybe the communication you mocked out is incorrect. So the next thing you can do is you can create a copy of your database and locally and low preloaded with local data, right? That would be kind of more a trustworthy thing because you're doing more of an end-to-end test.

[00:05:57]
But by doing that now your system is slow because you maybe can't execute fast enough. But it turns out if you wanna do something with databases, there's lots of object ORMs, object relational mappers, right, where instead of talking directly to the database, you talk to an ORM, which is a object-oriented representation of the data.

[00:06:18]
And that naturally creates a place where you can mock, so then you have a choice of whether I wanna talk directly to the database or talk to the ORM, and I can create mocks for them in that way. The other thing you can do is you can easily create like an in-memory database, like SQL DB has an in-memory version that you can get where you can preload a data set that's useful for testing, right?

[00:06:47]
Because there is a data set that is kinda a production and the production data usually is not the ideal data for testing because there's lots of boring stuff in there that isn't really that interesting. And for unit tests or for other kinds of testing thing you want the corner cases, and so you want a database filled with all kinds of interesting corner cases, and even the fact of understanding the corner cases of interest.

[00:07:12]
In that kind of situation, you can split it in two ways, you can say that you have a well known database which was prepopulated with these corner cases, the problem is, how do you communicate that to all your developers? Or you can say like, no, I'm gonna bring a brand new database, and as part of my test, the setup is responsible to write into the database and set up the corner case that you're interested in.

[00:07:38]
And I think the second one is more interesting one where you are guaranteed that every single test is basically starting with a blank slate, with nothing in a database. And then through something like a page object or some form of a DSL, you describe what the database should have that preloads it.

[00:07:57]
And now you have a system in a correct state into in terms of what you wanna test, and then you can go ahead and apply your assertions, you can poke at it to get the output that you want out of the system. How does this work? And I think the way most people do is they write code here, and I intentionally made this box red because that's where the bugs get put in, right?

[00:08:22]
So the software engineers write code, and then they have some QA and they expect that the QA will catch their bugs. And then you hire some test engineers and say, why don't you automate all this stuff so we don't have to do it all the time. And so, for some reason, when you first talk to people, they always assume the testing magic happens at this level.

[00:08:45]
They're like, get some magical software, pay company some money or whatever and this problem will go away, right? And this is not the correct mental model, this is, you cannot fix the problem at this level. And fundamentally, the reason we cannot fix the problem is, I have this opinion philosophy, I don't know what you wanna call it, which is that the person who creates a mess needs to be the person who suffers the consequences, right?

[00:09:12]
If the two people are different, nothing ever is gonna get better, right? So if the developer is the one that's creating a mess making the code untestable and then the automation engineer is suffering the consequences, this isn't gonna get better, right? You need to make sure that the software engineer, not only is responsible for writing the code, but also is responsible for writing the test.

[00:09:35]
And then they see the consequences to their actions, and now they have a closed loop and immediately they can learn and say, I shouldn't do this, it hurts when I do that, don't do that, right? And so this is not where the testing magic is, that's where the testing magic is.