Types of Tests & Tradeoffs

[00:00:00]
>> So the way we develop code is essentially this, right? Everybody does this. The question is, when we do the testing, do we do it manually or do we do it in an automatic fashion, right? Nobody goes and writes huge amounts of code at once and checks it in.

[00:00:17]
Everybody writes a little bit of code and then you do something to test it. Whether it's manual or automatic, it doesn't matter, but you do something to test it, right? You don't write code for months without ever running it.
>> I was gonna say, hopefully [LAUGH].
>> Hopefully, I mean, but the point is that there is always this testing cycle that happens, right?

[00:00:39]
And so the only thing I'm trying to convince you of like, look, you're gonna do this anyway, let's just not do it manually. Let's just automate it so that the computer does it for you so that you can really focus on just writing code. And as you are developing things, you have more and more tests that you build up over time.

[00:00:59]
And these tests were essentially requirements, or assumptions, or something you expected to be true all the time. And as time goes on, you just build up a huge repertoire of it. And hopefully, you've read it in a way that is easy for others to kinda look at it and understand like, yeah, I understand why they wrote this test because they wanted to assert something or something useful.

[00:01:19]
This should be true or something of that sort, and then you build up over time. So the thing I'm trying to convince you of is that at the end of the day, this is how everybody does coding. I have never seen anybody not doing it this particular way.

[00:01:30]
The question just becomes, are the tests automated or not, right? If they're manual, then they're fundamentally limited, because you're not gonna remember to do all the possible testing all the times. You're gonna only do a subset because it takes time and chances are you gonnav forget stuff and problems will arise.

[00:01:52]
So what kind of tests do you have? Well, you can split it in a lot of different ways. You can talk about testing level, for example, unit test, integration test, system test, acceptance test. You can split it by functionality, for example, does it do the right thing? Does it do it fast enough?

[00:02:10]
Does it do it in a secure way? Does it do it in a way where usability matters, right? You take a random person off the street and sit them behind the software, will they be able to do the right thing, or does it take a complicated set of training?

[00:02:24]
Is it accessible, right? Those are different kind of tests. Automation, which is, have I managed to automate it or is it manual tests, right? If you think about the old way of doing QA, what you would have is just documents, pages and pages of checklist of what the QA would have to go and click on and assert, etc.

[00:02:44]
You're just using a human as a very slow and inefficient CPU. And of course, we have regression and smoke. Regression being, I thought the system was working, I discovered a bug. I'm gonna add this particular scenario now to my test suite so that it doesn't happen again. And a smoke test usually happens when you have a large system and you wanna determine if everything works correctly.

[00:03:07]
But the number of tests you have is so huge that it takes maybe hours to run. And so you select a subsection of the test that kind of give you a sanity check. Did you really break anything or the basics stopped working? So you would have a smoke test.

[00:03:21]
So there's a lot of different kinds of way of doing testing or what we can talk about.
>> Do we need to follow the test-driven development or is it optional?
>> So test-driven development, that's a very good question. And it's a complicated answer, because I'm a big fan of test-driven development, but at the same time, I am not gonna lie to you and tell you that I do it all the time.

[00:03:47]
It's a mixture of both, right? Sometimes you do TDD, because it makes sense for you. Sometimes I'm not exactly sure what I'm doing and so I'm kind of experimenting with things, and then I have to go back and add the test afterwards. And I can tell you that whenever you go back and add the test afterwards, you're gonna miss stuff.

[00:04:05]
You're never gonna cover everything. So I try heavily to prefer TDD whenever I can, but I don't always do it and there are various set of reasons why not. But in general, you want to do it in a test-driven form first. An example of that would be you wanna make a new feature, right, and so you write a test that basically demonstrates this particular feature in its simplest form.

[00:04:37]
And now you have a test that doesn't work, right? So now, you have something that you can easily run over and over and easily get under the debugger and easily put console log statements in your application that gonna figure out what's going on. And once you have this scenario that isn't working, now your job is to go back and go and implement this, right?

[00:04:55]
And so you start implementing it, whatever it takes until the test pass. And when the tests pass, the thing you should be doing is to try to implement just the bare minimum of what the test requires, right? So you do the bare minimum first, and then you write another test that is a little more complicated, a little more involved, or test a corner case or something of that sort, and then you go back and implement it.

[00:05:21]
So that's the idea of test-driven development. But as I said, while I think ideally you wanna do that all the time, sometimes there are reasons where that might not work. So I don't wanna just basically preach something that I myself sometimes don't do. Okay, so let's talk about scope.

[00:05:40]
So unit tests are the lowest scope, and we're gonna actually have hands-on building, etc, this is kind of a theory portion, so to speak. And we're focusing on individual units. And so what's a unit? A unit would be a individual function, or a class, or a file. Typically, in the same file, you put related code together, right?

[00:06:02]
So that will be a unit. And what we mean by unit is that really, you can think of a test as a harness in the sense that it takes that unit that you wanna test and then pokes at it with different scenarios, inputs, whatever you wanna call it, and it asserts as particular outputs.

[00:06:21]
It's like, if I do this, I expect that to happen. And a unit test really just says that it's very clear as to where you should look for a problem, right? If I poke over here and the stuff doesn't work over here, there should be very limited number of suspects for you, right?

[00:06:40]
If I have a function that is sorting things by distance from something, then it should be very easy for me, if the sort comes out incorrect, to pinpoint where the mistake is. Not necessarily where it is, but which code to look for it, right? There's another extreme, if you have an end-to-end test which tries to log you in and then goes to the Settings page and change your email address and this test fails, who knows where the issue might have been, right?

[00:07:11]
There's so many layers involved that becomes very, very difficult. So I'm just kind of putting the two extremes on a table over here. And so usually, what we mean by unit test is something that is very well defined. The other kind of hallmark of unit test is that they're isolated, right?

[00:07:29]
You will never have a flakiness inside a unit test, or you shouldn't. I can't imagine how you would get a flakiness into a unit test, right, because it's fully isolated. And because of that also, they're extremely fast, right? You should be able to run thousands of them in a second.

[00:07:47]
That should be not a problem. As a matter of fact, if you think about it, unit tests are typically CPU-bound. And it's typically extremely difficult to keep a CPU busy for a second, unless you're going through huge amount of IO where you're moving a lot of data across.

[00:08:05]
Actually, just keeping the CPU busy for a second with just pure computation is kinda tricky or difficult. The amount of code you would have to write would be ridiculous. So unit tests are essentially tests that then make an assertion typically about how things are done. And so they do tend to fall into this category of like, if I change the way I am doing things, I'll probably invalidate a whole bunch of unit tests, right?

[00:08:31]
I don't know, if you're processing an order and one of these steps in the order is to sort things by the distance in your warehouse or something like that and you decide that that algorithm no longer is true, but instead you wanna apply a different kind of algorithm.

[00:08:43]
You're probably gonna invalidate a whole bunch of tests that have to do with distance measurements, right, but that's kind of expected. So that brings us to a higher level test, which is the integration test. An integration test basically says, hey, can I have a bunch of related things to together.

[00:08:58]
So I'm no longer testing one thing, but whole bunch of things together. So a order of workflow might be something like an integration test, right, where you're saying, if I start with this workflow, this particular order, and I go through these steps, I assert that I have it in stock and I wanna ship it, etc.

[00:09:18]
The state machine changes in, I should eventually get to this particular output. And so there, it becomes a little more nebulous as to if things fail, where to look for the problem, right? Because who knows which part of the system has kinda produce this particular issue. And so integration tests, though have really nice property, though that typically they don't tell you how things are done.

[00:09:49]
They only say, given this beginning, I wanna make sure I end up over here. How exactly you get there is not my concern. And so integration tests do actually help you with refactorings in the sense that if you wanna change how something works, integration tests typically give you a lot of leeway.

[00:10:07]
So for example, when I work on my framework, which is QUIC, there's a lot of integration tests that basically create a little tiny program, a little tiny scenario, like a counter. And the counter has a button that you can click on that increments the count and then I assert that the DOM on the output has updated.

[00:10:25]
I don't care how it did it internally and who knows how much code had to execute in order for this thing to to actually happen. All I'm saying is, hey, if I have this tiny little program that represents a counter and then I click on a button, I want the count to increment, right?

[00:10:39]
That's a good integration test, because it allows me to refactor how the internals of the frameworks work without necessarily forcing anything on the user. And then, on the other extreme, we have end-to-end tests. And the end-to-end test basically to think about it is, I am pretending to be a user, right?

[00:11:03]
So you will probably open up a real browser and you will simulate real clicks and you will go through a whole testing scenario with it. And these kinds of tests are typically slow. They tend to be flaky. You might have to mock out databases, and so it just gets complicated to write these kinds of tests.

[00:11:28]
They're still useful in a different way, but it gets complicated. And this is somebody was asking, well, what about the role of the QA? This is a perfect place where QA can help, because they can help with setting up, whether it is setting up mock databases so that we have expected set of orders in there, etc.

[00:11:46]
Or setting up a test harnesses or easy way to assert what's happening, what we call these page objects or just things of that sort.