Code Coverage, Coverify, and NYC

[00:00:00]
>> Okay, so the next thing that I wanna talk a little bit about is code coverage. Code coverage is a metric that you can use to know how much of your code base is being exercised by your test suite. You can imagine if you have tests but you're only ever hitting like ten lines of thousand line program, then that's not gonna really tell you a whole lot.

[00:00:23]
But if you can hit every line in your test suite, then that's a lot better. And so code coverage will let you tell that, we'll answer those kinds of questions. So I'm gonna talk really briefly about how code coverage works behind the scenes and we'll get into using it in a little bit.

[00:00:39]
So, code coverage tools generally read the AST of your code. So this means that you can take a piece of code like a piece of JavaScript and turn it into a data structure that you can manipulate other tools. So I can show you what that looks like. So here we have a file foo.js.

[00:00:59]
And I can require a library that parses the AST like acorn. It has a parse function. And so I can do do fs.readFile on that foo.js file and convert it to a string. And here we go, we have this data structure now that represents this entire program. So there are other programs that you can run that manipulate these data structures.

[00:01:29]
And what you can do is, you can imagine if you can write a program that could modify this data structure to put a little wrapper around every single line. So it'd be like like countedLine (0) and then countedLine(1), countedLine(2), then you can imagine that every time that this program ran, it would print out information about where in the program you were.

[00:02:04]
So that's really handy if you wanna implement something like code coverage. So I have a little transform that works in node in the browser for doing this called coverify. So with browserify you can specify different kinds of plugins that will work on the source code. So in this case it coverify uses the AST to do exactly like I was just showing you putting in all kinds of little statements everywhere just to see where in the program has been visited.

[00:02:37]
So if we run our tests, if I hop back into that example here where we had index.js I think. Right, so why don't I write a quick little simpler tests cuz that other one is quite complicated. So here we've got something more like those original programs I was showing you.

[00:03:06]
And we can elevenify 1 and then get back the result, so. The result should be 111 in this case, and if there's an error we wanna know about it, so that should be two assertions. Okay, so if I browserify this code, there's an option if you wanna run your browserify code in node you can pass in --node.

[00:03:34]
And so I guess the plan was wrong for that. Or plan (1) should be plan (2). Okay, so, start works. What that code is gonna look like on standard out is it's formatted all into just one big blob that like, if you were writing browser code, you would stick this in a script tag like we did earlier.

[00:03:55]
If I run this with the coverify package however, we see pretty different code. So here we have all of these statements coverage wrap, which are just all over the place. And how this works is every time it gets to one of those statements, if it's the first time that it's seen it, then it prints a message called covered.

[00:04:16]
And it does that just like tape using console.log. Which is something that's gonna work in node and in the browser, so it's kinda handy. So if I run that in node now, in addition to the tap output, I also see all of this coverage information. So you can see that okay, the first thing that it prints is for every file it provides a list of all of the places that ought to be covered by the code coverage.

[00:04:44]
And this is done with a with just the first pass over the AST. Then every time it visits one of those places for the first time, it prints a message. And so you can pipe that through another program that comes with coverify if you do npm install dash g coverify you'll get the program.

[00:05:03]
And if you do npm install coverify without dash g, you'll get the transform, as a library. So if I pipe that output through coverify I guess that's in node_modules/.bin cuz I didn't install it globally, then I get a little coverage report. So in this case, the coverage is 100%.

[00:05:24]
So we don't really need to know much more about the code. But if we had places in the code, like if I had some deadCode down here or maybe I had a secret case where like, if n = 5, then I'm gonna make it mess up. Can I call it 554?

[00:05:45]
Yeah, it'll be good. So if we had some special cases in our code for example, then when I ran the coverify commands, it's able to detect those. So we see that the first part of this if statement did actually execute, but n was not 5, so we didn't run into this case.

[00:06:09]
So our code coverage is now less than 100%. But if I modify the test suite to add a second case for that, so I'll do for 5 now should be 555, no 554, can see that. It's going to fail because the 554 case, but the code coverage should be 100%.

[00:06:38]
So if I modify that, code back to something that works. And now we should see 100% again. There we go. So that's kind of the basic idea of how code coverage is implemented. And there's another tool that's a little much more robust than this one. This is it's still useful, but it can't handle quite all of the cases that this toolkit called nyc.

[00:07:09]
So you can install it with npm globally and it'll give you a command. And then you can just pass nyc in front of whatever commands that you wanna run and it's somehow able to know. I don't know exactly the mechanics that it uses behind the scenes but probably something that the AST in some really hacky stuff.

[00:07:31]
I can only imagine what it's doing. But I can show you what it looks like on a real code base. So here's just a simple little package or an npm test, and I get this output or I could run tape test as well. So tape test.js. If you put nyc in front of any of those commands then it's gonna print out a coverage report.

[00:07:52]
And I can show you what that looks like if all of the code isn't covered by just like putting some deadCode in here as well. So, extremely simple module. But if we had deadCode. Then nyc would tell us about that. So yeah, so uncovered lines 8 and then there's a bunch of commands that you can use to sort of drill down more into where in your code base you haven't covered.