Developer Tools: JQ

[00:00:00]
>> ThePrimeagen: We're gonna switch gears now and go into some more important parts of life, okay? Which is, you should learn a bunch of cool stuff when it comes to your operating system, okay? There's a lot of fun things that are just available and really what it comes down to is the Unix tool set.

[00:00:16]
I still remember this one time I had 2011, I wanna say it is it might have been 2012. Yeah, 2012, I went and interviewed for a company called web filings at the time now known as we're Kiva publicly traded company, Bozeman, Montana, Bozeman. And when I interviewed there, one of the questions went along this lines.

[00:00:35]
We have been hacked and they put a bunch of salacious links in our website, viagra.com or something, right? My gosh, they're like very salacious and so they're all throughout our website and you need to go find them all and delete them all. And this is during the day where you still FTP a lot of the times, this is a long time ago.

[00:00:54]
And so it's like, okay, so that means we need to go find this. So what did I do, as a budding young student? Well, I did the only thing I knew how to do. I did public static void main args, string args, did all that, got it in there, tokenized, walked through all the files, every directory recursively walked through it, did the whole thing all the way through.

[00:01:15]
And then they're like, wow, that was really good cuz you used anything else. So I'm like, I don't know, right? I had no idea if you could use anything else cuz I didn't know that I could just simply use Zed, not Zed. If a Zed is an editor, Sed is n not an editor.

[00:01:31]
If I already knew the link I was looking for, I could just find him and just replaced him. If I just wanted the file and locations, I could have used crap. These things just didn't exist in my head when I was younger, and I spent countless hours writing programs that already exist and can be used in an ad hoc way all the time.

[00:01:46]
And so don't be me, it's not good, right? Don't be me from the past or the future really, don't be me either way. And so I'm gonna show you a bunch of stuff that is just super great. If you do create your own tools, just to always remember accept standard in, go on standard out for the output, standard error for any sort of errors.

[00:02:04]
Don't break that because then you can chain things together, and chaining things together makes your life really good. Don't forget that. All right, so FZF, we don't really need to talk about FZF. We talked about FZF, FZF was great, you saw how powerful it is. So we're just gonna kinda leave that one out and not because I want to leave it out because you guys already know about it.

[00:02:22]
All right, so we're gonna talk about one really cool one. This is like a hidden treasure of the CLI tools. Okay, there appears to be only a few of you here. Hands up, who knows about JQ? Okay, half of the people here know about JQ. Which is always kinda shocking because JQ is an exceptionally convenient thing to have on your system.

[00:02:48]
You will be just blown away by how amazing it is. And I will tell you personal side story, that I have probably saved 100 to 200 hours worth of programming by spending five minutes learning a couple things and asking ChatGPT, how to do various things. It works actually pretty well with GPT.

[00:03:05]
GPT knows exactly what to do with this one cuz this falls under GPT best thing, which is, it only has so many options, and it can just do it. So this is really, really good, very hard for it to hallucinate, though it still figures out how to do it even with JQ.

[00:03:18]
All right, so let's create this file. If you guys wanna follow along, create a really simple file that has a type that is foo or bar just two types. It doesn't really matter what they're, I just want them to be distinct, and then I want a values array that are numbers.

[00:03:35]
And then the other one, I want it to be a values dictionary with a and b. They're intentionally different. Foo has an array, bar has a dictionary, okay, or an object. Since this is Javascript object notation, it would be an object a POJO, it has a POJO. So now that we have that file here, I should probably copy and paste this file myself.

[00:03:57]
I'm gonna copy it and jump over here. We can just do it in the dev directory, it really doesn't matter. So I'm gonna jump over here. Not the dev directory, dev-env directory. All right, so I'm gonna jump over here and I'm gonna create a file called out, right?

[00:04:13]
It doesn't really matter, I don't really care what this thing is called. There we go, it has all these beautiful things in it, okay? There we go, there's all of our items. And so now if I go here and I cat out, there they all are. Okay, everyone sees them?

[00:04:25]
They're right there. So here's some cool things you can do with JQ. We're gonna start with something really, really simple. You can either pipe it to JQ or JQ can read it as a file. So if I pipe cat or cat out piped into JQ, and now makes them all pretty.

[00:04:38]
Look at all that, look how nice that is. You can then re-pipe that into the JQ and make it compact. That's pretty cool, right, look at that. Okay, that's just a little bit of cool, just a little bit of cool. Notice it's by line. So that way you could have, like imagine you have log files and you need to go through thousands upon thousands of log files.

[00:04:59]
You can now start saying, okay, I can see this thing becoming inherently useful. Just wait, we're gonna get way more awesome. I think I'm jumping ahead, you have prettify log files. You can also do it this way, right? JQ with no first argument, and then the file it wants to read, and I'll just do that as well.

[00:05:16]
I'm always just in the habit of cating the file and then piping it, I don't know why. Okay, this is just who I am. All I know is, every single time I do that, there's always some guy, grep actually can take a file and it's just like, I know.

[00:05:27]
I just don't remember, it's just who I am. Okay, all right, so check this out. Do you see this thing right here? All this stuff, what I'm gonna do is, I'm gonna do something. Okay, I'm gonna do something, it's gonna get a little weird here. I'm gonna go back to devprod-2.

[00:05:40]
I'm going to search for, wait, what was it? Check this out, okay, so here it is. There's that data that I just showed you, right? It's big, okay, I'm in vim right now. This is fully unprettified version. I'm gonna grab this and I'm gonna highlight all of this, and I'm gonna take this, and I'm gonna pipe that out to jq-c.

[00:06:03]
Look at that, right, in your editor. You can actually use your editor to manipulate with JQ as well, which makes it even better. Now, this does, I'm sure, what's it called? VS code has ways in which you can send stuff out to something like JQ and then have it edit it and then replace the text and do all that.

[00:06:19]
JQ just allows really fun stuff. So if you just had a log file that you knew you could do destructive operations on, you could do stuff, see what it does, and go, okay, I like this. Okay, I like this, I like that. Okay, good, boom, it's in the right setup.

[00:06:33]
Now I'm happy. Now I can walk away and so you can do stuff like that, which makes it super, super fun. Absolutely think this is just the best way to do things. All right, sorry for the little detour. I'm really sorry, that's actually pretty awesome. All right, so, let's look at the data.

[00:06:46]
Let's pretend we want to sum all the foo values per struct. You had a log that had a list of numbers and you just wanted the sum out of each line. If you're not familiar with JQ, you'd probably naturally just reach for a node program, cuz J sounds really easy, or maybe rust and define all your stuff.

[00:07:06]
And you do something, right? You would probably reach for a program. Well, that's generally pretty annoying. I would say, when you can just go cat, let's just cat the out. I know you can provide the file, shut up. All right, so let's go like this. Let's go select.

[00:07:19]
So this allows us to select objects that have a certain thing about them, something like this. How about, if ( .type == "foo"), There we go. If I just execute that, look at what I get. I only get two results, I just get the two foo lines. Okay, so now I'm selecting the things out I want, that's pretty nice.

[00:07:40]
Well, then I can take that and I can pipe it in, or I can just select out, say the values. Okay, that's pretty nice. Now I can take that and I can add them all together, and there we go. So I just went through, I filtered my objects, grabbed out a property and then summed on that property.

[00:08:02]
It's pretty great, right? I mean, I think that's pretty great, by the way, JQ does not stand for jQuery, okay? JQuerius is very upset about this land grab. It's not true, no matter what people say. All right, so let's do the same thing again, but this time, I don't wanna do a destructive operation where I lose my object.

[00:08:27]
Okay, so what I wanna do is I can go like this, valueSum equals, I think it's this. I may have to remember again. So forgive me if this doesn't work, it does work. Okay, there we go. We now did type foo values. And then I created another object called valueSum and they're now still JSON objects.

[00:08:47]
So that's super good, right? Now we're actually able to do operations and change the underlying object itself really easily. Again, I can't tell you just how much time these type of things have saved me in my life. I don't have to write programs, I don't have to do line by line stuff.

[00:09:04]
And it'll get even better when I show you other tools that you can use with JQ at the exact same time, we'll get there. Which is pretty great. So now you can even do even further amazing stuff, which is we can take this and we can re-select on top of it.

[00:09:20]
I only want things with a valueSum that is greater than 20. Now it only gives me that one that has a sum that's greater than 20. So you can keep on filtering and selecting and doing these really cool kinda ad hoc operations, that allow you to kinda filter through job or a JSON.

[00:09:36]
And so there's just a tool and you should just get familiar with, at least know it exists, get it on your system, and then talk to ChatGPT about it. And GPT will probably help you do a lot of cool stuff, I've used it a ton of times. I just wanted to show you that it exists, this will just massively improve your life knowing about it.

[00:09:54]
And you can do all this fun stuff like this just giant thing that we're doing right here. Theoretically I could grab that whole thing, jump into here, highlight this, and do the exact same thing, and now we just have a single item right here. So I can do cool destructive things.

[00:10:09]
So if you figure out how to do it in your editor, you can do all that, I mean, just think that's great. All right, fantastic.