Linux Core Utils & sed Tips

[00:00:00]
>> Here's another one that I thought was ridiculous that I can't believe I didn't use before. Let's solve one more problem using just the shell right here, we're gonna have a bunch of lines of text. Each line of text is gonna contain whitespace separated characters, right? Or whitespace separated words and each one of these words are gonna be a number, we're gonna wanna sum them all together.

[00:00:21]
And then, I wanna find the maximum sum. Now, if you remember earlier, if I go to work temp, pipe actually kinda did this. It went in here, it split the line and it did a sum. What it didn't do is it didn't find the maximum one. So we actually kind of did this once before.

[00:00:38]
Which this took, requiring a module, creating an interface and reading it line by line and printing it out. So if I wanted to sum it, I'd also have to be able to wait for this thing to close, then go through and find the largest number. It'd be a little bit more work to make this one work.

[00:00:54]
But it's a little bit easier with some core usals. So let's go up here and let's do this one, there we go. We already have ourselves a couple lines that have a bunch of numbers in it. There we go, look at that, look how easy that is. So what I can do is I can actually take this and I could pipe it to tr.

[00:01:12]
And remember tr does all those sweet things. I'm gonna take it and I'm gonna replace every white space with the plus sign. Look at that, how great is that? Now I can take that and pipe it to calculator. Look at that, I just summed it. It just summed all those numbers.

[00:01:28]
I mean, how fantastic is this, tell me you're not feeling fantastic right now. I'm feeling so good. And so then after we take that, we gotta find the highest number. So remember, we can sort with natural numbers and reverse it. 39 is now on top. This is looking really good.

[00:01:43]
So then all we need to do is go ahead and we only want the first result. Look at that, I just solved it, just solved that sweet problem in just so few letters. This is one of those, I cannot wait for the day that I have an interview and someone asked me something like this and all I can do is just write core utils instead of writing a program.

[00:02:00]
It's gonna be the greatest day on earth. I just cannot wait for that to happen, because they're so useful and people don't realize how many easy problems you can solve just using this. I just used a couple little quick things. Yeah, sure I'm spawning a bunch of processes.

[00:02:14]
It's not resource efficient, but it is certainly by the way factor efficient. And the reality is, is that if you're doing your job, often you have some sort of log output. Some sort of set of strings you have to manipulate or search through or to understand to be able to debug something or gather some sort of information.

[00:02:33]
If you know core utils better, I find that this makes my job so much easier cuz I can do so many forms of grabbing. I remember having this memory problem and I had to go through a whole bunch of stuff and it only happened in certain memory conditions.

[00:02:46]
So I couldn't just put a breakpoint in it, plus it required a bunch of programs to run. And so instead I just printed stuff out and I printed out 10 million lines of logs. And so I just needed to find a series of things happening in those logs, it was obvious I could never use my eyeballs.

[00:03:01]
So instead I just kept reducing it using some simple string processing and it became magical. It's just so nice having these things in your tool belt, I strongly recommend taking a day and go and learn some core utils because it can just solve so many problems you don't realize it can solve.

[00:03:18]
Now, not all the problems are great, but some of them are really, really great. Now here's another question. I'm curious how many people would have used awk, write a program or just used bc sort head. These are all things you could have used. You definitely could have used awk to do that, awk has this cool kinda like array syntax, you can print it.

[00:03:34]
But then the printing is a little, not as easy cuz you have to do for each entry in the dictionary and print out each line, and blah blah blah, but either way, this is pretty cool. The fact that you can use these is really, really nice. There's some other utils that used, I've used all these right here, throughout this presentation.

[00:03:51]
You should just know these, grep is probably the one I haven't used the most. And by the way, just me saying the word grep, the proper Rustaceans would have interjected, well, actually RG is better, RG's grep written in rust. So you should use that because it's faster and safer and better.

[00:04:06]
And lastly, I did wanna give you one more quick tip on sed. If you're wondering why I'm moving a little fast, I have a really super cool session. One last thing I want to cover and we're kinda getting near the end of the day. And I didn't really wanna talk in depth about a lot of these core utils.

[00:04:19]
I just wanna show you the type of problems you can solve with it. So you really feel empowered to go look for the answers, as opposed to me just giving you the answers, cuz ultimately, you're not gonna learn if I only give you the answers. You're only gonna learn if you do this yourself a few times.

[00:04:34]
So some tips on sed. Sed is like one of those things that feel super confusing. I'll give you an example, echo hello world. Now, you can take this and you can pipe it into sed, and you can say, I wanna take each line, and I wanna replace occurrences of world with front end masters.

[00:04:53]
End it with the slash and there you go. So instead of saying hello world, it says hello front end masters. It's pretty radical. I love these kinda things, I think they're super cool. You can do really cool things solving it with sed. Look at that, someone's doing a nice little sweet sed move instead of using tr.

[00:05:13]
Cool move sed guy. So there we go, you can do that, awesome. You're using sed, you see that we can do simple replaces. Well, for me, the language you can use in sed can be sometimes a bit confusing. But what I found to be really nice is that I use VIM.

[00:05:32]
By the way, I don't know if you know this but I actually use VIM, I needed to just say that one last time. So I wanna find, let's see, there has to be a section that is kinda confusing. This one here, actually, let's go back to here and let's use, I'm in my, there we go.

[00:05:46]
Let's use the sessionizer, here we go. Let's say we wanted to take sed and replace everything that is found in quotes with something else. Now, obviously we could open up our editor, use find replace, go through each one of those, but let's just say we just wanted to do that really simply.

[00:06:03]
We wanna replace everything in quotes with the word quote. So, how I kinda go about some of these things is I actually end up using vim. So I come in here and I go like this s slash, or before I do that, you have to set incsearch. This is very important, set in search allows you to have incremental search.

[00:06:20]
If you forget these things, I believe it's chapter four inside of the vim tutor, which by the way, World Record speed runner vim tutor. Just wanna throw that out there right here. World Record, 2 minutes and 57 seconds I believe, all commands. 100% no cheats, nothing but pure jumps.

[00:06:36]
It's beautiful. So, if you wanted to do that, what I do is I go like this. I go, you can start searching and it will auto highlight for you. This syntax is almost identical to sed syntax. So often I just use them as my test ground, and then I can just go and do it elsewhere.

[00:06:55]
So let's go like this. Let's take this and I go, okay, I wanna search for that. And I could then .start to the next parenthesis. That sort of works. If you have multiple strings on a single line, my guess is that won't work. So let's try it. Let's go right here.

[00:07:10]
Yeah, look at that, it doesn't work. And so you can kinda see how these things work. You can kind of work with it and you can start playing and go, that doesn't work. Maybe I need to do some other type of lookup. I don't know how these things work, I'm terrible at regexes, I don't plan on ever using regexes.

[00:07:25]
But either way, you can go through, use all these things and figure out how you want to replace stuff. And then you can go to sed and use that. That's kinda been my strategy for a long time to use more complicated sed operations. Just cuz it just makes it a lot easier and you can visually see it happen.

[00:07:41]
Yeah, that's what I've always done with them. And I figured that'd be like a nice little, a quick tip on sed if you ever have something complicated that you need to do with a log. And you can do it, just some ad hoc regex, open up vim, put an example in there, do it in vim, go, okay, that's the exact one, I can see it happen incrementally, awesome.

[00:07:59]
Put it inside sed, it should work for the most part. I wanna get to this one last little section. This is the last and final section of the day, and this is where I feel like I'm gonna be the most excited. Cuz this is kind of interesting, cuz we're gonna talk about today and the future.

[00:08:16]
The future of mankind, 2022, 2023, who knows, maybe even 2024?