Cron for CI

[00:00:00]
>> So we're gonna make a CI/CD pipeline, but the first thing we need to do is a stopwatch. How do we do a stopwatch on Unix? We use something called cron. Cron is really powerful, but at its core it's a kind of a simple concept, it's just do this thing at a set interval.

[00:00:17]
That's it, nothing magical about that. But, if we try and cron, do this thing on an interval with a shell script that says, hey, here's a shell script that pulls from GitHub. Anybody see where I'm going now? Yeah, we can say cron, pull from GitHub and check for GitHub every minute, so it's always gonna be pulling into production.

[00:00:41]
So we're making a CI/CD pipeline. Without the tests, I wanna be clear, we didn't write any tests for this, you should not do this, this is a hack, but it's a good example of how CI/CD works. I even called fake CI/CD cuz I want to be really clear, you should not do this in a production system.

[00:00:59]
This is just for entertainment versus purposes and to make our lives a little bit easier. When we're writing code on our computer, we now don't have to pull every time from the server, the server will do it automatically for us. So what we wanna do is we wanna create a bash script that get pulls from the main branch.

[00:01:18]
We wanna create a cron job to execute that script every two minutes, or one minute, it actually doesn't matter too much if it's two minutes, one minute. But I'll walk through this because we have not really touched on bash scripts, but when you see it you're like, that is surprisingly, that's much easier than I would have thought, Jem.

[00:01:38]
It sounds complicated, but it's not too bad. So the first thing we do is I have an example of a shell script here, Here at the bottom, and that's just what a shell script looks like. You start with user bin hash, the location of the shell, and you can read variables, you can say now is the date.

[00:02:05]
You can have input using the read dash p, we'll provide input, and then use echo with all that out. And I see some confused looks on your faces. How about I just write this out? Would that clear things up a little bit? Yeah, let's give it a shot.

[00:02:23]
Let's give it a shot. So I want to make a new shell script, so I'm gonna say, vi, and I'm gonna call it test.sh. The sh just means it's a shell script. Wait, I don't know where my bash is. So before I do that, every shell script has to start with the user, then dash, the location of that.

[00:02:54]
But I can just say which bash, and it tells me the location of the bash, and it's user/bin/bash. That's usually where it is, you never know. So vi test.sh, insert, and we always start with the octothorpe hash, whatever you want to call it, bang, that means it's a script.

[00:03:18]
We'll say usr, /bin/bash. And I won't do the whole now date time thing, not super critical. It's just a way of setting dates, but I will say we'll do a read. This just means it's waiting for input from you, the user. We'll say, what is your name? And then we're gonna assign that input to a variable called name.

[00:03:59]
And then we're gonna use our good old echo command, and we'll just say, I don't know, Have a great day, $name. Then we're just gonna right-click that, I'll give you a second if you wanna play along, you don't have to do this part. This is just demonstrating shell scripting and kind of, it's pretty fun, it's pretty easy but very useful, really, really useful.

[00:04:29]
You don't have to spin up an entire programming interpreter like JavaScript or Python, or Java text you commands on your machine. All right, so let's go and try to run that. Sweet, should just be able to execute it, up 10 knots, so that's sh. We're gonna try to run it here.

[00:05:01]
I need to set permissions. So, I'm gonna hack because I don't remember the permissions to run it, so I'm gonna CHMOD this. Don't do this at home, 777, guarantee the work, oops. Still not working. There we go. Jem, have a great day, Jem. And why do we have to change the permissions?

[00:05:36]
Because by default it's not set to execute. Nothing's set to execute by default. I probably should have tightened down the permissions a bit, but it'll work for now. And that's it, so you just wrote a program without actually having to open another file, have it run in node or anything like that.

[00:05:53]
So shell scripting is really, really awesome for writing scripts. You're probably not gonna write very robust things in it, but it's a good tool to have in your back pocket, and it's pretty straightforward when it comes to shell scripting. So now, let's jump back to our original task which was, we wanna create a bash script that pulls from main branch.

[00:06:15]
So what's the command the pull from main branch?
>> Git pull origin main?
>> Git pull origin main. And how to write a bash scripts, just by putting the hash exclamation bang, hash bang, I think it's called hash bang, yes. You put the hash bang user bin bash, and then you say git pull origin main.

[00:06:38]
That's it, and you wrote a shell script to pull from GitHub. Not too scary, right? When we started out, you saw these instructions you're like, that sounds really bad. I don't know if I'll be able to do that, but now you see it's not too bad. So I'm gonna do that now, before we move on to the cron job part.

[00:07:04]
So I'm gonna rm test.sh, and I'm just gonna call this one, vi github.sh, and I'm gonna use my hash bang. So usr/bin/bash, And I'm just gonna say git pull, origin main. That's it. I'm not sure if I have permissions on this one either, nope. So again, I'm gonna do this the bad way, I'm just gonna chmod 700.

[00:07:58]
And let's test it out. There we go. So we wrote a two-line bash script to pull from GitHub. So that's half the battle here. We have a way of automating it, but we wanna do this on some sort of cadence. So to do it on a cadence, we're gonna use cron.

[00:08:18]
In cron, the first time you see it, it can look a little confusing, it's a bunch of numbers, and then a path. Doesn't make a lot of sense, but when you break it down by understanding, the first one is actually just the minutes, the second one's the hour.

[00:08:33]
The third one's the day, the fourth one's the month, and then the third one is the day of the week, it's important to you. Then you're saying, what should I use to execute that script, and then pass the script. And that's all a cron job is, it's just a timer and the thing to execute on whatever cadence you want.

[00:08:52]
And the way we edit our cron jobs, we use something called crontab. So we say crontab -e. So I'm gonna show you a much easier way of doing this, cuz you're gonna be like, what's the task again? Is it run at every 10 minutes? Every minute, how do we do that?

[00:09:08]
So, thanks for the beauty of the Internet and other people have struggled through cron jobs and trying to get the timing down. We can look at a nice website called crontab.guru. And I love that .guru is a TLD. I should have got that full stack .guru. Darn it, see?

[00:09:30]
This domain names I always have regret. So crontab.guru is amazing because you're like, I wanna run a job that runs every, I don't know, every 10 minutes on, A Tuesday. I think Tuesday is a 2, yeah, there you go. And it shows you that's how to write a cron job that runs at minute 10 on Tuesday.

[00:09:57]
Or if I say, I only want this job to run, I don't know, at minute 10 on Tuesday in April, we can say that, if we say, So, that's actually how to run every tenth minute, the */, otherwise it's gonna be every minute 10 on Tuesday in April.

[00:10:20]
So, crontab.guru makes it really, really easy, to figure out, hey if I wanna run a job on every 10 minutes every so often, how do I do that? So much easier than trying to play with it and hoping you get it right. So what do we say our job was?

[00:10:38]
We wanna run it every minute or every 2 minutes. So we would say, that's a star, that's a star, we want it every 2 minutes, And that's what we'd write a cron job, turn every 2 minutes. Now if you want every one minute you can say 1, or you can just leave a star.

[00:11:05]
And it's by default, it's gonna run every minute. So now, I feel like I wrote my scripts, I feel like I'm prepared to set up the cron job, let's go ahead and do that. Clear, let's run crontab.e to edit the cron jobs, and I'm gonna use Vim, it's gonna ask you the first time what you wanna use, let's stick with Vim.

[00:11:37]
I'm gonna jump to the bottom here, And I'm going to write a new line. So, what was that command again? It was star, I'm just gonna be explicit, 2, hour, month, day, week. Then we're gonna use bash to execute this script, so we're gonna say sh. And then we'll give it the path to execute on, so that was /var/www/app/github.sh.

[00:12:34]
Everybody here with me so far? Yeah, not too bad. When we started out, it sounds much more complicated to this. But again, a stopwatch and a few lines of code, we can create a fake pipeline. So let's go ahead, and I'm gonna actually set it to a minute, just to be sure what's happening.

[00:12:56]
Just to make it a little bit easier on myself, so I don't wait 2 minutes. All right, so how do we know if it's running? We can look at the syslog where cron jobs are gonna be output. So we'll say sudo, cat, var/log. And our syslog captures everything going on in our system.

[00:13:25]
And we see here, it's just saying, hey, Jem edited the crontab, just letting you know. Cuz everything in your systems gets logged somewhere, and generally Syslog captures most of what you're doing. But we wanna wait, we wanna make sure it's running. So I'm gonna check it again. It still hasn't run because it's been a minute.

[00:13:48]
So I'm gonna use this command instead, And we'll talk about how to read logs in a second. But I'm gonna use the tail command and we'll set it to follow. And that's /var/log/syslog. Now we wait. So we see here it's running, but it doesn't look like it's outputting anything, why is that?

[00:14:16]
We didn't tell it where to output the logs, so it's not actually doing anything right now. So I'm in my /var/ app folder, and I want to take a look at that GitHub script cuz I forgot one thing. And I did this last time, I do this every single time and it's so easy to forget, I forgot to add this line.

[00:14:40]
So we had the get pull origin main, but we forgot to tell it where to go. Cuz we made the assumption that the script is being run from this directory, but crontab does not run from this directory. Crontab runs from, I think root, so we have to tell exactly where to look.

[00:14:54]
So, first step is adding the CD/var/dub, which is just the location of our current script. The other line I'd add, and this is not necessary but it'll make your life a little bit easier, we're just gonna say fast forward only. That way automatically it'll pull in the changes that are different, but it's not going to collide with your existing changes, you won't have to do a git rebase or something like that.

[00:15:27]
I should add, it's --, there we go. So we've gotta make sure with any script we're running that's dependent on being in a certain place and location, we have to specify that for crontab. And there's one other change I wanna show you because again, I went on about redirection and logging, and I said well, we need to log these things, but it's still not working.

[00:15:51]
Let's go ahead and edit our crontab one more time. So that's crontab-e. We actually need to pipe our output properly into the Syslog. Cuz as it is, the Syslog will just, or Cron job will drop logs, it won't necessarily put them in Syslog until we tell it where to go.

[00:16:13]
So this is just gonna pipe the standard out and standard error into standard in for the next command, and this is just shorthand. There's different ways of doing it, but it's 2>&1. Then we're using the pipe, and we're gonna pipe it to something called logger, and logger is pretty easy, it just logs things for you.

[00:16:45]
You don't have to give it a name, the T is just saying what's the name of the command we're gonna log, it'll just make reading our log file easier. But you can drop this part, you can just pipe it all into the logger. So now, let me quit out of here.

[00:16:57]
Now if I say sudo tail -f, and we're looking for that log. And it's still gonna be in Syslog, we didn't change that, var/log/syslog. And now we see the logging is working correctly, cuz we piped it to logger, and then we gave it a name, we just called it github.sh.

[00:17:23]
We could have named it anything, we could've dropped the name, it just makes it easier to parse the log file. And now we see it's pulling regularly from GitHub, because we didn't make any changes, it's already up to date. So again, a few extra commands makes it all work together.

[00:17:41]
I got a little excited about the other things, I forgot to actually wrap it all up. But now, if you make a change on your GitHub page or on your repo and you push it up, it will pull down automatically to your machine. That's pretty nice, so we don't have to do much more coding, or we'll say JavaScript programming on the computer anymore, or on your server anymore.

[00:17:59]
We can do it on VSCode or whatever you like. All right.