curl & HTTP Verbs

[00:00:00]
>> I wanna show you now, we can come back over to our primary. If you want to, we'll just do this, I'll separate this out into two different terminals so you can kinda see them side by side. This is one reason I do like Windows Snap Manager. So you drag it to the side and it snaps to the side.

[00:00:22]
I think you can do that as well with Mac, I'm too lazy to have enabled it. So now I can come over here and I can say curl that address, put http in front of it, ://:8000. And if I do that, you can see here, it sends me back that same HTML document that I got in the browser.

[00:00:47]
And you can see it hitting here, and it looks pretty much the same. The only thing that changed is the IP address, .the 1 is the host machine, right, my Mac, and the .2 is itself, right? So this is actually requesting against itself. And then it's doing it over HTTP 1.1 200, right, everything else looks relatively the same.

[00:01:13]
So again, this is just putting it out to standard out, right? But if I say the same thing, and I put that into res.txt, does the same thing. You can see the request happened over here, but now it goes into res.txt, right? So that's how you would do that same sort of wget sort of behavior, right?

[00:01:39]
And another way to do that, You can also just do dash, I think capital O? Yeah -O res2.txt. And you can see here I now have. Did that not work? It didn't, was it lowercase o? Yeah, okay, sorry res lowercase o, that'll output it to res.txt. So that way you don't have to pipe it, it just ends up in that file.

[00:02:17]
And then if I do -O with no file name, it'll just keep the same file name. So let's say I was doing it to /output.txt. You can see here, it actually overwrote it. Cuz it's requesting from the same directory, right? You can see here it hit output.txt, it got a 200, so on and so forth.

[00:02:46]
Good so far, make sense? All right. So let's say I'm doing an API endpoint, and it's something you need to POST against, right? Well, that's really easy to do just to curl -X POST. And this changes the HTTP verb. If you're not from the HTTP verbs, just go look it up on the Wikipedia, the Wikipedia article is great for it.

[00:03:10]
But there's GET, we've been doing GET requests this entire time. There's POST requests, PUT requests, PATCH requests, DELETE request, OPTIONS requests, HEAD requests. And there's more, but those are the primary ones you would ever see. But anyway, you can do curl -X, and then you can put whatever verb you want there.

[00:03:28]
You can even do made up verb, and it's like, okay, I can do that. I don't know what it does, but you can totally do that. Let's do POST for right now. And then against the same http:, what is it, 192.168.64.2:8000. And you can see here it actually responds with HTML, it says we don't know what to do with a POST, which is fine, right?

[00:04:02]
We don't expect Python to be able to deal with it. But you can see over here, we were doing a GET request before, right? And here, it's now changed to a POST request, right? And in this case, it gave you a 501, 501s mean that there was a server error, right?

[00:04:18]
In this particular case, the server's like, I don't know what to do with POST requests,so you messed up, right? Okay. Good so far, make sense? All right, so let's move on the next one. Let's say you wanna send a POST body. So if I do dash -d, and then here I can say, this is the post body.

[00:04:51]
And I send that across here, notice that this is still a POST request. And it's a POST request because GET requests don't have bodies, right? So put only POST requests have bodies. And so implicitly by giving it a body, it's like, okay, cool, I'm gonna make this a POST request.

[00:05:08]
So yeah, just know if you do a lowercase d like this and you send a POST body along with it, it's gonna implicitly do a POST. If you wanted to, you can do a -X PUT here. And you can see, it'll send that along. It is a PUT request, but it's also sending a body with it as well, right?

[00:05:25]
So you can kind of mix and match those options as you need to. And yeah, I mean, you can come in here and you can do wherever you want. I can do curl -X DELETE. Do I have that here? Good. Let me just copy that so that I don't have to keep writing it every single time.

[00:05:55]
You can see there DELETE work. And again, just to prove my point is you can put LOL_ANYTHING here, and it'll go ahead and send that. And it'll show up here as an HTTP verb over there. So hopefully you're starting to see this can be really useful for testing out your API.

[00:06:11]
So you don't have to have some sort of fancy Postman client or anything like that, you can just use curl. Another thing that's really common is you need to send cookies with it as well. That's something you can definitely do here as well. You can say curl -b, and you say "name=brian", and that will be sent as a cookie.

[00:06:34]
And I can then add like -X PATCH. And this doesn't show it here, but it's also sending whatever I put in the -b as a cookie along. Something I'll frequently do is I'll actually go start a session in the browser, get all my cookies set. I will then copy everything that the browser's outputting for those cookies, and then I'll put it into a file.

[00:07:00]
And then you can use something called a cookie jar. Not making this up, that's actually what it's called. And I think it's -B. You can see here curl has an enormous amount of options as well. So one of these in here, so that's cookie, it's -c. So if you put -c, you can give it a cookie jar file, which is just a file that contains all of your cookie information in it.

[00:07:26]
Cuz normally you have long things that have session IDs and stuff like that. So that's useful to do as well. And yeah, cool. You can have it follow redirects. By default, curl won't follow redirects. So I have a short link for this course that if you go to https://bit.ly/linux-cli, it's gonna say this file is actually a redirect that goes here, right?

[00:07:58]
And it's also sending, I think that's gonna be probably a 302 redirect. So if you give it a -L, it'll follow the redirect. You can see now if I put the -L, it actually sends literally everything for that course website along for the ride. Typically, that's not something you have to do, but it's good to know that it won't follow redirects.

[00:08:26]
You have to tell it, yes, I do want you to follow the redirects. You can send headers as well. So curl -H "accept-language: EN-US". And if you wanna send multiple headers, you just give multiple -Hs. So you could give it like an Authorization header Bearer: 12345. Http:, yeah.

[00:09:03]
Okay, and that's how you would send this here, just use capital H multiple different times. And then I'm gonna show you something that this is probably one of the things I do the most. So let's say I come back over here to my browser. And let's just go to my website, so bit-ly, this is fine.

[00:09:28]
Okay, so this is in Firefox, what I'm showing you right now, but this will work in Chrome, this will work in Edge. And I imagine all of you are using at least one of those. I imagine it works in Safari, I honestly didn't check. Okay, so I'm gonna come in here to the Dev Tools, which I opened with Cmd+Option+I.

[00:09:47]
And I'm gonna refresh the page here so I can find all of the network requests in here. And let's say I wanted to get this one, right? So I'll click on this network request. You can right-click on it, and you can say Copy as cURL in here. And again, I checked this with Edge and Chrome, and it should work exactly the same way.

[00:10:14]
So Copy as cURL, okay? So I'm gonna click that, and then I'm going to open, This again, and then I'm just going to paste. And you can see here it copies this as a perfectly formed curl request. So imagine that you're on your website, and you're browsing around, and one of your network requests doesn't work the way you expect.

[00:10:36]
You can just right-click, copy that as curl, and then you can replay it again and again from your command line. So you can see it's sending to the same URL -H, we know that that's a header, so it's giving it the same User-Agent. This is the same Accept header, Accept-Language, it's sending it as compressed, right?

[00:10:55]
So it's gonna actually compress it and GZIP it, Connection: keep-alive, and some other stuff. And also, if you do it with a POST request, it'll copy all of the request body and the verbs and all that kinda stuff. So now if I run this again, you'll see that it gets the exact same request, right?

[00:11:14]
So again, this is super useful if you have something not working the way that you anticipated, you can go and copy it and run it over and again and again. Does it makes sense? It's cool. Again, honestly, if you're a web developer, that might be one of the most useful things I tell you today.

[00:11:36]
All right, and then one more thing about curl that I want you to be very careful about. So if you go to nvm, which is the tool that I use for managing my Node version, and you go down to the installation instructions, It has this right there. Let me make that a bit bigger so you can see it, as the installation instructions.

[00:12:08]
So you do a curl request, and then we know what this is, this pipe, right? So this is going to feed it into something right? And this is going to feed it into bash. Why does this maybe give you a bit of pause? Cuz you need to really trust what is ever in this file, right?

[00:12:27]
So if I go look at this, it's a long Bash script that does a lot of stuff to set up your nvm. Which is a great tool, it's what I use for installing all of my Node stuff. But this is really scary, because whatever goes into this, it's going to execute, and you don't actually know what it's gonna do.

[00:12:47]
So it could totally just dump all of your environmental variables and send them off to a remote server, and you would have no idea, right? So I just wanna give you a huge red flag whenever you see pipe into Bash, that this could be very catastrophic for you.

[00:13:00]
Now, if you asked me well, Brian, then how do I install nvm? It's like, I do this, to be honest with you, I do this. Because I know that GitHub is not going to replace it underneath the hood. I personally trust GitHub. I also work for the parent company of GitHub, so if they did something nefarious, I would be the first to be very upset about it.

[00:13:19]
But if this is Joe's blog, I absolutely will not pipe anything from a domain that I do not trust, right? So what you should definitely do then in that particular case is you can come in here, you can say curl blah, and into my-file.sh, right? And then I will run it from my-file.sh, right?

[00:13:46]
But this way I can verify that I got the same file that I thought I was gonna get ,right? Because again, the server can be smart enough to tell that you are downloading something, and it can switch the file underneath you, right? So you can open it in the browser, it'll show you one thing.

[00:14:02]
And then you can go to curl, and it will change the file to be something else. It's actually even possible for it to detect that you are curl into Bash, right? So it can actually detect that it's being piped by the way that the network traffic looks, right?

[00:14:15]
And I left an article down at the bottom of the notes, it's really interesting to read. Of how you can actually detect not only that you're curling it, but you're also curling it into Bash. And so they can only switch it when you're doing curl Bash, which is wild, right?

[00:14:30]
It's some crazy stuff. So suffice to say, curl it into a file, read the file, then execute it. Unless you really trust, it's coming from GitHub, it's coming from, I don't know, Stack Overflow or something like that. But in any case, always read the file too, right? Okay, so I just wanted to throw that out there.

[00:14:49]
That's a really huge vulnerability that you're opening yourself up to whenever you curl into Bash.
>> Does the curl request itself change when you pipe?
>> That's a valid question. So does the curl request itself change when it's piped? There's kind of two answers to that question. 99.999% of the time no, it doesn't.

[00:15:18]
And it's just requesting something and then putting that into a stream that can be read from or written to, right? As we have seen with other Linux programs. However, that article that I dropped at the bottom of the notes will actually show you it's possible to detect the network pattern of piping something using curl.

[00:15:38]
And so it can actually detect that on a server and switch the file out from underneath you. So it does have a distinct network pattern, I guess, is the the answer to that question. But that requires some real nefarious actions. And something like GitHub's just not gonna do that to you.