Web Scraper with Firecrawl MCP

[00:00:00]
>> Stee Kinney: So you go into Cursor. We saw those Cursor settings earlier. You go into tools integrations, you can hook up GitHub, you can hook up Slack and you can also say add custom mcp. Cursor has some functionality now where the web. You can have a button called Add to Cursor.

[00:00:13]
You can click on it, it will do all this for you. But if you wanted to pull in one because a lot of them could just be NPM packages or you can write your own that has like some bespoke functionality and like hook it into your own systems, so on and so forth.

[00:00:26]
I think the power is probably building your own. But yeah, you go hit that add custom mcp. There is some configuration that I didn't want to show you on mine because I had API keys in there. So this is one for Firecrawl, which is like that web scraper that I told you about earlier.

[00:00:44]
And now it has access to all the tools of that web scraper. So we're gonna use a very, very practical example. I'm sure you had those course notes before on my website. I'll show you again. All the notes from our courses are here. There's also like the previous courses which as you can tell is way out of date for the number of courses that I do.

[00:01:09]
Could I update this by hand 100%? I could, I think I'm missing like five courses at this point and I haven't. So what we could do is we can go to Cursor. Let me just go into my website repo. Open that up in Cursor. I like when Electron apps decide they are only going to open the background of other apps if your computer's been open for too long.

[00:01:38]
That's a thing that they choose. All of them choose to do at some point or another anyway. So we can go into the settings. As you can see, I actually want the Cursor settings which don't have a shortcut. Tools and integrations and I have a few of these.

[00:01:54]
But Fire Crawl has a bunch of tools. Firecrawl Scrape, Map. I don't even know what they do, it doesn't really matter. Let's do, let's see, I wanna make. Doesn't really matter where I do this cuz I'm not actually going to keep it right now. Cuz I'm definitely not deploying something to my own website that I live coded with an MC server in front of you.

[00:02:16]
I will be reviewing it beforehand. I'm just going to find a file and we'll make one called Could I tell the LLM Make a file in this place. I could totally do that, I don't trust it. Take that for whatever you will. Then what I'm gonna do is I'm gonna go to frontend Masters teachers.

[00:02:45]
There we go, look at that handsome devil. Grab that URL. And I'm gonna say for all of the courses and workshops, for each one, I want the title, the thumbnail picture. I'll just say the image and the link. If I was smarter, Dustin sent me an affiliate code, I'd be like and append my affiliate code to each and every one of these links and write it to a JSON file in at.

[00:03:40]
I just made a new one in routes. Hopefully, I picked the right one cuz I used my handwritten one as well. Write it to a JSON file in Svelte. Cool, cool, cool, now my LLM has superpowers because I put in that Firecrawl MCP server. So now it knows it has the tools to scrape a webpage.

[00:04:03]
You'll notice though, there is a little bit of hope here. Probably could have mentioned Firecrawl, but you can see that it chose this new tool that it has because it's the one that Firecrawl created. And you can see that it is going to that page, it is scraping that page and it is starting to put together my JSON file.

[00:04:28]
And I could save this to use on a notepad or I'll show you a Cursor command later and I could run that and I could literally get an updated JSON file of the courses that I could then use for the Svelte site, right? Those are the things where it's like, do you want to say, build a SaaS for me in a weekend maybe?

[00:04:47]
Again, people are doing it, I believe them that are making money. However, writing a scraper was a Saturday that I did not want to spend, clearly because I didn't do it. And then pulling in some of these tools takes things that are not necessarily Writing a scraper is not necessarily like a big brain activity.

[00:05:07]
It's tedious. And if you've never done it before, totally do it, cause it's fun. You'll find out all the ways different websites try to stop you. Real estate websites. Yeah, ask me how I know. But that stuff where you can start to make progress and move stuff. Cuz again, there is the stuff we do for pride as software engineers and the fact that most of us work at companies and we need to ship shareholder value.

[00:05:34]
And being able to pull in one MCP server, add this functionality and do something that I've been putting off forever. Pretty powerful, right? That said, like I said before the joke we said earlier, the S in MCP is for security. Thank you everyone in the chat who donated that gem.

[00:05:53]
That's not leaving my lexicon for a while. And this is one, it's using mpx, so it's pulling it from npm, which is a supply chain nightmare waiting to happen, right? So you could clone it down to at least have a known go one and audit the code. You can also there are some that are remote servers now you're playing with extra fire, you know, and there's a lot of trust but verification where it's like, there's like go find.

[00:06:23]
There's like G Drive. And you know it's not made by Google, if they call it G Drive, by the way, there's like five of them made by just randos. And I'm sure you have to put in an access token. I'm not installing one, so I couldn't tell you.

[00:06:35]
But I am sure that you have to give them an access token to your Google Drive. Absolutely not, right? So keeping that in mind, don't be, wow, he just scraped a page. And that was so cool as a permission slip to be irresponsible, but understand that especially if you are willing to, if you write your own to extend something, I think that's.

[00:06:59]
Obviously, I can create my own security bugs, trust me. Or known vetted ones. I think the amount of trust you can give to the official GitHub one is probably higher. Now, no company has never not been hacked. So understand that there are risks in this. It is an emerging protocol.

[00:07:24]
It has its rough edges, but there is power. And also know that for everyone you add, you are effectively paying a tax.