Jobs-Based Tools

[00:00:00]
>> Brian Holt: So let's do a jobs based one here as well. So this would be very useful. As you can see here, Sonnet is doing just fine with this. We are kind of being slightly wasteful with how many tokens we're going at with this cuz it's having to do several API calls.

[00:00:16]
This would kind of optimize that a little bit, but not necessarily the reason that I'm focused on this. If you're working in AI, you just have infinite vc, just go burn their money. That's what I'm saying, right? That's really what I'm saying is just go burn your VC money with this.

[00:00:33]
Where this actually becomes very useful is you're on an edge device. It can only fit so much space on this particular model. It's a Raspberry PI, it only has a couple of gigs of VRAM for you to mess around with. This becomes even more important so that you guarantee that your job is getting done correctly the same way every single time.

[00:00:52]
Let's go, let's do that. Yeah, the success in particular is like night and day. This went from never working with the API based MCP server to working every single time, right? That's how big of the difference was. So, we're talking about here how I would decide to do something like this.

[00:01:17]
If I see people using the API the same way every single time, that's a good way to designate. It's like, okay, this is a something I need to go build as a job into my MCP server. So one of these things as the NEON product manager of our MCP server is we saw a lot of people trying to run migrations through their MCP server.

[00:01:40]
And as we've discussed previously, LLMs are just not good at managing database structures and actually giving them the raw capability of just doing it Yolo. They'll just drop tables and say, whoopsie, right? I didn't mean to do that. What do you want me to do now, boss? Right, so we went and built a tool for migrations so that we could kind of control some of that chaos, right?

[00:02:04]
So let's say that you're the product manager of this particular issues tracker. Let's say you see that people are creating new bugs and marking them as high priority. And that continually happens again and again and again and again. Rather than making LLMs try and go out and say, okay, go fetch the bug's ID and then create an issue with it.

[00:02:25]
Just make a create bug job, right? And so that it's just built into the server and it just works every damn time, right? That's my point here with job space. Thing is like try and condense this complex cascading effect of like do this, then this, then this, but don't do this first.

[00:02:41]
You have to do this one first. And then to just like, here's the job that you wanna do, give me the information you need for this job and just call it good. There's not really any code difference here. This is just a mindset of how you build these things differently.

[00:02:56]
Let's go. And we can actually just copy and paste this whole thing because I don't really want to make you write it again, which is to say that I don't want to write it again myself. We're going to say copy that. We're going to go over to my VS code here.

[00:03:12]
Right here I have API based one and I'm going to create a new file here called Jobs based tools.JS and just paste all that in there. It works exactly the same way. Job space tools, make requests. That one exists in there as well. We have a create bug.

[00:03:41]
A it creates a bug with new issue with high priority and a bug tag. There's create feature request, creates low priority feature request and update ticket status, right? So let's say that I work with a bunch of customer support people and I just saw that this is all it was.

[00:04:00]
It was either feature requests or bug statuses or updating tickets. I think you could probably see that that's probably where a lot of these could go. I would just create these three tools and not give them anything else, right? As opposed to you get bad human actors that might go in there and say, I'm gonna create this as an urgent bug because it's urgent to me and I'm gonna assign it to you because I know you'll get annoyed by this, that you'll go do it.

[00:04:26]
And then you can also guard yourself against bad LLM actors of like they're creating tickets willy nilly that don't fit in any particular way and they're not being assigned correctly and all that kind of stuff, right? So we did jobs here. We're gonna just comment this one out and say, import jobs based tools from jobs based tools.

[00:04:52]
And then I'm going to comment that out and I'm going to say jobs based tools. So instead of doing API based tools, we're going to do job space tools. We're gonna save that. We're going to come into Claude here. We're going to restart that. Come back into Claude.

[00:05:17]
Now, if we look here at our tools, we have the issues. It has three tools. Create bug. Create a feature request and update ticket status. Create a feature request to add Dark mode to the admin app. Fill it out with a very convincing monologue about the dangers of white light in your eyes and how Dark mode is the singular path to glory.

[00:06:03]
I have no idea what's going to happen there. Yeah. So, I have to go get an API key. [LAUGH] Add dark mode of admin. A critical enhancement for human survival. This one's. That's amazing that we both know it should be critical. This is amazing. So let's go see what it actually wrote there.

[00:07:18]
Man. Every day countless administrators suffer under the tyrannical glare of our blindingly white interfaces. Their retinas burn, their souls wither. Their productivity crumbles like ancient parchment in the desert sun. Jeez, this wax poetic. Dark mode is destiny. Dark mode is excellence. Dark mode is the way. I'm just gonna put that as my email signature anyway.

[00:07:44]
Okay, well, so you can see this is obviously the best way to do MCP servers. I have proven this now, right? This is actually the best way to do MCP servers. Thank you, Claude, for your assistance. Then let's add, so if we look at one of the other ones we added here, which is job space, we had an update status here as well.

[00:08:17]
We need the ID here as well. And here we can say, please update Dark mode to done. So it'll call our tool and we'll see. You can get an amen [LAUGH]. Good Lord, this is how I keep my job entertaining. Praise the dark mode. Praise to be the shadows.

[00:09:14]
The way has been illuminated by darkness itself. [LAUGH] Glory achieved. I saved. That's amazing. Okay, so any questions about jobs based on tools or anything like that? Again, it's a paradigm shift. There's no coding difference here. It's just like a way about thinking about creating these things. This is probably one of the most important things that I've learned in terms of just like authoring MCP servers over the year-ish that we've been supporting it.

[00:09:42]
It definitely will help. And if you Google API based MCP Server, there's a bunch of thought pieces on this now as well because a lot of people are hitting kind of the same results because it's like a natural way to think about it. It's like, it's just an API.

[00:09:55]
Let's just wrap our API in MCP server. So don't do that.
>> Student: Can we think of jobs based tools as more specific API based tools for particular common use cases. Or is there something more to them?
>> Brian Holt: One, it's a term that I'm inventing. There's no official capacity for this.

[00:10:15]
It's more just like working backwards from what users want, which is what we should be doing always, but working backwards from what we want an agent to accomplish with our. In this particular case, again, if I was building this for like a support center, those are the things that I want them to be able to do via this service.

[00:10:32]
So if I just give them an API based MCP server, they're going to be able to do anything the API can maybe, if it succeeds, right? Or I can work backwards. Like they create feature requests, they create bugs, and they update tickets and that's all they want to be able to do.

[00:10:50]
Less tools, less room for possibilities of going wrong. It's just cleaner, it's better, it succeeds more frequently. You're less subject to the chaos of the LLM, which is what you want, right? So again, there might be a better way of describing this. Maybe not jobs based, but the way I got that is jobs to be done, which is a big framework for creating product descriptions, right?

[00:11:19]
I just kind of worked that into jobs based MCP servers.
>> Student: How do you structure your MCP API surface for exploration, for example, when you don't know all the jobs yet, but want to provide your company with as many capabilities as possible?
>> Brian Holt: That's a good question. I still wouldn't give it the entire API because at that point you're still kind of like subjected to all that chaos and your LLMs are either going to just waste a bunch of tokens or they're going to just not be successful.

[00:11:54]
So more I would just go broad in terms of here's all the possible things that I would want them to be able to do and then I would watch the usage of those APIs and if you create one for auth issues and no one ever reports any auth issues, one, maybe they're just not being able to log into the site, which might be a problem itself.

[00:12:15]
But if no one's ever using that particular tool, then I'm at that point might just go remove it. I just go broad and have 20 tools and then reduce it down to the seven tools that actually get used. Something like that. We had a problem with that with the Neon MCP server where we had one tool that we wanted people to use, but the LLMs are always selecting a different tool and then composing that to solve for the other tool.

[00:12:44]
And so that tipped us off that the prompting for the tool that we wanted people to use just wasn't prompted very well. So we went back and refined the prompt, and then once we did that, that everything worked well. This all comes down to monitor and observe how people are using your stuff.

[00:12:59]
Right. That's critical.