What is Tool Calling?

[00:00:00]
>> Scott Moss:So the first time I ran an LLM API, I was blown away. I was just like, I don't know, I felt like I had a wand in my hand and there was like somebody on the other line that was like a genie, right? But like, this genie kind of sucks because I can't really ask it to do anything past its training cutoff date, right? Because the way models work is that they're trained up to information on a certain date, and by the time you get access to the model, that training cutoff date was probably almost a year ago because there's like tons of checks for safety.

[00:00:27]
So if I ask this thing like, what is the, or if I asked it like who won the NBA or the 2025 NBA finals, if this thing wasn't trained after that event, it's either gonna lie or it's gonna say I don't know, right? So I'm gonna run this again and we can see what it comes up with. So it's thinking really hard now. And it says, do you, did you mean who won? And then it says, who will? Um, oh, there it is.

[00:01:02]
I should also note my training only goes to June 2024, so I don't have confirmed 2025 results. I can help make a prediction. Uh, so as you can see, it's really not that useful. How do we make it useful? Well, we have to give it the ability to interact with its environment and the way that we do that is with something called tool calling. So let's hop into tool calling. What is tool calling? Well, it's the mechanism that allows LLMs to interact with the outside world, and we're gonna learn how to do that because if you know how to make functions and you know how to describe what those functions do, like you would normally write comments above your functions, then guess what?

[00:01:44]
You know how to make tools. That's it. That's mostly what a tool is. Um, there are different types of tool calling. The one that we'll be doing is called function calling. Function calling, and they're used interchangeably. So I guess when you hear them both, they both mean the same thing, but function calling just means a type of tool calling or the tool that's being called is a custom function on your system.

[00:02:06]
It's a function that you wrote. There are other types of tool calling in which the execution happens on another system, like an API or on the provider's side or somewhere else. But function calling just means you wrote the functions running on your system, and that's what's happening. How does it work? Well, you define the tool. It's a combination of like, you know, a description of what this thing is supposed to do and when the LLM should use it and maybe what it returns.

[00:02:35]
You can give it, uh, you can describe the inputs of that function and as a schema, and the agent, depending on the model that you're using and the SDK that you're using, will abide to that schema when it passes in those arguments. So we get guaranteed contract, here's what you're going to get to this function, right? And then the third thing, at minimum usually is the execute function. So this is the actual function that gets called when this tool gets selected.

[00:03:03]
Uh, so what happens is when you give a model a set of tools with those descriptions, and then you prompt the model with some requests like we just did, the model will then see that it has access to these tools by reading those descriptions. And then based off those descriptions, they'll be like, hmm, you know what? I could probably use this tool to get closer to answering this question. So I'm gonna go ahead and use that tool.

[00:03:28]
But the model itself doesn't actually run the function, it doesn't run the code. It's not gonna take your function and put it on OpenAI server and run it. Like that wouldn't even work that way. You're not gonna serialize a function as a string and go eval on some server. So what the LLM does, it returns back an object that says, hey, here's an object. This object describes the tool that I need you to call.

[00:03:51]
You said that I have access to this tool, and you said that I have to give you these parameters if I want to call it. Well, here you go. I'm telling you, I want to run this tool and here are the parameters that I need you to do. Can you go do that for me and give me the results? And that's what we need to do, right? So we wait until the agent says, hey, I want to run this tool, can you go do it? Here are the arguments and then we'll do it, we'll execute that tool, we'll grab the results from it and we'll feed it back to the LLM so it can see it and then once the LLM sees it, it'll decide what to do next, right?

[00:04:25]
And that's the loop. It'll keep doing that until it's like, oh yeah, I have enough to answer or I gave up or some other thing, but either way, that is the whole point of just observing, thinking, observing, thinking, and then eventually reaching a conclusion. So why do they matter? Well, as you saw, this is the only way to provide up-to-date resources and context to our agents. Without this, LLMs would be only good at writing greeting cards or sci-fi stories.

[00:04:55]
They wouldn't be able to do anything recent, so that's the best one. You wouldn't have a coding agent if that coding agent didn't have access to read, write, update, delete files, or if it didn't have access to search the internet, or if it didn't have access to your terminal to do commands, there would be no coding agents, right? Those are tools that that agent has, right?