Scaling Function Calling

[00:00:00]
>> Okay, I mean, I think there are obvious reasons why this is hard to scale in production. [LAUGH] You can do anything, you can literally do anything. So don't do everything, that's basically the hard part. Don't let this thing do everything. Depending on what model you use, for instance, GPT-4 is so much better at determining what function to call.

[00:00:26]
If you had ten functions in here, this gets complicated, right? Then it's just like, whoa, which one do I call, which one do I call first? Which one do I call second? What's the order? Some of these overlap. GPT-4 is a lot better at that than GPT-3. And then, yeah, trying to figure out these things, the descriptions, is kinda hard.

[00:00:48]
So I actually use GPT to write the descriptions for me. I'm like, [LAUGH] I'm trying to teach GPT about this function it can use. And here's what the function does, but I need to describe it to GPT. And then here are the parameters I need to describe. Write the descriptions for me, and it'll write really good descriptions that I can use.

[00:01:03]
So it's AI all the way down. [LAUGH] It's insane. So that's tough, trying to get this to work. But the good thing is that you always will get back structured output. You'll always get back this. This is the beautiful part. This is so good for AI, you can write tests on this now.

[00:01:22]
This is deterministic, every single time, it's great. So that's beautiful, but yeah, you have to think about issues around latency with the function calling, security, obviously, different things like that. And then you have to think about moderation around what is being passed to the AI. And there's so many edge case I'm not even thinking about, but this is incredibly difficult to do well.

[00:01:55]
You ever use ChatGPT plugins, right? If you ever been to ChatGPT and they have plugins you can add now? Okay, that's what this is. A plugin is a function that makes an API call. That's what a plugin is. That's literally it, and there's a spec for it. A lot of websites have, openai api spec, right, they have this, let's see.

[00:02:19]
Are they gonna show me here? My God, I don't know why I can't find one, this is annoying. [LAUGH] But they're everywhere. Anyway, that's the new thing now. So it was cool to like, all right, you're a SaaS app, now you gotta have an API for developers. Yeah, cool story, we all know that.

[00:02:33]
Now, it's like, now you have to have an OpenAI endpoint for the AI to hit your thing, and that's what's new now. So it's not enough just to have API for developers to hit. You gotta have an endpoint for the AI to hit, and that's what a plugin is.

[00:02:46]
A plugin just describes, it just teaches ChatGPT how to call your API. It's just a manifest for that, and then they just do function calling. Because, guess what? You can make a standard function call that's just called fetch. And its description is, make an API request, [LAUGH] that's it.

[00:03:03]
And then, it just takes in a URL to an API or anything, right, and it just does it. So that's what a plugin is, it's pretty intense. But yeah, that's function calling. It's probably the most powerful thing ever that program has ever done, in my opinion. It's terrifying, but also very exciting.

[00:03:23]
And, yeah, I'm glad I was able to walk you guys through that today.