Create a Movie Search Tool

[00:00:00]
>> Scott moss: Now we wanna make the tool for our agent to take advantage of the query. Right now, the agent can't just use this queryMovies function, we have to make a tool. We know that, we made tools before. If you haven't, you're about to make your first tool. So what do we do?

[00:00:16]
We go into the tools folder. We make a New File, we're gonna call this movieSearch. Actually, I already have it, it's just blank, kidding. So a tool is just two things. We need to export a definition, and we need to export a function.
>> Scott moss: So let's do that.

[00:00:37]
So first, I'm gonna import this type () from '../../types'.
>> Scott moss: Okay, it's gonna be called ToolFn. And then, what else do we need? We need, yeah, zod and then the actual query. So I'm gonna import {z} from 'zod'. And then the queryMovies, there we go. Export const movieSearchToolDefinition, Which is gonna be an object that has a name.

[00:01:24]
>> Scott moss: I'm gonna call this movieSearch, you can call it whatever you want. I'm gonna have parameters.
>> Scott moss: And they're always gonna be an object. Pretty much you have to put an object here, you can't do anything that's not an object. And these parameters is gonna be the query, which is gonna be a string, and we'll add a describe here.

[00:01:48]
So what is this string? A 'query used for vector search on movies'. And I don't know, we might have to workshop this with an eval to see if this is good or not, but that's what we're gonna do. We will also take the filters here, so in this case I might say genre director.

[00:02:11]
We're gonna skip that for now, I think it's gonna get too crazy if you start doing that, in my opinion. And then a description of this tool, which is also a string. So what is this tool? Use this tool. So typically I'll do something like 'use this tool', or 'this tool retunes' or something like that.

[00:02:37]
So I'll just say 'use this tool to find movies or answer questions about movies and their metada like score, ratin, costs, director, actors, and more', something like that. You can do whatever you want.
>> Scott moss: I'm pretty sure I did something different in here. Yeah, searches for movies, yeah, exactly some way different.

[00:03:19]
>> Scott moss: And I'm gonna make my argument type using this cool little hack from zog, so we got that. And then last time we need to just export our actual tool. So I'll say const movieSearch, it's gonna be a ToolFn at takes in the <Args> here, = async () => {, like that.

[00:03:48]
It's always gonna have the original user message, and it's gonna have the toolArgs, which is where the query will be, so we got that. And now it's, essentially, what we wanna do, we don't have to worry about the filters in this case. Is we just want to query the movies, pass in the query and format the results in a way that we want the AI to see it.

[00:04:16]
And then stringify and give it back to the AI, that's it. So, again, it all depends on how you do this, is how the AI is gonna see and how it's gonna interpret it. So let's do this. So one thing we're also gonna do that I'm sliding in super quick, is I'm gonna put this in a try catch, because if this breaks, well, first I wanna log it, I wanna know.

[00:04:43]
>> Scott moss: I don't want my agent to die. And because this broke, I don't want the code to break. So I'm gonna catch it, but then I'm gonna tell the AI, hey, there was an error. And I'm gonna say, Could not query the db to get movies. So now the AI knows that, it tried to do it, but it broke and it might respond in a way to the user.

[00:05:07]
And then you can have something in a system problem that says, hey, if you see an error, say this to the user, and then it'll say that. Whereas if this just broke, what's the user experience like? It's just dies, they don't get anything. So this is a graceful fail, or letting the AI know that something bad happened.

[00:05:28]
>> Scott moss: So we got that, so I'm gonna say results = await,
>> Scott moss: QueryMovies (), just gonna pass in the (toolArgs.queryMovies) like that, this thing's freaking out cuz, it's an object.
>> Scott moss: There we go, we'll get back our results here,
>> Scott moss: And then with the results, yeah, I mean, you can kinda do whatever you want here, how you wanna format them.

[00:06:08]
I think the way I did it here was I just didn't want the metadata field on it, I thought that was redundant. So I got rid of the metadata field on every single result. But there's probably a better way to do this. So basically I'm just gonna return.

[00:06:26]
Actually, I'll just say-
>> Speaker 2: Put metadata over the return object and then add the other one?
>> Scott moss: Yeah, exactly. So results.map, get the movie, I guess, or let's just say result => { like this. And then I can say result, get the metadata, get the data. And then I can just return a new object with the, what did I call the data one here?

[00:07:00]
I called it a description, so I'll say description. Actually, I'll do that one last, I'll just say metadata, and then description: data.
>> Scott moss: But again, there's no wrong answer. You could just literally stringify these results and be done, that's fine too. I'm just trying to pass, anything that is not relevant, I don't wanna give the AI, cuz then it's too much noise.

[00:07:31]
So that metadata field is so irrelevant, I don't wanna even know that that's a thing, so I just got rid of it. And then you always have to return to string, so JSON.stringify(formattedResults), I'll do null 2, so it's just formatted. Okay, now we have that. What we need to do now is add this to our tools array.

[00:07:59]
So inside of our tools Index.ts file we're gonna import that tool
>> Scott moss: From the '.movieSearch', the definition, not the function. Add that to our tool array.
>> Scott moss: And then lastly, we need to go into our tool runner that's responsible for actually executing the tool, and we need to add it here.

[00:08:31]
So I'm gonna import () from './tools/movieSearch. And I wanna get the movieSearch, and then the movieSearchToolDefinition. And then down here I can just say in the switch statement, case movieSearchToolDefinition.name. Then I want to return calling movieSearch(input).
>> Scott moss: So now our AI knows about the tool. Our tool runner can call the tool.

[00:09:14]
It should work, [LAUGH] let's see what happens, as always. So first what I'm gonna do is I'm gonna clear my chat history. I could do that by going to db.json and pretty much just blowing this out. We'll get into summaries later. I'm gonna same "messages": [], is that.

[00:09:36]
And, wait, do I need to put "summary" here? I might have to put one here, let me see. I'll just do it, just in case. So "messages", "summary", empty. Yours might already be empty, might not be, and then now I can say, npm start. Let's just make sure this still works.

[00:09:54]
Okay, looks like something died, so string, but got an object. Did I put an object for a description somewhere? Let's see, movieSearch. Yeah, I don't know why I did that. It's just a string here getting caught up. Yeah, that's just a string, not an object.
>> Scott moss: Gotta love good error messages.

[00:10:24]
[LAUGH] So yeah, make sure you put a string here on the definition, not a zod string. That's only for the parameters. Okay, let's run that again. Now I think, no, I actually do think this is messed up. Yeah, let me get rid of this. Yeah, okay, try that again.

[00:10:42]
Hi, cool, it didn't break. So let's try to search for a movie. "Find me a scary movie about ghosts". MovieSearch, done. It said it did it already, that was really quick, let's see. Wow, that was really quick. So yeah, I found Annabelle dead awake. So yeah, pretty good.

[00:11:10]
We can go look at the output to see what it got back. So you can see here, this is what it got back from the vector database. It literally got back those movies,
>> Scott moss: And it just stored them in here, and then it answered. So pretty damn good, I would say.

[00:11:29]
Obviously you have to test it and make sure it's doing what you expect it to do, whatever the objective of your LLM is. Is your LLM meant to be a replacement for a search bar on some Netflix-like app. So you would wanna prioritize searching capabilities. Is your LLM more about fact answering, about answering questions, factual, answers about movies and less about finding them?

[00:11:51]
So you would wanna eval towards that. So it really just depends on what your system is. So this Arg is just a generic implementation, which is why I took away the filter, cuz that's going towards more of like searching and finding a thing and less about answering your question.

[00:12:07]
>> Speaker 2: So how would you change your approach to evaling this?
>> Scott moss: Evaling RAG is way different because you have to have different metrics for different parts. So you would have multiple scores for multiple things. So just for the retrieval part you would want to set up a metric that scores how well your retrieval logic is finding the right documents.

[00:12:37]
So you have to have a score for that. And it's kind of the one that showed in RAGus that did the precision context one. Either have a score that checked the output of the AI to make sure, if given the right context and asked a certain question, that the answer was actually factual.

[00:12:57]
So you wanna score that, right? You also wanna score whether or not it picked the movie tool, like we've been doing. Those are probably just three I can think of. There's probably ten other ones you probably would wanna-
>> Speaker 2: So you can break it down to its component parts.

[00:13:11]
>> Scott moss: Yeah, exactly. So, yep, there's a lot to do there. Let's do something cool. Let's say, "ok, now make a movie poster for", which one, Annabelle, Insidious? Let's do Insidious.
>> Scott moss: It's gonna make a movie poster. I don't need the approval works on this branch, so that's why, even though it says, do you wanna generate image, is still doing it.

[00:13:54]
We haven't gotten there yet, so let's see, still gonna do it, I think. Yep, it did it. [LAUGH] Let's see what it makes. Here's a chilling movie poster for Insidious. That's actually really good. Okay, good job Dolly. Dolly's like, yeah, I'm here, I'm back.
>> Speaker 2: Which upstash plan do you recommend if we use this in production, a fixed or pay-as-you-go?

[00:14:21]
>> Scott moss: I would just go pay-as-you-go until you hit the limits, cuz I think the pay-as-you-go also has daily limits still. So once you get past those daily limits on pay-as-you-go, then you need to do fixed.
>> Speaker 2: And in production, do you recommend using a database for evals, or is this method of storing on a JSON file?

[00:14:40]
>> Scott moss: That's really good, I don't think you would do either. I don't think you would store it in a database or putting it in a file, I think you would have some third-party platform to help you do this. Because even if you put your data in a database, you still have to build an entire pipeline, an application to do anything with that data, and that is a whole startup on its own.

[00:15:02]
So unless you work at a team with thousands of engineers twirling their thumbs and they need stuff to do, I probably wouldn't waste time trying to build some evaluation framework that could do all that. I would just use some third party-tool. Some of them are free that you can self-host and are open source.

[00:15:21]
A lot of them are paid, but I would still use something that was a complete solution.
>> Speaker 2: Regarding namespaces, do they affect the query result or just classify it?
>> Scott moss: Do they affect the query result? Yeah, so the namespace has methods on it, right, so, where do we do that?

[00:15:43]
>> Scott moss: So first is on ingest. I'm up starting on an index that's global, but I could also upsert on a namespace, and it's the same thing. So you can do a namespace for an upsert or query, it's up to you. So yeah, if you have users always create a namespace, I just use their user ID and I upsert with that namespace, and I query with that namespace.

[00:16:09]
And the alternative is that you would go here and you would go here without a namespace. You put userId and then it would be this. And then every time you query you would filter by this userId. So it's up to you how you wanna do it. I don't think there's a limit on namespaces, at least in upstash.

[00:16:27]
And then if you're on the dashboard, if you wanna see different namespaces, it would be here. So as defaults, I don't have any, but if you had some, they would be here and you could search for them.