Create a Movies Query

[00:00:00]
>> Scott Moss: Let's query this now. So, if we go to the query, we need to write a function that can query this. This could get really advanced. I'm going to try to keep it somewhat simple. The strategy here is to basically, take a query string, just like I was typing in the dashboard.

[00:00:19]
Optional filters object, so an optional object that represents these fields, and values that you wanna match one. And top K, optional. Top K just means, how many you want to get back? The top scores, right? So if I go here, if I were to say, right here, this is topK, so this is 50.

[00:00:39]
If I say, give me the top 5, I'm only gonna get back 5, right? So this is topK. And they're sorted by this score, which is the distance that this was on the plot from this. So the WALL E was the closest. Given this metadata for filter, and this vector WALL E was the closest in distance, when plotted using cosine similarity.

[00:01:09]
But again, that doesn't mean it's relevant, all right? What if I said, mad movies? It put, I mean, I don't know. Is Mad Max a mad movie? I don't know. [LAUGH] So like you can tell like it's good, but it's not always relevant. Why did Zootopia come up as mad?

[00:01:29]
Because it is [LAUGH]. It is kind of mad. Why did it say that, though let me see. I'm curious, cynical, maybe, I had to copy that and see that's interesting, it's so funny, okay. So the way this is gonna work is, one we need to build a query string, because the way that upstats vector does its upstat, its metadata filtering, as I showed before the break, it looks like this.

[00:01:58]
It's like, show me an example. It looks like this, right? You have to give it a string of filters, like a SQL, like syntax, so, like, we have to turn that object into a string to send it up. So, that's what this first part is doing. It's converting that object into a string, you would have to optimize the hell out of this for this to be good.

[00:02:23]
I can promise you it's not gonna be good. This is so hard to do because, you have to ask yourself, where are these filters coming from? Who's making these filters? So, either you have a UI, where users can select these filters, and then it's perfect every time. Or, if you're in a chat app or something, AI is determining what these filters are based on what the user asked.

[00:02:43]
And I promise you, that guy's gonna get it wrong. You'll be like, so my ass,show me any movie with Brad Pitt in it. And then what it'll do, is it'll give you a filter and it'll say, movies with only Brad Pitt. And you'll never find a movie, because there is no movie with only Brad Pitt in it.

[00:03:03]
He's in movies with other people. So, it's just like, trying to get that to be good is really tough. But, we're gonna do a very naive implementation of it. So, we'll have that, and then from there it's pretty simple. The index gives us a query method that we can pass in our query.

[00:03:18]
It gets vectorized for us on the server, our top K, our optional filter string. And then these are just saying, hey, when you return, can you return the metadata, and return the data field. So like this field that we created up here, this field, can you return both of those?

[00:03:37]
That way, the AI can see it. We wanna give that data to the AI, so it can see it, right? If we don't include these, the AI won't get back anything but the ID. So I don't know how that's helpful for the AI, right? Does that make sense?

[00:03:55]
Okay, let's do it. So the first thing is let's get our index.
>> Scott Moss: From,
>> Scott Moss: Upstairs vector.
>> Scott Moss: Okay? And, the other thing that might be helpful for people, is indexes also have namespaces. So, a namespace, if I remember index.namespace, yes. So if you do something like this, you give it a namespace.

[00:04:38]
For instance, I would use it for like some user ID here, you can separate. So instead of creating a whole new index for like a user or something like that, you can just create a new namespace on this index, and you can make it a unique thing. So this is how I like separate one user's data from another, by giving them their own namespace.

[00:04:56]
And the namespace is their their user ID that they would have. So whatever their user ID that would be be here. And then, this namespace has all the same methods like query, fetch, upsert the same thing that index has, but on this namespace. So, good to know if you're building real world applications.

[00:05:12]
So, use an index for global data that everybody can use, but, specific to a user you might wanna create a namespace and do it there. Because the alternative is like, putting a metadata field on each thing that you have, and the field owners, user ID, and then you just filter on that.

[00:05:30]
I guess that works too, you could just always filter on the user ID field as part of your filtering. So, that works too, but this is a database level partition, so,
>> Scott Moss: Cool, we got our index, I don't need any of my args there. And then we're gonna say, what did I call this?

[00:05:50]
I don't even care about the TypeScript type. So we're gonna say, query movies takes the query filter and top k. So, let's call that export const queryMovies, async.
>> Scott Moss: It's gonna take a query, which is gonna be required in a string. It's gonna take filters, which I was gonna say any.

[00:06:20]
You can add the texture type if you want, and then topk, which will just be a number,
>> Scott Moss: Like that. And I'll grab those here, so query filters, topK,
>> Scott Moss: I'm just gonna go to 5. You can do whatever you want,
>> Scott Moss: Cool. Now to generate the filters, essentially, again, this is just looping through all the filters, which is gonna be an object that looks like this, and convert them to strings.

[00:06:59]
Convert them to a filter string. So, this really nothing magical here. Honestly, I would even say maybe you can skip this one for now, because if you do this right now, and you don't get back any results, you're gonna be like, why is this not working? And it might be this.

[00:07:11]
But, maybe for now skip this. When we make the tool, we can come back and we can optimize this, because I don't want people not getting results, because this is too strong. So skip the filter string for now. And let's go straight to the query. So the query is pretty simple.

[00:07:30]
>> Scott Moss: We can just say, return, index.query. I'll do that. The query is going to be the actual query. I'm sorry, data is going to be the query. Topk is gonna be top k. And I think that's pretty much all we need.
>> Scott Moss: No, we need to include the other stuff, that's all right.

[00:07:57]
Include the data, include not the values, but include the metadata. I have that, include the data, there we go, cool.