Structured Output

[00:00:00]
>> Sabrina Goldfarb: Let's talk about structured output, right? I keep telling you all, well, it didn't give us the output because I didn't tell it what to give me for the output, right? So finally, we can talk about structured output. So we want to make sure we're getting consistent formats every time, right? Why do we care about structured output? Let's think about why we would want to get consistent formats every time.

[00:00:26]
Well, again, if we're talking about building an AI application, right, if I'm connecting that LLM to something else, for example, on my side project, right, when I was trying to send a bunch of data and get it categorized, I needed to make sure that those categories would send in a very specific format to the next section of my application, right? So I know what to do with them because if I could get a paragraph or a sentence or an array of items or a JSON object, then it's not useful to me, right?

[00:01:01]
I can't actually do anything with that. So if I want to make sure that I'm getting consistent formats every time, this is when I'm going to focus on getting structured output. It tells the model exactly how you want information, right? And then we use examples, templates, or schemas to enforce format. So this is where we're really starting to see that all of these techniques are going to be utilized together, right?

[00:01:28]
So we're learning a ton of different techniques that get us more accurate information or get us more on the tone of what we're looking for, or more the length of the response that we're looking for. But now we're also utilizing this to enforce the format that we're looking for. This is really good for test cases, for documentation, for data extraction, for code generation. Literally any time that you're tired of having to reformat your responses.

[00:01:56]
So maybe I'm just chatting with ChatGPT and I'm like, hey, I'm having a problem with this bug. Can you show me the JavaScript code that would help me to fix it? Instead of having to copy and paste everything, be like, just show me the JavaScript, nothing else, right? This kind of structured format allows me to simply copy and paste instead of focusing on reformatting anything. So, again, anytime we need a specific format, anytime we're going to be generating multiple similar items, this can be really useful for, it's kind of like, if you think about like factory functions, right?

[00:02:34]
It's kind of very similar in that sense of like, I'm just putting something in and I'm generating the exact same output every time or exact same format of output every time. So I can just toss in my input and get my output and copy and paste. And I know that might not seem like a lot in terms of just chatting or like using this for code generation. But if you think about saving 5 minutes every time you chat with one of these AI models, you can end up saving yourself literally like an hour every single day, and an hour every single day ends up being what, you know, like 40 hours every week and then, you know, it just kind of goes from there.

[00:03:15]
But again, the more important thing for this is when we're thinking about AI applications. When we want to make sure that within our AI applications, we are able to just send the right information, the right data down to the next part of our application, so we don't have to worry about it. Something interesting about LLMs is obviously they don't fail the same way that traditional code fails, right?

[00:03:41]
When they fail, they hallucinate. They send the wrong type of output. When they fail, they do something unexpected, but it doesn't just generate an easy error code for us. So this is kind of one of those ways that we can start to control in our application and look for those failures as well, because we can say, if I don't get this exact format, right, maybe I'm using TypeScript or maybe I'm using a typed language.

[00:04:06]
If I don't get something in this exact format, something went wrong, right? So, let's try out a couple of examples of this first in our Claude chat. So, let's try out a first example of this. Extract the meeting details from this email, right? And format like this, right? We don't have to be fancy. We can literally just say format like this. Date is going to be a date, and time is going to be a time.

[00:04:53]
Location is going to be a location, and topic is going to be a topic. And then the email itself is going to say, let's meet tomorrow at 2 p.m. In conference room B to discuss the Q4 budget. Right, and let's just see what happens from Claude. Okay, great. We've cut out a lot of fluff, right? Now we just have the date tomorrow, the time at 2 p.m., the location, conference room B, and the topic is Q4 budget.

[00:05:32]
We can see how, even though this is kind of random, right, we can use this in an application. We can format things. We can think about inputting or taking inputs from forms and using that information later and allowing the LLM to do whatever it needs to do. Maybe it's going to take in full emails and format them for us, right? Maybe in the past, it was so important that we wrote code that was like, okay, now you fill in this line and you fill in this line, but now we can just say, hey, you know, GPT-5, hey, Sonnet 4, whoever you are, like, take this lengthy email, summarize it into these very specific things, and send this down in this exact format so we can make applications better for our users and easier for our users just by doing stuff like this and, you know, allowing more creative freedom in the way people submit things to our applications.

[00:06:41]
So let's try one more of these and say, okay, convert this error into JSON. Return only the JSON, no explanation, right? So in the last one, I wasn't this specific about only returning the JSON or returning only the data, right? But we saw that it still did only return the data in the format that I requested. But if it is absolutely critical that I only return the JSON with no additional explanation, it can't hurt to add that as an addition to our prompt, right?

[00:07:15]
If we look at it and if we were to count the amount of words and divide it by 0.75, I mean like 1, 2, 3, 4, 5, 6, 7, 8, maybe I'm using what, 5 extra tokens, maybe. I mean, it's not going to be something that's significant enough for me to care about, but it's always going to get me that correct output. So, error 404 occurred. I can't spell occurred, sorry. That's not how you spell occurred. At 3:45 p.m.

[00:07:58]
When trying to access /user/profile endpoint. Okay. And let's see what Claude does. Great. So now we can see JSON error code 404, timestamp 3:45 p.m., endpoint /user/profile, message error 404 occurred at 3:45 p.m. When trying to access the /user/profile endpoint. So now we can see that we can utilize the structured output, not just with words and tables and markdown or whatever we might want to use, but we can actually use this to just create JSON, right?

[00:08:28]
And this can be really helpful for us. I imagine maybe I'm creating an AI application where I track the errors in a user's application and I try to make something from it. I try to let the AI help surmise where could that have maybe come from, right? What happened? Was it a network failure? Was it a server failure? Was it human error? What was it, right? And so maybe I'm utilizing this to try and make an AI application that helps with that, but also the first thing I want to do is make sure that I'm passing along the right JSON.