Context Window & Rules

[00:00:00]
>> Steve Kinney: All right, the other main thing, it will be the common refrain that we talk about, this idea of a context window. If you're not familiar with what that is, it's basically like when you talk to a ChatGPT or a Gemini or a Claude or what have you, there is a limited window and that window grows exponentially all the time, right?

[00:00:22]
Like Gemini's is like 2 million tokens and a token is roughly four characters. So, a lot. But that wasn't always the case. It's not always the case for every model, right? And the bigger and better the model is, the slower and more expensive it is. But even then, having too much stuff in context makes it hard.

[00:00:43]
You lose the thread. Just imagine being a human and having too many things in your head at the same time. It doesn't work out well. And arguably this is going to be the biggest theme of our time together is this idea of managing the context window and what you have in the context window and how much stuff.

[00:00:59]
And making sure you have the right stuff in the context window is effectively the answer right now. The tricky part is the nuance of how do you do that effectively? The answer of how do you get better results? And how do you not end up as like, hey, I just sent off to do a thing and it did something wild to like, hey, I said to do a thing.

[00:01:18]
I roughly got what I was expecting. Sometimes that's breaking the problem down to smaller pieces. Sometimes that's making sure that it knows about the right things and it's not reading old deprecated legacy code to then pattern match your old deprecated legacy patterns. Totally. The other one is my old boss used to have a sticker on his iPad that he only used to log into Zoom meetings, which was weeks of coding can save you hours of planning.

[00:01:46]
This was 2017. But it's true, it turns out clearly writing down your plan explicitly about what you want. If you say do auth, it will do auth. If you say, hey, use Auth0 as JavaScript SDK to implement a react hook that checks for user authentication and hooks into Google OAuth, you're going to get something closer to what you wanted, if that's what you wanted.

[00:02:22]
Actually, a lot of times having plans, even stuff like architectural decision records and stuff like that, turns out to be wildly effective. Like we said before, name of the game is managing the context window every line of your chat. These are not stateful machines as much as they feel like they are.

[00:02:42]
When you use any of these APIs you're not just sending the most recent message. You are sending the entire chat history as if it had never seen any of it before. There's no sessions, there's no cookies per se, other than for the auth, you're sending it everything. So some of that is just clearing out the chat, starting a new chat.

[00:03:02]
Some of it is compacting it down. Some of it is to make sure it has only what it wants. Some of it is instead of saying, build me an enterprise SaaS app, starting with I need a drag and drop file upload widgetin the UI. Okay, now when someone drags it and they hit submit, let's go put it in S3 bucket.

[00:03:27]
Keeping these tasks again, a lot of this applies to humans too. And breaking down those small pieces, you are going to get better success than if you say, go off and do this wild thing. And I will argue there are two phases. In some cases you call this plan implement and verify.

[00:03:49]
So it gets a third phase. But I think the verify is implicit, which is there's a difference between brainstorming and executing and especially working alone. A lot of times even having the brainstorming part of. Let me think through this. The idea of rubber ducking where you talk to rubber duck and all of a sudden you solve your problems.

[00:04:06]
This one will talk back to you. Or even stuff like, hey, I wrote this code, what am I missing? Some of these tools that work with the PRs, like GitHub has Copilot now in a PR, you can have it review your PR. Sometimes it's like, yeah, I did absolutely miss that.

[00:04:23]
Because as the manager of a team, if I open a PR for my team, I always get it looks good to me because nobody will ever. They just assume I know what I'm doing. I'm like, I'm doing this during a meeting. So I think there's a time for brainstorming and there's a time for executing.

[00:04:38]
Those are not at the same time, right? It is okay to sit there and kind of come up with the plan. We were talking before. More than one of us have the strategy of come up with a plan with ChatGPT03 and then bring it over to Google Gemini.

[00:04:52]
Be, poke holes in this plan and then bring that over to Claude with the plan and the poked holes in the plan and come up with a third, right? There's a time and a place for that too, which I think is valuable. You should always be the arbiter of how that ends up.

[00:05:05]
If you just take the plan and you don't read it, I don't know why you did that then. But there is a place for the brainstorming. There's a place to be executing, trying to figure out what you want to build while you're building it is not going to go well.

[00:05:20]
And then again, it is on you to keep your house in order, either starting your day or end your day auditing everything. Because a lot of these tools, Claude code is really guilty of this, which is it'll start just making a v2js file and leaving the old one there and a simplified TS file.

[00:05:43]
When you said this is wild, what are you doing? It will just leave those around and all these tools again, they're not magic and they're not human. Cursor is using a vector database to index your code base and it kind of knows what the legacy code and the real code is, but only based on heuristics, which it could make mistakes on.

[00:06:09]
So trimming the dead code, keeping up with things, looking at it, trimming your own rules. I had an issue where I thought it wasn't listening to me. What I realized is that my Cloude MD said one thing and my Cursor rules said a different thing cuz I updated one of them, right?

[00:06:26]
A lot of these are issues that we think these tools are bad or we think they're magical when they're not. They're tools. And if, I mean, we've all had bad days, but if regular TypeScript or JavaScript wasn't doing what it was supposed to do, we kind of know it's kind of our fault, right?

[00:06:45]
With these tools, we assume it's their fault, which is sometimes true, but a lot of times it's still only one of you is going to change.