Caching Tradeoffs & Invalidation

[00:00:00]
>> Jem Young: So a cache is always a secondary representation of your data, and that's the thing you have to remember. It's not the primary, it's always the secondary. So you have this inherent tension of the performance—a cache can be faster versus the freshness of the data. How do we know this is the most up-to-date data? And some of the different architectures and the strategies we talked about can help with this to guarantee the data's fresh.

[00:00:29]
But inherently, it's like looking in a mirror is time traveling in a way, you know, the time it takes the light to bounce back to you, it's very, very fast, but it's also in the past, it's always in the past, which is a weird concept when you think about it. You've never seen yourself in the present. No one's on it long, like, wow, Jem, it blew my mind. That's wild, yeah, time traveling. You know, there's, it's always a tradeoff.

[00:00:59]
The longer you keep data in the cache, the better the performance is going to be. But also you're running a higher and higher risk that that data is stale. So, what's that called when we have to trade? Yes, tradeoff. There's a special name for it though, when it comes to caching. Saying like, hey, this data isn't the right data anymore. Stale. Yeah, there's a very special name for it, evictions or...

[00:01:33]
Yeah, evictions. I don't know, I don't know why I'm testing you all so much. It's not important. Invalidation, cache invalidation. You've probably heard this a million times. Yeah, how do we invalidate our cache? How do we say, hey, this is out of date? One of the easiest ones is going to be TTL. Everything has TTL. TTL is time to live. It just says, hey, no matter what happens after one minute, this cache is no longer valid.

[00:02:05]
Just erase all this, or any time frame you want, a second, an hour, doesn't matter. TTL, it's pretty straightforward mechanism. However, you do have to understand your data. You have to say, hey, is this data going to be stale after an hour? I don't know, depends on your application. You can have event-based, some other services watching, and it's looking at the freshness of the data, said, hey, we just had 10 database updates.

[00:02:36]
Let's go ahead and validate the cache. And you have a service that sends an event and says, hey, cache is no longer valid. Let's seal it back up. Yeah. Could you use that for like, for instance, if you cache like what roles a user has, or you know, authorization user has in a query to check their authorization, obviously, and then the user or employee gets fired, you obviously need to validate that cache ASAP.

[00:03:08]
Can you have that be part of an event-based, I would guess, yeah, strategy? Yeah, we'll touch on that in security, but like that's a really interesting point. How do you invalidate someone's credentials? You're, a lot of times you're caching credentials because like why do that lookup all the time? If you say this is good for an hour, we assume it's still good for an hour because it's not very often we have to invalidate that and say like we log people out.

[00:03:29]
But if you don't have a strategy for invalidating that cache, you can get into a really weird state where people are still logged in even though they shouldn't be. Yeah. Someone in chat is confused what invalidation is. Oh, it just says, hey, all this data here, not useful anymore, it's not accurate. That's it. Yeah Mark, you give me a look. Well, I just, yeah, bypass the cache and yeah. So invalidation, I'm no longer useful, I'm no longer valid.

[00:04:06]
Do not use this data. We can't guarantee it's, we can't guarantee it anymore. That's it. You can version your cache and say, hey, this version. I won't say version one because obviously it can be more complex than that, but if you have long-lived data that, hey, it's only changing every hour or so. I'll say, I don't know, I'm going to use like a news homepage example. That's not changing that often, but it's changing often enough that you could probably version that because it's only going to change like another hour or so.

[00:04:33]
That's a pretty good cache and validation strategy. And pretty easy too, because you can actually have this versioning across different layers of your cache or different types of caches. And then you have refresh ahead, which is an interesting one, and we did look at that example yesterday on the mobile app, where the mobile app is just reading from the cache and the whole system is designed just to update that cache periodically.

[00:05:01]
So that's refresh ahead, keeping that data up to date automatically. Different cache and validation strategy. And then we have the other one, Kaylee mentioned this, cache eviction. So caches are finite. Whether you're writing to memory or disk, doesn't matter. There's a finite amount of space on any computer. So how do you decide what data to evict? So cache invalidation is saying like, hey, different sort of similar concept, but it's saying like, hey, this data is no longer fresh, cache eviction is saying just this particular data, we can't fit anymore, but we still have to keep writing to it and updating the cache.

[00:05:51]
How do we decide what strategy we want to use? We won't go too deep into this because this would be a whole thing. It depends on the data and it depends on the use case. So you have first in, first out, that's going to be ordered by kind of freshness, or the other way around. Last in first out. Yeah, anyways, I don't want to mix myself up. LRU is a very popular cache, also a very common interview question.

[00:06:25]
I'll give you a hint. It's a doubly linked... Doubly linked list. That's an LRU. I failed that one. I failed that one too. If it helps, I also failed a Netflix interview for LRU. I, you know, it's a couple of years of my career at Netflix, I was a UI engineer, I said, oh, I'd love to work on TV. TV is like the cool platform because that's where most streaming happens. So I was a performance engineer and I love performance engineering, or I used to, but I decided it's really hard.

[00:07:00]
So I interviewed and they're like, cool, implement an LRU cache. And I was like, well, crap. So if it helps, I worked at Netflix and still failed the interview, so it's okay. But LRU is very, pretty common, pretty straightforward to think about, what happened most recently? The most recent thing and it's just a stack. Doubly linked list, that's the secret. So different caching strategies. It depends on the size of your cache.

[00:00:00]
If it's very, very large, you're probably implementing a cache eviction strategy. If it's a pretty fast moving, not a large cache, more like it's all in memory, you're probably just doing cache invalidation because it's just as fast.