Key Challenges: Adaptability & Security

[00:00:00]
>> Erik Reinert: So we talked about complexity. Let's talk about adaptability, right? Designing systems that adapt to changing business needs is challenging, right? It's a challenging part of building an architecture. But it is something that you should consider, right? Especially, if you have multiple systems, you don't really want to, what I would kinda say, keep piling systems on top of each other, right?

[00:00:26]
You don't wanna say, well, we're just gonna do it this way, this new way this time. We'll just do it this new way this time. Then you're gonna have seven different systems that handle seven different ways. And then all of those ways have to be troubleshooted and supported.

[00:00:39]
And it becomes really challenging, again, what we call friction at a company, right? Developers should really only be expected to handle so much at any given time so that they can focus on building the things that they are. But it's kind of like going into Costco and then coming out with a huge grocery cart.

[00:00:57]
It's gonna take a while for you to unload that. And I think adaptability here is really important to making sure that you're not overloading the developers, but at the same time you are considering, okay, we use HTTP everywhere. Maybe we don't change that, right? Or, okay, we use queues everywhere, maybe we don't need to change that, right?

[00:01:24]
And that adaptability can also help you grow and say, okay, well we made this decision now that we do this, we can handle all of these this way. And I talk about this even as well in the enterprise, cloud infrastructure course. A common thing is like tiers, right?

[00:01:40]
So you have a worker tier, then you say, okay this is how workers process data, right? And then you have a, what we call, a runner tier, which is like Crons, okay, this is how Crons work, right? Having less conversation around a specific technology and having more of a conversation around what problems these are solving, can make that adaptability easier.

[00:02:05]
And as example again, where I work, I think we've broken it down to, five different tiers. We have a frontend tier, where all of our frontends go, and all of those frontends can adapt on their own in that tier. We have a backend tier, whereas really all of our internal services, so nothing public facing.

[00:02:25]
That's the first thing to note with our backend tier, no public facing, right, whatsoever. What we decided was to call our public facing services gateways, right? They're a gateway into our system. Is that something that I learned in a tech course? No, it was just us trying to figure out a way of designing this so that developers know these are private, these are not, right?

[00:02:47]
And that adaptability has made it so we have 50, 60 back-ends, and it's really easy to add, and developers have a better focus on, okay, well, I need this, I can adapt to this. So we have frontends, backends, gateways, runners, and workers, and each one of those solves different problems that we can adapt and grow if we need to.

[00:03:11]
Another thing to note about that adaptability is that those categorizations make it so that if I need to change a specific thing, I don't have to change everything, right? If I just wanna change frontend so that they can support serverless or server side rendering or whatever. Then all I have to do is focus on that one component or that one tier rather than saying, okay, how do I think about this thing and how it works with everything else?

[00:03:39]
There's a bit of a separation there, that I have more flexibility with working with. So here's a good design of kind of this adaptability. With Quirk and what we're doing now, we're doing a lot of serverless. And the main thing I want you to take away from this design is the patterns that we have here.

[00:04:00]
Basically, we have multiple entry points for different types of the system that then work the same for each component, right? Now this is actually for our logging and how we do metrics and things like that. But because we have tons of lambdas, we wanna be able to have a similar process for each one of those.

[00:04:22]
But if we change those and adapt them, we should be able to change them all at the same time. And so in this scenario, we have lambdas going to a internal forwarder, which then has its own autoscaler, which sends out to Grafana Cloud. Again, the main takeaway here is that we decoupled these different parts so that we can change or grow different parts if we need to.

[00:04:48]
Like I said, we will have tons of lambdas, but we don't wanna have to go into each lambda and figure out how we're gonna forward that traffic, right? So what we did was, we decoupled that and made a forward itself so that then we can adapt in the future and say, okay, well, if we change the metrics we send or things like that, we only have to change those in the lambdas.

[00:05:10]
We don't also have to change where they're being sent to or how we're batch processing them or things like that. And another nice thing about this is that, this design could potentially support not just lambdas, but other things. Because we've designed it in a way where the forter can kind of be plug and played with many different components.

[00:05:31]
It doesn't matter if it's a lambda or a service or anything like that, we built it so that it can adapt and we can get that telemetry out of the box. So security is also really important to backend architecture design for a lot of reasons, maybe you have legal requirements that you need to be able to satisfy.

[00:05:49]
Maybe you also have other policies that your SecOps team wants you to follow, right? And I would say, more often than not a lot of places consider security to be one of the last things that you think about, right? Whether it be internal security, how many people actually have authorization between services at where you work?

[00:06:17]
Okay, two, okay, kind of, yeah. It's very common to say, well, it's in the Internet, or it's in internal networks. Don't worry about it, they can talk to each other, right? But if somebody gets into those internal networks, even though you might think they might not, it could still show as a really big problem, right?

[00:06:37]
And so considering the security of these services, even how they authenticate with each other, the complexity of that is really important. Even something as simple as having rotated passphrases or keys that each service can use to communicate with each other. This is something we did where I work, where we just created what are called consumer secrets.

[00:07:00]
And we loaded those in from vault, and then whenever a service started it had its own consumer secret and those secrets were shared between services, so that they knew how to communicate with each other. Not super scalable, especially when you have 100 services or something like that. But at least it gives you some kind of system that you can iterate off of and say okay, we'll rotate consumer secrets every week, every month, whatever.

[00:07:25]
And then we don't have to worry about the security of that. Another thing to note about security is maybe your service becomes public by accident, right? This is actually very common as well, where maybe it's not deployed properly or the routing is set up wrong, or rules are, permissions are firewall whatever, aren't set the way that they should be.

[00:07:45]
And so because of that, now a service is accessible SecOps is freaking out and you're kind of scared for your job [LAUGH]. Yeah, security is important man, you really need to think about it. Not having any kind of security around something can cost millions of dollars. And again, I work at a billion dollar company.

[00:08:08]
We're insurance and we have a lot of customer data that even if a little bit of it got leaked, we're talking about not even just it got leaked, but also potentially getting sued or fined, right? And it might not even mean that it actually leaked anything, but that potential and that not knowing could be detrimental to what you're doing.

[00:08:32]
So security is a pretty massive concern when talking about architecture.