Vertical vs Horizontal Scaling

[00:00:00]
>> Jem Young: So we got to scale up. Our original functional requirements were something like, I think it was like 1000 users and 10 concurrent, something like that. Pretty low, low scale. Our very simple client-server database will do that just fine. 1000 is nothing. Our phones could probably do 1000 if you want to make our phone a server, which someone did ask earlier about our phones and server. So full circle.

[00:00:25]
How do we scale our system? What does scale look like? Is it scale number of users, requests? We're looking at latency. The two types of scaling we're going to talk about, there's vertical scaling and horizontal scaling, and we'll go back to our pizza analogy. So we have vertical scaling. Vertical scaling in our pizza example is the equivalent of buying a bigger oven. Hmm, that works, right? We want to make more pizzas, we can buy a bigger oven.

[00:00:54]
You know, what's the problem with that? I need more space. Yeah, you need more space. And there's a finite physical limit on our systems. You can only buy, in this case, a pizza oven that's so big. Technology just isn't there yet. Even the space in our pizza oven only fits so big, you only fit so much in a box. So, vertical scaling is this idea of adding more power. But we generally don't do that if we're talking about distributed systems.

[00:01:30]
But by adding more power, more memory, more capacity, you know, we're adding more memory, adding CPUs and GPUs, adding more hard disk space. All these are making the server more powerful. And vertical scaling is easy. You don't change any of your code. If I say, oh man, the system's slow, what's the first thing you usually do? Measure it. Yeah, I think, think higher level, much easier than that. What do you as a developer do?

[00:02:04]
The system's running at peak load. What's the easiest thing to do here? Kill it. Yeah, I guess you can turn off the server. The thing most people are going to do is just make the server more powerful, add more CPUs to it, add more GPUs to it. It's the easiest thing, especially with cloud computing. Hey, let me just switch my instance size. I've done that plenty of times. There's a problem with doing that because you didn't identify the core problem, but it tends to minimize things and it's very, very simple.

[00:02:37]
I don't change any code to do that. Easy. However, there are limits to doing that. So you don't necessarily want to throw away vertical scaling because it makes sense, but horizontal scaling is about real scaling. So instead of going bigger or up, is what vertical scaling, we go sideways or horizontally, we add more pizza ovens. Easy, right? What's the theme of the day? It depends. Trade-offs, trade-offs, depends.

[00:03:07]
Yeah, there is a trade-off here, which we don't talk about when you or people often don't talk about it in system design because they're already thinking about scale, and when you think about scale, it's almost always horizontal scale, but we want to have a critical eye here. We forgot about when we're actually running our service and our system. Someone's got to operate that. And in the case of our pizza, building a pizza store, pizza shop, pretty easy.

[00:03:31]
You got one chef, one big oven. If you had another oven, an even bigger one, it'd still be just one chef. Easy. But now, you have a bunch of chefs you have to manage, which means you need to hire something or have something to orchestrate all these people. This means complexity. So horizontal scaling, again, is not a free lunch either. You're adding more complexity to what you're doing. So the default is always, oh, I'll just horizontally scale.

[00:04:04]
Yeah. Consider if that's the thing you need to do, because now it's going to get harder. When you get really, really hard, you get into things like Zookeeper or distributed messaging on how to manage all these, and there's entire software systems that that's all they do is manage complexity of distributed systems. So vertical scaling, more power, better components, a bigger server. Horizontal scaling, more machines.

[00:04:35]
I talked about pros and cons. Again, it gets a bad rap. It's adding a bigger server, easier, easy maintenance, but you're going to approach the physical limits of hardware. GPUs and CPUs can only get so fast. And when you only have one big server or a few big servers, what happens? Like what's the danger there? The request throughput gets bottlenecked elsewhere. Could be, yeah, that's true. Yeah, the resiliency is down, because instead of one, like distributing your problems against a bunch of servers or if one goes down, you can handle it.

[00:05:08]
You now have maybe 5 or 6 servers, and if one of them goes down, that's a portion of your network, a large portion. And it's also expensive to scale up and scale down, because instances aren't free and the bigger the instance, the more it's going to cost you on AWS or Azure or Google Cloud or something like that.