Requirements

[00:00:00]
>> Erik Reinert: We've got one last definition section that we're gonna go through, and then we're gonna jump into a lot of the examples and things like that. And this last section that we're gonna be going over is really how you implement backend architecture designs, right? So we talked about what they are, right?

[00:00:17]
We talked about when you should consider making these changes. Right now, we can actually talk about how to implement them. So, good practices, good processes on how to think about, well, how am I gonna do this? And I really like this GIF, because I feel like we've all been there, of like [SOUND] I don't know.

[00:00:37]
[LAUGH] So hopefully this section will help you guys with that confidence. So again, we're gonna talk about another quote from AI here, the prolific AI wisdom, which said, mastering something isn't a matter of precise science. It's a craft honed through the persistence of trial of experience, learning and adapting as we navigate the ever-unfolding journey.

[00:01:05]
[LAUGH] I love how prolific AI is. So, one of the things I did like about this was, it's absolutely right in the sense that these designs are normally built based off of past experiences, past knowledge. You normally won't get a junior engineer working on high level architecture designs.

[00:01:29]
And that's not out of rudeness but it's because they've probably only experienced a couple of different systems. They've probably only had the opportunity to experience a couple of different ways of doing things in that whole process. But after a company or a person has spent a long time building these things, understanding how they work, understanding their fail points, and things like that, that's really where that value of expertise comes in in improving your decision-making.

[00:01:58]
And again, it's not precision. You could work at one company spending your whole time there, building systems one way, and then go to a different company and even be frustrated, cuz it was like, woah, I'm so used to this. And it's like, well, you weren't working in this field before.

[00:02:14]
What are you gonna do? The only thing to do is to learn, adapt, and grow. And so that's a lot of what I would like you guys to at least keep in mind when we talk about how you implement these designs. So, how we implement these designs in my opinion really comes down to three stages, research, implementation, and then the last one, iteration.

[00:02:40]
And something about these design stages are they are recursive, meaning you will research something, you'll spend a lot of time figuring things out and how to do it. You'll then implement that design, right? Get it into production, get it into people's hands, and then you'll probably iterate, right?

[00:02:58]
You'll probably make more changes over it, and then figure out where you can improve, where there might be other changes that need to be made. But then the process starts all over again, right? With iteration becomes potentially more research, more implementation, and then again, more iteration. So this is a cyclical process that never ends.

[00:03:21]
If you build an architecture at a system and then leave it there and never touch it again, it's for either one of two reasons. One, the system works so well [LAUGH] that nobody needs to touch it, which is rare but can happen, or two, it's being neglected, right?

[00:03:45]
It's something that was just put there, done, and then move on, go to the next thing, right? But massive companies that deal with these crazy architectures almost are always touching the same systems over and over and over again, iterating on them, improving them, right? They understand, as an organization, that it's ever-growing, just because we built something one way today, we might change it.

[00:04:10]
And again, even companies, we'll be talking a little later about Uber, we'll be talking about Google, Netflix, stuff like that. And you can really see and even documentation or things that they've shared, that it's an always growing process to the growth of the company. First off and foremost, we're gonna set requirements, right?

[00:04:30]
We're gonna identify the primary purpose and the scope of the system. Why are we building the thing that we're building and the purpose of the system? If you have multiple systems that you have to make, break it down, right? So what is this system's purpose? What is the reason for why we're doing this specific thing and scope it?

[00:04:49]
You don't really wanna deal with what's called scope creep, or basically having your scope continually grow bigger and bigger and bigger and bigger. That might mean that you're actually not focusing properly and you need to separate those into two different scopes. So having a good understanding of the primary purpose, keeping that very simple so that you understand it, and then defining really good scope around the research that you're gonna do, right?

[00:05:17]
>> Speaker 2: Do you have any advice for like, say you're on a project, and in the middle of implementation phase, it's clear that scope is creeping a lot or maybe not enough time was spent on the design [LAUGH] phase. If you're not necessarily the person in charge of the project, how do you encourage other people to maybe walk things back or press pause so that you can-

[00:05:41]
>> Erik Reinert: Right.
>> Speaker 2: Get back to a scoped thing that's well-designed?
>> Erik Reinert: So it's my seriously firm belief that everyone has a vocal voice in the engineering process. Even if it's a junior engineer who sees something and goes, we're kinda taking a left-hand turn here, man. I don't know if we should do this.

[00:06:00]
It's gotta be up to the culture to accept that. Your are getting paid to do your job, right? And so I think that you should be in a team that gives that opportunity, for starters. If it doesn't, then that's a problem you need to fix, right? You need to be able to go to your manager and say, hey, listen, I'm trying to say something about this, but nobody's listening, and this sucks, right?

[00:06:26]
That's your manager's job. It's the same way of having an employee who's really not doing anything but you don't know how to address that, right? That's part of your manager's job or your PM's job. So I think the first thing to do is address if it's ignored, right, and then go to your manager and say like hey, this is a serious problem.

[00:06:52]
But also having a really good argument as well, right? When you're moving really fast, it's hard to be like, okay, let's change everything when you're halfway in, right? But if you can, again, sit down and write a 30-minute document on why this is happening, and kind of have a really good argument to convey, then that should also be acceptable.

[00:07:19]
But you definitely don't wanna be the person who just walks into the discussion and is like, everything's on fire. [LAUGH] That doesn't really help anyone. So, being accurate with the problem and then making sure you have a good culture around vocal concerns and stuff like that. Yeah, it's not easy, especially if a lot of money's been put into it, a lot of time's already been put into it.

[00:07:48]
I've been in that scenario where halfway through, I've been like, guys, I don't think this is gonna work. [LAUGH] And then they're like, well, we're too far, and it's like, all right, well. One of the things I was taught by a mentor of mine was it's never too late, because it's going to be even worse later on, right?

[00:08:07]
And you may have accidentally just allocated too much resources in the wrong place. And so, yeah, again, it's a bit of that salesman thing again, you really gotta be able to sell your argument.
>> Erik Reinert: Okay, so yeah, again, back to research and setting requirements, right? We've talked about this a bit already, but now you can kinda see how it's coming together, which is, we first talked about the primary purpose and the scope.

[00:08:38]
We now need to determine the expectations of the end users of the system, right? And normally, the end users of the system are one of the two. It's either developers using the system for development purposes, or, which is more common, the users themselves and how they're going to use the system.

[00:08:56]
And you might think that, well, that's just the forward-facing API that they're interacting with. No, it's anything that that API talks to or anything that that API interacts with. And this is where, again, huge companies that have incredibly large, distributed systems have to factor in everything downstream. I don't know if you guys have seen it, but there's a funny video online of microservices, and this guy being like, we can't do it because of this, and it's gotta go to this, and then over here.

[00:09:28]
And that is absolutely accurate in a lot of ways, and even gave me PTSD watching it. Because, yeah, it's not just one component, and the more you drill down into that, the more you realize that the user's touching a lot more things than you actually expect it to.

[00:09:45]
So, if you have a huge architecture for one specific thing, you might need to understand if that need and expectation is there, right, versus just having a massively over complex system. And then finally, or one of the final ones, is determine the functionality, right? So we understand, who's gonna be using this, right?

[00:10:10]
What is the functionality of that system and what is its purpose? Again, going back to the whole scope creep. This is actually a really good way to have a valid argument where you can say, well, listen, this is just for the user's functionality to do this. Why are we including this, right?

[00:10:27]
That's a really easy way of being able to communicate that scope creep may be happening, that you might be in a situation where it's going too far or getting too complex, and that can reel back as well, cuz anyone can understand that. Anyone can understand this is [LAUGH] too much for what is needed.

[00:10:47]
Performance as well, you do wanna think about performance, right, not so much in the sense of, can it take a billion requests or something like that? But is it performant, right? What happens when we throw a little bit of load on it, going back to the UUIDs don't scale example.

[00:11:06]
There is a valid argument in saying that if one specific function has to make seven lookups to a user, maybe you have your user model spread out too much, right? Maybe your data is not all in one place as efficiently as it could be. And so because of that, it's going to make it so that if we do ever have a thousand requests at one time, we're gonna DDoS the database or something like that.

[00:11:31]
So understanding the performance that is actually needed for the system and not, again, having that over complication will make it so that it should be. It might not scale perfectly, but it's the KISS principle, right? Keep it super simple. If you keep it super simple, then you'll probably be able to scale that system better as well.

[00:11:55]
And this is also really important as well. Any constraints that you can think of whatsoever, any constraints. It can be budget, it can be resources, it can be time, challenge yourself before letting others challenge you. Because you'll be more impressive in a discussion if you've already thought about what a concern of somebody else might bring up.

[00:12:17]
And that's part of the salesmen part of building an architecture, right? You know what, I've already thought about that and this is what I found, what do you think, right? That makes the person listen to you and wanna buy in. The conversation becomes less about I'm right, and you're wrong, and more about let's work on this together, here's what I found first, right?

[00:12:38]
So, seriously guys, challenge your own designs, right? It's really important to look at something and be like, how would I push back on this if I was to, right? It can help a lot, and those constraints can help narrow scope, they can help lower budget costs, they can help with a lot of stuff.

[00:13:01]
>> Erik Reinert: We talked about this a bit, but now we're talking more about the actual scalability of the system itself, right? How can it grow and adapt? For example, going back to the user service scenario, if you're heavily, heavily dependent on user data, and you know tons of requests are gonna go to that service, consider separating that out.

[00:13:25]
Consider, is this going to be something that's going to be reused by a bunch of other services. A good example of that is in the V2 version of Quirk, what I'm building, I wanted a plug and play bot system. And what I mean by that is, is we knew that in the future we were gonna support Twitch, we were gonna support YouTube, we were gonna support Discord, other platforms, but I didn't wanna have to reinvent the wheel every time we connected those systems.

[00:13:56]
And so what did we do? We built what we call our bot engine, which really just means that you can send any kind of text to it, and it knows how to respond with natural language, NLP basically. And so that made it so that when Discord was ready to be connected, it was just like, all right, create the listener and then send all the messages to the same place.

[00:14:19]
The way that it's processed, the way that the AI actually computes it and returns it, it's the same. It works the exact same across all platforms. And what's nice about that, Plug for Quirk, whenever it comes out, [LAUGH] hopefully Twitchcon, [LAUGH] is that you have a unified user experience as well.

[00:14:40]
Now the user knows that they can expect the same AI ML responses on Discord, on Twitch, on YouTube, whatever. And we did that not only as a experience part, but also just make it easier so that when we want to in the future, we can hook in more systems, and it's easier to process.

[00:15:04]
I mean, theoretically, if we wanted, we could hook it up to Twitter, it's just text. It just processes text at the end of the day. Again, security requirements are also really important. You should absolutely be thinking about data, protection of data, processes within systems, how they communicate with each other, right, things like that.

[00:15:26]
And to be clear, this is all part of the research, okay? This isn't taking a design and saying, does this fit their requirements? This is no, we're going to think about all of these things beforehand so that we step forward with, again, scalability concerns, security requirements, stuff like that.

[00:15:46]
And again, this kinda goes into this as well, but I have to mention it just because I work at a company like this. Regulatory or compliance requirements, right? If you have HIPAA requirements that you need to stick with, be sure that anything that is considered knows how to play process or handle those things.

[00:16:08]
Really research should be thinking about anything that you possibly can. And if you also have integrated systems, say you already have existing systems, this is part of that conversation of preparing for anyone to bring up maybe something that already exists, right? So if you're integrating a system, you wanna think about, okay, well, maybe a requirement is, is that it uses HTTP versus GRPC or maybe that we stick with the same kind of protocols.

[00:16:38]
So that, again, we're not changing something on the fly, and then we can't hook that system up to other things in the future. All of these roles play together.