Research

[00:00:00]
>> Erik Reinert: After you set your requirements, I do recommend some exploration. Give yourself, like, that's the fun part of the job, right? Giving yourself an opportunity to play and explore, researching of testing of new technologies. But can anyone guess why we do that after setting the requirements [LAUGH] right?

[00:00:23]
>> Speaker 2: We can find the right design and tools?
>> Erik Reinert: Exactly, yeah, and I get this question all the time, which is how do you pick the right tools for the job? Well, I define the job first. [LAUGH] You know what I mean? I sit down and I say, well, here's my requirements, and then it becomes really easy to throw out things or keep things, that scope becomes a lot smaller.

[00:00:42]
So, set your requirements first and then explore, right? You'll avoid falling down rabbit holes, you'll avoid wasting time, but it still gives you an opportunity to try new things if you want. It's just a more structured way of doing it.
>> Erik Reinert: Also new methodologies and tools, right? So we're talking about technologies and stuff like that, but we also wanna talk about methodologies.

[00:01:06]
Maybe you want to readdress how inter service communication works, right? Maybe you want to readdress how a specific thing is maintained or managed, right? So it's not just about exploring new technologies, but it's also about exploring new methodologies, new approaches to things that could improve other systems in the future.

[00:01:29]
A good example of that is going back to the Kubernetes example that I'm doing at my job, we have to use ECS to deploy all of our services at the moment. And ECS is great, however, it doesn't really provide us with a lot of deployment options. You can get like basic blue-green, but canary deployments are a little bit harder to do.

[00:01:56]
Say you wanna do user testing, where like 50% of your audience goes to one service versus 50% goes to another. This was actually a massive argument that we were able to win because we know that on Kubernetes and using the right system, we can do that very easily, like, very easily.

[00:02:15]
And that's not as easy with our current platform. So, exploring the tools that you're using and asking, is it worth going back to the question we had before, when did you know to change everything or a specific thing, this can help. As you start seeing more things change, you might start saying, well,g, there's a lot we need to change.

[00:02:37]
>> Erik Reinert: This one, however, is probably the most important, especially in exploration, which is determine if the goal is to improve the system design entirely or innovate. Or make small changes when needed iteration. You might find yourself in a scenario where you start exploring, you start looking at the system and then you go.

[00:03:00]
We could also just make a very small change here and probably work off of that for the next six months and then readdress this six months later. That's fine, honestly, that's fine. You might be like, why say that? But the reality of it is, is your job is just to serve the company the best you can.

[00:03:20]
And innovation sometimes does impact the ability to keep moving and keep growing. And I mean, more often than not, I'm sure we've all dealt with this where we wanted to change a lot, but then realized that it was gonna be a ton of work, a ton of time.

[00:03:34]
And it could have potentially add six months to a scope of a project that probably do in, like, a month, depending on whatever you're trying to achieve. So, be honest with yourself. During this exploration time, after setting your requirements, ask yourself, do we really need to innovate here, or do we just need to make some small changes in certain places?

[00:03:57]
After you've done your requirements, after you've done your exploration, this is where technical documents come in, right? So, start writing documents down and start getting comfortable with writing documentation and understanding how to put your thoughts, your ideas and everything you're trying to do on paper, right? Whether it be, I've written technical documentation around stuff we've explored and never used, right?

[00:04:24]
But a big value for that is, is the next time that somebody else comes to the table and asks, well, what about this? We could say, well, here's the doc on last time we explored this. And a good example to that, is right now, we are working at making a centralized observability system for testing it at Hippo.

[00:04:43]
Basically, we have 40 integration test repositories and they all use JUnit behind the hood to produce report artifacts and stuff like that. But we don't have a way to satisfy what our engineering managers now want which is understanding how often are those failing, right? How often are they flaky?

[00:05:06]
What are flaky? Where are they failing? All these kinda things, because the errors are literally in console logs. [LAUGH] That's the only way you can get to the errors. There's no report that you could just easily look at. And so, we discussed it, and I was like, well, we need a portal of some sort.

[00:05:23]
We need some kind of UI or something, and they're like, yeah, we looked into it, but the cost was too much. And I was like, I didn't though. So I'm gonna try looking into it. And sure enough, I went back, I looked at a few of the open source and even proprietary examples.

[00:05:39]
And one of the examples that was proprietary had moved to an open source model. And we didn't check it, we were just like, well, the article's right there, whatever. But because it was there and I was able to look at it and read what they said and then go back and say, okay, now I can go and say, well, hey, this article is actually out of date.

[00:05:59]
There's the potential of us being able to use this software. And here's how we can, like, update the article, here's how we can update and try a new POC. And that's what we're doing, we're trying a new POC of a product to hopefully help solve that problem. So technical documents can help.

[00:06:15]
They can understand where you've been as well a iterating over them and helping to where you want to go in the future. And as I just mentioned, provide detailed descriptions of those explored systems, right? Architectures, models, interfaces, data, anything relevant to the project is super helpful with making your case helping understand how you're trying to implement things.

[00:06:39]
But you really want to, you really wanna, like, you are an explorer in a new world, right? This is your journal. This is your way that you're going to figure out how to showcase and share where you've already been, as well as where you wanna go.
>> Erik Reinert: And then a big part of reason why technical documents are important is that you can then take all of that stuff and not have to say, over and over and over and over and over and over and over and over again in meetings, right?

[00:07:11]
Instead of having to have long Slack conversations about the same thing five or seven times, you just give them a link. Say, here's the link on this architecture and why we decided to do it this way. If you have any comments, add it to the article, right? Don't put this in a Slack discussion or in a teams or whatever you use, but keep the conversation together and let everyone be a part of that conversation, whether it be development teams or stakeholders.

[00:07:38]
And again, this will facilitate communication and collaboration, you do not wanna work on an architecture designed by yourself. You might start it, you might create the initial foundation for it. But you do wanna facilitate that communication and collaboration because when you go to present it, you'll have more people behind you, right?

[00:07:58]
You'll have more backing and it'll become easier. Going back to the EKS versus ECS conversation. When we tried presenting EKS for the first time, it was just the dev team that had the backing, that was the DevOps team that had the backing. But we learned from that and we decided, okay, well, let's figure out a way of piloting a certain part of the company to try this product, to try this approach, and then six months later, we had great results.

[00:08:25]
And this team literally went in front of the entire engineering staff and said, we've had zero problems with this infrastructure for the past six months. That is irrefutable, you can't say that it doesn't work because at that point, we've already tested it with people. We've already facilitated a lot of that communication and collaboration to where we solved a lot of those problems because we worked very closely with those teams.

[00:08:52]
>> Erik Reinert: Okay, so again, that's research. Really, that's all about research, right? When you're ready, you can start implementing the systems design of your choice. And these are really steps to that, right? First off, choosing the appropriate design, right? Defining the architecture, developing the architecture, testing it, deploying it, and maintaining it.

[00:09:19]
All of these are very, very crucial, but also very, very different steps of the process that you really wanna go through. I said before, you don't wanna create something and then just put it in production and hope to God that that works. You can, but that might not help work out well for you.

[00:09:37]
[LAUGH] So trying to do as much as you can before you put it into the customer's hands can be really important. So, again, some of this is a little repetitive, but it really should help you understand now why we talked about these things. You wanna choose the design that best fits the project requirements that you set, the team expertise that you have, the understanding of the budget.

[00:10:01]
The time constraints outlined in the technical documents mentioned, everything that you have built to understand why you're building the thing that you are should at this point help you choose the appropriate design. If you get to this point and you're okay, I still don't know what to choose, then you need to go back.

[00:10:19]
You need to go back to research, you need to go back to exploration and you need to go back to working on those technical design documents because realistically, more times than not, it should just be a no-brainer. At the point of where you finally have collected everything, you should have at least enough of the puzzle together to where you could say, well, here's the project requirements.

[00:10:40]
Here's the team expertise, here's the budget, here's the time constraints. This is the best solution, yeah.
>> Speaker 2: How much time I realize every project is different, but sort of like, proportionally speaking. Do you feel like, you spend on sort of the research requirements defining phase?
>> Erik Reinert: Yeah.
>> Speaker 2: Versus when you're in, I don't know, I mean, there was six things on that list.

[00:11:00]
>> Erik Reinert: Yeah.
>> Speaker 2: [LAUGH] Implementation so if we just take all of that, like,
>> Erik Reinert: It can be anything from a week to a month.
>> Speaker 2: Okay.
>> Erik Reinert: Yeah, I think, I honestly don't think you wanna go too far beyond a month, at least without knowing something. Otherwise, that's a lot of time wasted, [LAUGH] like, researching, you do wanna find a good balance of where.

[00:11:20]
It's gonna get to a point as well where your team lead, or whoever your managers is gonna kinda be like, hey, what's going on? [LAUGH] Like, what's happening? I don't know what you're doing, right? And you don't wanna be, you don't wanna be concerning people about what you're trying to do.

[00:11:34]
Or, that means that you've kind of been black boxing yourself, and not a lot of people understand that might mean that you need to have better communication. You're not talking to the right teams. And I dealt with this a lot, actually, when I first started getting into DevOps and platform engineering and stuff like that.

[00:11:52]
I would work solely with myself, and then like, two to three weeks later, nobody knew what I was working on. But it wasn't because I wasn't working, it was because I wasn't talking about it. I wasn't sharing it with the other teams. Nobody was seeing me ask questions, nobody was seeing me do what I was, especially supposed to be doing.

[00:12:13]
And so, that made it harder to understand what the progress was. More people, more opinions to make it easier to get to your point quicker basically.
>> Erik Reinert: So, we talked about choosing the appropriate design, now we wanna define the actual architecture itself, right? So, at this point you should have a good understanding if you wanna have high modularity, right?

[00:12:38]
You should be able to have a good understanding of how things should work. And so this is really where you start putting pen to paper and what I would kind of consider is like the jigsaw stage. Where you're gonna take each one of these components, you're just gonna throw them down on a piece of paper and you start moving things around until they kind of start meshing together.

[00:13:00]
But they all need to be their own individual ideas or components. And then, this in my mind is where workflows and charts and designs and diagrams and things like that can become massively valuable, right? You throw all of your components onto a piece of paper, or into a diagram.

[00:13:23]
And then you start moving things around and figuring out does this interface work with this? Does this work well with this, is this architecture fit well for what we're trying to do. This can solve a lot when it comes to trying to actually figure out what you're trying to define.