Continuous Delivery & Continuous Integration

[00:00:00]
>> And yeah, even going back to the service management and that question, right? Okay, so something happens, we configured something wrong. Whose responsibility is that? Who would you say? Somebody deployed a bad config. Is that my job, DevOps? Is that the developer's job, yep?
>> So it's the configuration for the software, the product.

[00:00:29]
>> The environment, yeah. Like my prod secrets, right, values, that broke, whose responsibility is that? Who do you think?
>> It's on infrastructure.
>> Really?
>> I think so, yeah. Because I don't know. I mean, the way that I see it is that you're trying to set up processes that are actually safely, to use the word, container, actually, cordoning off.

[00:00:53]
>> Yep.
>> So that sort of thing doesn't hit prod without someone seeing it.
>> Yeah, you're right.
>> [INAUDIBLE] Handling it.
>> So you're right, however, you're also wrong. [LAUGH] So we went through this at Hippo, actually. It was a very interesting conversation because that's how it was before for us.

[00:01:14]
Every time an outage would happen, every time something would break, DevOps channel, DevOps channel just became support ops for the whole company, basically. And we started recognizing that. We're like, dude, before you merge the PR, check your spelling. Okay, just do that and it'll be great. And what a lot of people don't realize is that's the most common scenario of a lot of these configuration problems.

[00:01:39]
Copy pasted it wrong, or something like that. Now, is it worth? And again, this is gonna sound really weird, but is it worth me as a DevOps engineer to take my time to try and fix that? We've already, like you said, we've already got the practices in place.

[00:01:56]
We've already got the container there, right? So if somebody comes to me with a broken environment and says, hey, then realistically, there's two options. Yes, I can take their hand and help them go through it, and I don't mind doing that. But if I have to do it five times for every, once a week, then that's Dev hours, that's money.

[00:02:18]
So what we realized was, like you mentioned, really tight scopes, right? But make it so that they can do these things, right? So our developers at Hippo actually have control over their own configurations. And we told them, we said, listen, you're in power of this. You break it, you fix it.

[00:02:40]
That's not us. We will make sure that the infrastructure is running. We'll make sure that the uptime is there. And we'll make sure that there's nothing underneath you that could fail, right? But, you are the one in charge of your service, not me. I am in charge of making sure that your service runs in the sense of it can deploy and things like that.

[00:03:01]
It's kind of like a boat. The moment that boat is ready to go out to sea, I stop doing what you would expect once the bottle hits the ship, right? At that moment, you're out to sea, and I'm here to help you. I'm coast guard, right? But you're the one on the boat with your service.

[00:03:19]
And so developers being empowered to manage their own services not only enables that. But it also makes it so that if you have organizational teams, like team leads and stuff like that. Then that means that those product owners now bring that ownership into the team, right? So product owners don't just think about, well, let's get our features out, but now they're also thinking, well, when are we gonna deploy?

[00:03:47]
So the next thing is gonna be continuous integration, right? Developers really care about how things are built, right? And so this is gonna be like, how do we build artifacts? Are we building docker images? Are we building binaries? What are we building, right? Testing changes, making sure that if I make a change, and this, man, this is a big one.

[00:04:08]
This is a really big one. Because we've noticed, even at Hippo, that developers are really like, they'll pick one approach and then they'll just use that approach. And a good example of that is is that we did create pipelines for our developers at Hippo, but what we realized was that's all they use.

[00:04:31]
We're like, what? We're like, are you testing locally? They're like, no, lemme just push it. I'm like, are you serious? They're like, yeah. But then we realized it's because the infrastructure got so distributed that that was the only way that they could test. And we're like, okay. So then we started looking at it, and there was two options, one, local.

[00:04:53]
[LAUGH] Let's do local. We didn't wanna do that. We have 60 to 150 services, depending on what part of the stack that you're talking about. So local for us is just impossible, right? So what did we do? We decided to improve our pipelines, make them faster. Make them run quicker and less cost heavy.

[00:05:17]
And what we essentially did was, is we started leaning on things like caching, making sure that branches have their own caches. Really trying to make it so what the developer's expectation, if they're gonna do that, let's make sure that they're not just burning money in the Cloud, [LAUGH] right?

[00:05:32]
And that was a big part of that. So testing is really important. Are you gonna test locally before you push? Are you gonna test in the Cloud, right? And what are the developer's expectations with that? If you're testing locally and in the Cloud, whose time are you wasting?

[00:05:47]
[LAUGH] Or if you have specific things that you have local, but then you only do specific things in the Cloud, then that might be a better approach to it too. Code coverage, this one is unfortunately not used as often as I wish it was. If you don't know what code coverage is, it's basically the concept of making sure you have a test for every line of your code.

[00:06:13]
Most companies don't do this. Call a spade a spade. [LAUGH] Most companies don't. They don't find value in it, right? Why do tests and coverage when I can just do tests? However, what we have seen at Hippo is that when you have code coverage, it gives ownership to the developer for those things.

[00:06:32]
And we're basically saying, hey, not only are you just making sure your tests run, but you're also monitoring the quality of your code, right? And so because of that, once again, you're giving them the ownership of their problems, right? You're not saying, well, code broke. I don't know what it is.

[00:06:50]
It's like, well, no, here are your tests. Here's your code coverage. You can find it, right? And then code quality. Now, code quality, DevOps really can only speak, I would say, about half on this conversation because the other half is the developers. The developers are the ones who are going to take their opinionations of how do we want to write the code?

[00:07:12]
What is it gonna look like? Are we gonna use formatters? Are we gonna use linters, whatever. Those things we don't, DevOps doesn't really care about. If you wanna lint, sure, we can we can add lint to the pipeline, whatever. But that's where we come in, right, is when that's where we work together with the developers.

[00:07:29]
And we say, okay, you want linting, how do we do it, right? So developers must have dependable and reliable pipelines for everything they need, basically. Yeah, and then finally continuous delivery. So going back to the example that I said before of, hey, an outage happens or something breaks, whose responsibility is it for, or is it?

[00:07:53]
I think it's the developer's, as we said before. So how do we do that? Well, we empower them to deploy their own services. And I'm gonna give you guys a quick story after this in a second related to this. And again, a big part of this is how do I deploy the Docker image, right?

[00:08:10]
How do I get that? And then again, how do I get that from staging to production to dev, to whatever, right? So here's my story. When I first got to Hippo, I found that every two weeks, the entire team or the entire company, which is about 100 people, would all join together on a Wednesday night, about 5 PM.

[00:08:34]
Get on a phone call, or get on a Zoom call, six other people, and we'd go through the whole production of deploying all 150 services. Yeah, it sucked. It was not fun at all. And I did that for probably about six to nine months before everyone on the team was like, my Wednesday nights are done.

[00:08:55]
I can't do anything, you know? And so, two things about this. One, it was a culture shift for us to think that developers could deploy things, hands down, right? Because from a business level, the reason why we did that was to make sure nothing broke, right? But what that really meant is we had no confidence in our developers for deploying things.

[00:09:22]
[LAUGH] That's what that really meant. This was every two weeks, right? So what did we do? We sat down and we figured out how we could empower developers to deploy their own services. Today, we had one service that could deploy once every two weeks. Now, it gets deployed four times a day.

[00:09:49]
Think about how fast you can reiterate if you just simply say, I'm not gonna be a part of this. You guys do what you need to do, right? And so that's what we did. We basically realized that as long as we have built a system in place that allows a developer to be like, send it to prod, and it goes, right?

[00:10:09]
That they take ownership of that. And don't get me wrong, there are times, we had a circumstance a few weeks ago, even, where a developer pushed something to prod, but it was broken, and we didn't know. But it wasn't a broken to where it wasn't running, it was an internal logic error or something like that.

[00:10:30]
However, they didn't check the service. They just deployed it to prod and then went home expecting that it was gonna be fine. The next morning, there's like tons of alerts and ta-da-da-da. And sure enough, we find out that yeah, they deployed it. They didn't think about it. Who's responsible for that one?

[00:10:47]
>> The developer didn't run any tests then.
>> Yep, exactly, yeah. You as an organization have to be ready for that. Because if not, then it's always gonna be on you. It's always gonna be like, well, DevOps does it. No, [LAUGH] we don't have to. And we changed that.

[00:11:06]
And so yeah, now we, across our 150 services, we deploy them multiple times a day. We have channels for them so that each team has their own channel they can do their deployments through. And we took that weight off of us as a team, and we pushed it on to the teams that own their actual products.

[00:11:26]
Right, so again, it becomes this separation of, if you've ever seen Ghostbusters, we're kinda like the gatekeepers, [LAUGH] you know what I mean? We only unlock the door, you're the one who has to walk through, right? And that empowerment's really important. So did you say developers maintain their own deployments?

[00:11:49]
Yes, I did. And I do think it's something that you guys should try and do. I think if anything, start thinking about it. If you're not doing that already, it's not like you couldn't. It's just that you really need to think about what are the pain points? How do they do it?

[00:12:03]
How are they gonna potentially hurt themselves, [LAUGH] if they do? But they're brilliant, talented people just like we are and they're hired to do engineering too so they should be able to. So why is developer empowerment important, especially for us? It saves us time [LAUGH] honestly. Like, there's two sides of it.

[00:12:29]
One, looking at faster development cycles. Like I said before, we went from deploying a service once every two weeks to deploying it four times a day. And this is our biggest, most important, and most touched service. And it gets deployed constantly now. So we see that loop, right?

[00:12:47]
The faster feedback loop with that. Faster time to market. Maybe you're in a scenario where your company needs to put something out, but you're blocked because you don't deploy except for on a specific time period. Well, then marketing has to think about, all right, well we gotta do it this day and we gotta plan around.

[00:13:06]
Like, it just, it becomes a burden and a lot of people don't really realize that it does, right? And then yeah, faster time to value through that. Because you can get to where you're trying to quicker. You get that value faster, you learn quicker and then you can iterate it.

[00:13:21]
So yeah, DevOps must empower developers to be successful. If we don't, the process doesn't work the way that we wish it would.