As physical infrastructure has transitioned more and more to cloud and Platform-as-a-Service (PaaS) providers, engineering teams have had to adapt and address deficiencies in the developer and operations roles that traditionally worked in silos to build and deliver software.
Many companies have attempted to solve these deficiencies by creating the DevOps engineer, but eventually end up with a very similar "us versus them" mentality between the DevOps and development teams. Engineering as a whole should instead be working in unison towards the same goal: to deliver a high quality experience to their customers.
DevOps at KnowBe4
At KnowBe4 we define DevOps as the interface between traditional development and operations teams. The engineering umbrella encompasses software engineers, quality assurance, site reliability engineers, and software engineers in test all working on these DevOps responsibilities in order to ship reliable and performant software quickly.
To support this culture, we needed to ensure our teams were empowered to engage with their software across each stage of its life — from developing locally, to testing and deploying, to running and monitoring in AWS. When our repositories were first being designed, we made the decision to store all of our infrastructure-as-code, CI/CD, and other similar configurations in the same repository as the code they support. This allows software engineers to work autonomously, making changes to their infrastructure without needing to open a ticket with IT to make necessary changes that impact the way their applications run.
Shared Responsibility Model
Years ago, Amazon Web Services (AWS) established their shared responsibility model, which compares configuration work that customers must own with the responsibilities that AWS retains in order to, in tandem, build secure and compliant software in the cloud.
KnowBe4 takes a similar approach to DevOps responsibilities. Traditional operations needs — building and monitoring infrastructure, deploying code, and maintaining security in the cloud — are a shared responsibility that all engineers must consider when writing their software. In this way, the traditional "developer" role is exposed to the big picture: life exists after local development is complete.
The "operations" role is still largely necessary in this model, but instead has a different set of responsibilities: to understand the needs of those filling the "developer" role, to build processes and standards that make those needs easier to meet, and to provide subject-matter expertise on the plethora of systems and tools that exist to deliver modern cloud-native software.
Execution
This all sounds great, but what does it actually look like in the day-to-day life of an engineer? At KnowBe4 we have identified a few key areas where collaboration between all engineering teams is paramount to maintain and proliferate a strong DevOps culture.
Pair Programming
Early on, we discovered that pair programming — a technique where two engineers work together to find solutions to programming needs — is an essential piece of the DevOps culture. At KnowBe4, we often have engineers from separate teams pair program to share knowledge and bolster a fundamental understanding of all the various components that comprise a successful system.
Even some of our most complex cultural shifts have been simple to implement by addressing them in one-on-one interactions while pair programming. It has been the single most effective way to get buy-in from other teams for new technologies and processes, and provides an avenue to solicit mutual feedback from both parties and arrive at a conclusion or solution that satisfies both parties' needs.
The Design Phase
Our most successful projects have been those engineered by multiple teams. Projects that involve discussions around the nuances of our system designs, scale, testability, and overall reliability during the design phase result in more effective collaboration amongst engineering teams with stake in the success of the project.
Approving Changes
A major part of our software development life cycle (SDLC) involves engineers approving other engineers' repository changes. Since our repositories include both code and infrastructure configurations, these approvals often involve multiple teams.
In order to maintain accountability and standards, infrastructure changes are always approved by both our software engineering and site reliability engineering teams. In this vein, such changes are highly visible, resulting in an increased level of collaboration between these teams.
This has also made our external audits more straightforward — since infrastructure and CI/CD changes live in the code repository and follow the same approval requirements, all such changes are easily demonstrated and audited in a way that is familiar to auditors and streamlines the audit process.
Observability in Production
One of the most important tools used in engineering is Slack. Our engineers rely on Slack for not just rapid communication, but also a variety of alerts and messages related to the status and health of the services they maintain.
Each production deploy posts and updates its status in a public Slack channel so maintainers have to-the-minute visibility into their deployments without having to adjust their workflows or login to another system. Build, test, and deployment failures are addressed in a simple thread, with the relevant subject matter experts coming together to identify a resolution quickly.
Our system monitors and alerts also target Slack as their primary means of delivery, allowing engineers to quickly triage potential problems before they become incidents. This is also a great place for the various engineering teams to collaborate on solutions and reach out for extra help all without leaving Slack.
What Comes Next?
While all of this has been overall successful, our journey is not over. As cloud providers continue to release new services every day, new technology trends continue to emerge that need an elevated level of collaboration within all engineering teams. Preventing work silos by enabling teams to work autonomously while continuing to support their existing runtime systems is something that requires intentional effort across the greater engineering department.
In 2022, our platform engineering team is kicking off a new developer experience (DevEx) team focused on developer productivity and enablement. Our goal is to take some of our most complex problems, solicit feedback on what makes them difficult, and build solutions to make them easier to solve and maintain.
Interested in joining one of our engineering teams? KnowBe4 Engineering is always looking for more talented engineers just like you! Check our open positions on our careers page - www.knowbe4.com/careers.
You could join our amazing software engineering team!
