From Vision to Reality: A Deep Dive into KnowBe4's Transformative Event-Driven Design

In the ever-evolving world of technology, staying a step ahead of the curve is paramount for businesses to thrive. At the heart of this mantra, KnowBe4's shift to an event-driven architecture (EDA) stands as a testament to forward-thinking and strategic adaptation. Not only has this new architectural pattern accelerated the time-to-market for new features, but it has also revolutionized the way platforms communicate within the KnowBe4 ecosystem. By decoupling direct service-to-service communication and centralizing it through a robust event bus, we've paved the way for faster, more efficient feature releases.

The Collaborative Power of the Platform Architecture Working Group (PAWG)

KnowBe4's pioneering event-driven architecture isn't solely a technological achievement; it's the result of a collective effort by the Platform Architecture Working Group. This group comprising of our internal engineering leaders, convened regularly over months to design and iterate on the architecture. Their discussions encompassed critical aspects like event-driven architecture across AWS Accounts, schema validation to maintain consistency within the event bus, data retention policies, scalability strategies, large event payload handling and event searchability mechanisms. The team dissected and reassembled the design blueprint multiple times, refining its core components to perfection:

• Event Bridge Across AWS Accounts Design: The team recognized the need for a centralized event bus that could handle events across multiple AWS accounts. This ensures that the event-driven architecture is scalable and flexible enough to accommodate future growth and easily managed versus a decentralized bus topology.

• Schema Validation: By introducing strict schema validation, the team addressed concerns of the event bus becoming too chaotic or akin to the "Wild West". This ensures consistent and predictable operations.

• Data Retention Policies: Recognizing the value of data and the importance of its management, robust data retention policies were put in place to maintain data integrity.

• Large Event Payloads: Large events are not a common occurrence in the KnowBe4 ecosystem, but cannot be ruled out. The team ensured that the event-driven architecture could handle large event payloads by offloading some data to S3 without compromising filtering, performance or efficiency.

• Scalability & Searchability: Given the rapid growth of the KnowBe4 ecosystem, the architecture was primed to scale effortlessly, without compromising on the speed and efficiency of data retrieval.

Harnessing the Power of Rust & Python

In the architectural fabric of KnowBe4's event-driven system, two programming languages distinctly stand out: Rust and Python. Both, in their unique capacities, contribute to the overall agility and efficiency of the new system.

• Rust: As a language known for its concurrency capabilities and safety features, Rust provides the robustness required for real-time, event-driven scenarios. Its memory safety guaranteed without a garbage collector ensures that the core functionalities run with maximum efficiency and minimal overhead. This means quicker event processing, and in an ecosystem where every millisecond counts, Rust's performance advantages become pivotal.

• Python: On the other hand, Python, with its vast libraries and frameworks, facilitates rapid development and seamless integration with various AWS services. Its dynamic nature and expressive syntax make it an ideal choice for scripting and automating various backend processes. Furthermore, Python's supportive community ensures a constant influx of tools and resources that can be harnessed to enhance the event-driven infrastructure.

Together, Rust and Python form the ideal duo for our platform needs. While Rust takes charge of performance-critical tasks ensuring speed and security, Python enhances the system's flexibility and adaptability, making it easier to introduce new features and integrate with external services.

Leveraging Terraform Modules for Infrastructure as Code

The KnowBe4 event-driven architecture is built on the foundation of Infrastructure as Code (IaC). This approach ensures that the infrastructure is defined, provisioned, managed through code, version controlled, and allowing for easy rollback in case of any issues. This not only reduces the risk of human error but also ensures that the infrastructure is consistent and repeatable. To achieve this, the team leveraged Terraform modules for publishing events and for subscribing to events. KnowBe4 development teams reuse these terraform modules to quickly implement their own specific event handling in AWS.

The AWS Partnership: A Catalyst for Accelerated Implementation

KnowBe4's partnership with AWS through their Experienced Based Accelerator (EBA) program proved to be the keystone for this initiative. This collaboration ensured the swift and efficient transition of the KnowBe4 Data Foundation Service (DFS) first subscriber to production. AWS's EBA program, with its hands-on approach and expertise, helped in identifying potential challenges and addressing them proactively, further reducing the time to market.

Marrying AWS Services to Power the EDA

This new architecture harnesses a multitude of AWS services, working in tandem to deliver an unparalleled EDA experience:

• EventBridge: At the heart of KnowBe4's event-driven architecture is the centralized event bus. This component acts as a hub for events generated across our ecosystem. Whether it's user actions, system updates, or application events, everything is funneled through this single event bus. EventBridge, an AWS service, is a natural fit for this role. It simplifies event ingestion, distribution, and management, allowing our engineers to focus on business logic rather than infrastructure concerns. By leveraging EventBridge's capabilities, we achieved seamless event routing across multiple AWS Accounts, filtering, and transformation, creating a robust foundation for our architecture.

• Lambda: In an event-driven architecture, scalability is a critical factor. AWS Lambda became the go-to solution for executing code in response to events. Lambda's serverless nature means we only pay for the compute resources used during execution, ensuring cost-efficiency. KnowBe4 engineers are no stranger to Lambda and serverless architectures. We have proven our platforms can instantly scale based on incoming event traffic. This elasticity guarantees that our applications can handle both routine operations and unexpected spikes in demand without compromising performance or user experience.

• API Gateway: For future external partners, current engineers, and applications to interact with the event-driven ecosystem, an API Gateway is essential. This AWS service acts as a front door, enabling us to create, publish, maintain, monitor, and secure APIs at any scale. By implementing an API Gateway, we ensured that access to our event-driven architecture is controlled and secure. We enforced usage quotas, manage API versions, and secure communication through authentication and authorization mechanisms.

• S3 and RDS: Data Management and Persistence A robust event-driven architecture doesn't just handle real-time events; it must also manage and persist data. Amazon S3 and Amazon RDS play crucial roles in this aspect. Amazon S3 provides scalable object storage, enabling us to store vast amounts of event data reliably. RDS, on the other hand, is leveraged for structured data storage, ensuring data consistency and durability.

Boosting Developer Productivity

One of the most frustrating things for a developer is having to wait for your CI/CD pipelines to deploy a one line code change. In order for our engineers to be able to work on their code locally, we needed to provide them with a way to run the entire event-driven architecture locally. This was achieved by leveraging LocalStack Pro to spin up all the necessary AWS services locally (EventBridge, APIGateway, Lambdas, S3, SQS, etc). This allowed our engineers to work on their code locally and test it end to end before committing their code to the repository. This reduced the engineering cycle and increased developer productivity.

In Conclusion

Our transition to an event-driven architecture not only embodies the commitment to innovation but also positions us at the forefront of technological evolution. The combination of visionary engineering, strategic partnerships, and leveraging cutting-edge technology ensure our continued dominance as market leaders in our domain.