Building Redpanda Cloud's privacy-first architecture

Anyone who's ever attempted vegetable gardening knows that it’s far easier to succeed with already-growing plants than ones you start from seed. Literally growing something from the ground up takes a lot of time, work, and careful monitoring. The same can be said of building software.

I’ve worked at both large and small companies in the past but, in both types of settings, my work as an engineer was already determined: the software already had roots; it was already in the ground, so to speak. As an early member of the Redpanda team, however, I’ve witnessed and contributed to the passion and care required to build software from scratch. Specifically, I was the first person at Redpanda to work on engineering our cloud platform.

Today, I’m a Site Reliability Engineer at Redpanda. I joined the team in the early days, back in September 2020. I live and work from my home country of Greece, collaborating daily with the rest of our fully distributed remote team via Zoom and Slack.

In this post, I’ll tell you about what it’s been like engineering our cloud from the ground up, as well as explain what our cloud product offers to our users.

The origin of Redpanda’s cloud

My first few months at Redpanda were spent ramping up our internal CI/CD process for the Core product, as that was where we needed to focus our attention at the time. However, as time passed, we knew more customers would want to have the option of infinite scalability offered by a cloud platform, and so our cloud initiative was where I began devoting all of my time. Building our cloud into what it is today has been both fun and challenging. When you’re building something completely from the ground up, there are a lot of decisions to be made, and you want to make them all quickly so you can keep moving the project forward and engineering the next phase of the platform.

The main challenge we faced was creating a cloud platform product from scratch. In an early-stage startup, objectives and requirements can change quickly, and you have to be agile in your execution to keep up with and meet demands. Because we were engineering our cloud from nothing, we also essentially needed to be the first consumers of our product while we were building it out. This is challenging because you need to use the product as you’re creating it. In the end, though, we built a cloud service that we’re proud of that provides a seamless experience for our cloud customers.

Fully-managed services your own way: Dedicated and BYOC clusters

As we built our cloud, we realized that many of our customers wanted the ability to choose between a Redpanda cluster deployed within our cloud infrastructure or within their own virtual private cloud (VPC). To give customers these options, we created two types of clusters:  Dedicated clusters and Bring Your Own Cloud (BYOC) clusters.

Dedicated clusters allow you to host your Redpanda clusters in Redpanda’s cloud environment. BYOC, on the other hand, allows you to host your Redpanda clusters in your own VPC. In both scenarios, Redpanda manages the provisioning, operations, and maintenance for you.  

BYOC clusters are unique to Redpanda Cloud, and allow you to scale on your own infrastructure while meeting any data sovereignty requirements you might have. Your data or cloud credentials never leave your environment. We believe this is the future for all cloud services.

Cutting-edge technologies in the cloud

As a core product, Redpanda is a cutting-edge platform to work on because it is able to make the most of modern hardware resources, and so it can perform faster than other data streaming platforms because it was specifically designed to take advantage of modern computing capabilities. For C++ engineers and developers working on the Core team, that’s an interesting kind of technology to be able to work on.

From the cloud perspective, if you’re a member of our Cloud or SRE teams, you get to work on the latest technologies being used in cloud platforms today, such as developing with Kubernetes or tiered storage in mind. You also get to operate, tweak, and work with Redpanda, which gives you the chance to be curious and experiment with your engineering work on a daily basis.

The majority of developers don’t get to work with the software they’re building this much at their day jobs, so it’s a very rewarding experience. In terms of the tech stack we use, I spend most days working with Terraform, Kubernetes, Grafana, Prometheus, and Golang, and some Bash magic is usually in there, too.

Building site reliability engineering into the Cloud team

In the first iterations of the cloud team, I was essentially the person working on all things infrastructure and platform, contributing to product decisions, and also developing small features. As the team scaled, we eventually formed a separate site reliability engineering (SRE) team within the Cloud team, which made dividing responsibilities a bit more manageable with the creation of specific focus areas. More recently, we’ve been working toward defining how reliability impacts our cloud product, leveling up our monitoring game, and defining a robust incident response process for Redpanda as a whole.

Conclusion

Building a reliable and fast cloud platform to host, stream, and transform data is, in my opinion, as fascinating as it gets engineering-wise. My advice for anyone interested in working on Redpanda’s Cloud team would be to come to the opportunity with, obviously, a background in engineering and the relevant technologies required there, but to also come with a sense of curiosity for what you can build.

Working for Redpanda has enabled me to be involved in many different aspects of what it means to develop a solid cloud product, from performance engineering to platform reliability to data management. In a world where real-time data streaming is becoming the norm, I’m excited to be working with a team at the forefront of that and to see what the future of streaming holds. If this sounds interesting to you too, take a look at the Cloud team positions we’re hiring for right now!

Join our Redpanda Community on Slack to see what others are building with Redpanda, or browse our documentation and GitHub to learn more about our platform.

‍