Introducing Redpanda Connect

Building an experience that wows users instantly is directly proportional to the effort placed by the builders at every turn, every commit. Not to mention hundreds of hours wasted using bad products to form strong opinions.

The time it takes to create your first, globally-distributed cluster from sign up in Serverless is ~100 milliseconds with Redpanda. 90 seconds to deploy a production cluster of Kubernetes on EKS. And of course, less than 1 second to bootstrap a 3-node cluster on your laptop. Sheer terror for anyone else.

Today, you can connect anything to anywhere in 140 milliseconds with Redpanda Connect.

time `rpk connect run`
rpk connect run  0.14s user 0.06s system 17% cpu 1.148 total

Any form factor. Any scale. Our largest Redpanda Connect user is pushing 50Gbps via 2X 25Gbps NICs to consume market data. Our smallest Redpanda Connect user is using 100 millicores and 10 MiB of memory, processing hundreds of messages a second deployed on a heavily resource-constrained environment.

The largest – 225 to be exact – connectivity platform for the Kafka ecosystem. There simply does not exist a more complete, end-to-end platform for streaming your data that spans every single form factor from embedded IoT devices to the largest cloud workloads.

Redpanda Connect comes via the acquisition of the popular benthos.dev stream processing framework. Ash, Founder and CEO, has been tirelessly working on Benthos for the past seven years, evolving the platform with some of the world’s largest companies spanning finance, oil & gas, healthcare, SasS, Adtech, and social media, etc.

We’ve always grown Redpanda with three core cultural values: Customer obsession, Be a good human, and Win as a team. While the traction of the product speaks for itself with almost 8,000 GitHub stars and users in every imaginable industry, the company bet was on Ash – the human and brains behind benthos.dev – and team. Thank you: we couldn’t have found a better partner, and welcome to the Redpanda family!

For Redpanda, this represents a meaningful transition to the most complete, end-to-end streaming platform. Whether you’re squeezing microsecond-level latency for the most demanding workloads, or you’re building one of the world's largest social networks like ShareChat with 390 million monthly active users, or you’re building the most reliable banking products like Zafin IO for the world's largest commercial banks, you can now build and scale your business without the need to stitch multiple projects together. All under one roof. I couldn’t be more excited about the future of Redpanda.

For users, this means continued support of 223 Apache 2 Licensed, open source connectors with the large financial backing of Redpanda and customer footprint. It means increased focus on reliability, day 2 operations and security. We’ll be bringing the known and loved Benthos Studio into Redpanda Console (UI) for a visual representation of the connectivity in your business.

Lastly, we know firsthand how difficult it is to build a framework that supports intelligent retries, routing, supervision, and built-in telemetry. We know many of you bet your business on the core engine of [Redpanda] Benthos, and we couldn’t be more proud to continue to support you by keeping the core engine MIT with all the freedom and protections it provides. We can’t wait to continue building together. Please see the core engine repo here.

Redpanda Connect: open source connectors

We couldn’t be happier keeping 99%+ of the connectors with an Apache 2, Open Source license. For the connectors below, you can purchase support from the Redpanda team for “Certified” connectors. These connectors have very large production users. For the databases, below we note that they tend to be in the regular test matrix. “Community” connectors are just that – Slack and GitHub support on a best-effort basis.

Every connector is in production today, but we want to ensure that your experience is curated and that Redpanda can stand behind the stability and scalability of the Certified group first.

‍

See our Connector Catalog for the latest list.

Redpanda Connect: Enterprise connectors

Two connectors require a paid subscription to Redpanda: snowflake_put and splunk. All future improvements, security vulnerability patches, and builds require a subscription to Redpanda. While we worked hard to keep most of the connectors permissively licensed, which may include competitors, I want to be transparent about our monetization model so there is no ambiguity or uncertainty.

‍

‍

Redpanda Benthos core engine

For developers and engineers who built their entire business on [Redpanda] Benthos, you can continue to do that, as we’re keeping the core engine MIT Licensed. All those hard-to-build, hard-to-test features that made [Redpanda] Benthos so reliable over the years are free and open source for you to use. This is a win for the community that built their business on top of [Redpanda] Benthos – we are so sure of it, we bet the future of Redpanda connectivity on this engine itself. I can’t wait to see how we evolve it together.

More to come, but don’t hesitate to continue collaborating with us on github.com/redpanda-data/benthos

Get in touch

It would be my and the team’s privilege to have the opportunity to partner with you. Whether you are using Redpanda Connect, or the Redpanda Console for the best UI in the streaming world or our storage engine Redpanda. Please reach out!

.alex founder & ceo

‍