redpanda-data /connect
Fancy stream processing made operationally mundane
★ 8,799⑂ 983👁 111⑃ 3,308● 1,642▤ 562
-1
Daily
8,799
stars
983
forks
111
watchers
3,308
prs
1,642
issues
562
releases
README
Redpanda Connect
[![Build Status][actions-badge]][actions-url]
[![Apache V2 API][godoc-badge]][godoc-url-apache]
[![Enterprise API][godoc-badge]][godoc-url-enterprise]
Redpanda Connect is a stream processor that moves data between a wide range of [sources][inputs] and [sinks][outputs], with support for [hydration, enrichment, transformation, and filtering][processors] along the way.
That includes a rich set of change-data-capture (CDC) connectors — for Postgres, MySQL, MongoDB, Oracle, MSSQL, and more — so database changes can flow through your pipelines as first-class events.
It uses [Bloblang][bloblang-about] for mapping, runs as a single static binary or container image, and is easy to operate and monitor.
Highlights
- Declarative pipelines — a stream topology fits in a single YAML file.
- At-least-once delivery by default — in-process transactions, no disk state required.
- A large connector catalog — cloud services, message brokers, databases, HTTP, and more.
- First-class CDC — change-data-capture connectors for Postgres, MySQL, MongoDB, Oracle, and MSSQL.
- Bloblang — a mapping language designed for stream data.
- Cloud-friendly — stateless and horizontally scalable, with metrics and tracing built in.
Example
Stream Postgres changes into Apache Iceberg tables on S3, one Iceberg table per source table:
yaml
input:
postgres_cdc:
dsn: postgres://user:[email protected]:5432/app?sslmode=require
schema: public
tables: [ orders, customers ]
stream_snapshot: true
output:
iceberg:
catalog:
url: https://glue.us-east-1.amazonaws.com/iceberg
warehouse: "123456789012"
auth:
aws_sigv4:
region: us-east-1
service: glue
namespace: cdc
table: ${! meta("table") }
storage:
aws_s3:
bucket: my-iceberg-warehouse
region: us-east-1
schema_evolution:
enabled: true
table_location: s3://my-iceberg-warehouse/cdc/
Quickstart
Install
Linux:
shell
curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip -d ~/.local/bin/
macOS (Homebrew):
shell
brew install redpanda-data/tap/redpanda
Docker:
shell
docker pull docker.redpanda.com/redpandadata/connect
See the [getting started guide][getting-started] for more options.
Run
shell
rpk connect run ./config.yaml
With Docker:
shell
From a config file
docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run
With inline overrides
docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
-s "input.type=http_server" \
-s "output.type=kafka" \
-s "output.kafka.addresses=kafka-server:9092" \
-s "output.kafka.topic=redpanda_topic"
Connectors
The catalog includes AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob, Queue, Table), GCP (Pub/Sub, Cloud Storage, BigQuery), Kafka, NATS (JetStream, Streaming), NSQ, MQTT, AMQP 0.91 (RabbitMQ), AMQP 1, Top issues
#271upgrade "files" input with watchdog13 comments · 2019-09-06#2522GCP cloud storage output error with Failed to delete temporary file used for merging: context canceled11 comments · 2024-04-16#1677`sql_raw` output should support batched/transactional sends when possible9 comments · 2023-01-16#978Investigate elastic/opensearch forks8 comments · 2021-11-29#4648redpanda (kafka (eventhubs)) input stalling for exactly 15 minutes8 comments · 2026-07-31#4387Redpanda output with idempotent_writes makes unordered results7 comments · 2026-05-04#1679gcp_cloud_storage(input) lack of randomize Objects List7 comments · 2023-01-16#1581Implement `event_window` buffer7 comments · 2022-11-21
Discussions
#5051Does the HDFS output support writing to Erasure-Coded (EC) directories?1 comments · 2026-10-26#4622Http_Client stream.enabled=true disable/removal doesn't reliably terminate streaming0 comments · 2026-07-22#4552opensearch output: action does not support "create"0 comments · 2026-06-24#4442Multiple outputs with different formats (VictoriaMetrics + PostgreSQL)0 comments · 2026-05-20#4425How to build redpanda connect and include my custom processor1 comments · 2026-05-13#4264Logging to stdout and file0 comments · 2026-04-16#4259delete before insert with sql_raw0 comments · 2026-04-15#4213Why does AMQP_1 not preserve properties like message-id?2 comments · 2026-04-03
Latest release
v4.118.0 v4.118.0 2026-11-13For installation instructions check out the getting started guide. Added - azure: Azure Blob Storage output now supports blob index tags for organizing and filtering blobs, with configuration-time validation and best-effort tag application for pre-existing append blobs. (@cadiidii, 4204) - telemetry: Pipeline ID from the control plane is now included in the anonymous telemetry payload for correlation with usage and billing data. (@prakhargarg105, 4758) Fixed - dynamodb: DynamoDB CDC input now correctly deduplicates snapshots and CDC events across different time zones and within the same