redpanda-data /connect

Fancy stream processing made operationally mundane

★ 8,799⑂ 983👁 111⑃ 3,308● 1,642▤ 562
Download: v4.118.0v4.118.0 ↓4.118.02026-11-13
-1
Daily
8,799
stars
983
forks
111
watchers
3,308
prs
1,642
issues
562
releases

README

Redpanda Connect

[![Build Status][actions-badge]][actions-url]
[![Apache V2 API][godoc-badge]][godoc-url-apache]
[![Enterprise API][godoc-badge]][godoc-url-enterprise]

Redpanda Connect is a stream processor that moves data between a wide range of [sources][inputs] and [sinks][outputs], with support for [hydration, enrichment, transformation, and filtering][processors] along the way.

That includes a rich set of change-data-capture (CDC) connectors — for Postgres, MySQL, MongoDB, Oracle, MSSQL, and more — so database changes can flow through your pipelines as first-class events.

It uses [Bloblang][bloblang-about] for mapping, runs as a single static binary or container image, and is easy to operate and monitor.

 Highlights

- Declarative pipelines — a stream topology fits in a single YAML file.
- At-least-once delivery by default — in-process transactions, no disk state required.
- A large connector catalog — cloud services, message brokers, databases, HTTP, and more.
- First-class CDC — change-data-capture connectors for Postgres, MySQL, MongoDB, Oracle, and MSSQL.
- Bloblang — a mapping language designed for stream data.
- Cloud-friendly — stateless and horizontally scalable, with metrics and tracing built in.

 Example

Stream Postgres changes into Apache Iceberg tables on S3, one Iceberg table per source table:

yaml
input:
  postgres_cdc:
    dsn: postgres://user:[email protected]:5432/app?sslmode=require
    schema: public
    tables: [ orders, customers ]
    stream_snapshot: true

output:
  iceberg:
    catalog:
      url: https://glue.us-east-1.amazonaws.com/iceberg
      warehouse: "123456789012"
      auth:
        aws_sigv4:
          region: us-east-1
          service: glue
    namespace: cdc
    table: ${! meta("table") }
    storage:
      aws_s3:
        bucket: my-iceberg-warehouse
        region: us-east-1
    schema_evolution:
      enabled: true
      table_location: s3://my-iceberg-warehouse/cdc/


 Quickstart

 Install

Linux:

shell
curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip -d ~/.local/bin/


macOS (Homebrew):

shell
brew install redpanda-data/tap/redpanda


Docker:

shell
docker pull docker.redpanda.com/redpandadata/connect


See the [getting started guide][getting-started] for more options.

 Run

shell
rpk connect run ./config.yaml


With Docker:

shell
 From a config file
docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run

 With inline overrides
docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
  -s "input.type=http_server" \
  -s "output.type=kafka" \
  -s "output.kafka.addresses=kafka-server:9092" \
  -s "output.kafka.topic=redpanda_topic"


 Connectors

The catalog includes AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob, Queue, Table), GCP (Pub/Sub, Cloud Storage, BigQuery), Kafka, NATS (JetStream, Streaming), NSQ, MQTT, AMQP 0.91 (RabbitMQ), AMQP 1, 

Top issues

Discussions

Latest release

v4.118.0 v4.118.0 2026-11-13
For installation instructions check out the getting started guide.

 Added

- azure: Azure Blob Storage output now supports blob index tags for organizing and filtering blobs, with configuration-time validation and best-effort tag application for pre-existing append blobs. (@cadiidii, 4204)
- telemetry: Pipeline ID from the control plane is now included in the anonymous telemetry payload for correlation with usage and billing data. (@prakhargarg105, 4758)

 Fixed

- dynamodb: DynamoDB CDC input now correctly deduplicates snapshots and CDC events across different time zones and within the same 

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON