igorbarinov /awesome-data-engineering

A curated list of data engineering tools for software developers

★ 9,264⑂ 1,679👁 243⑃ 354● 26▤ 0
CC0-1.0Created 2015-06-18
+1
Daily
9,264
stars
1,679
forks
243
watchers
354
prs
26
issues
0
releases

README

Awesome Data Engineering 

 A curated list of awesome things related to Data Engineering.

 Contents

- Databases
- Data Comparison
- Data Ingestion
- File System
- Serialization format
- Stream Processing
- Batch Processing
- Charts and Dashboards
- Workflow
- Data Lake Management
- ELK Elastic Logstash Kibana
- Docker
- Datasets
  - Realtime
  - Data Dumps
- Monitoring
  - Prometheus
- Profiling
  - Data Profiler
- Schema
- Testing
- Community
  - Forums
  - Conferences
  - Podcasts
  - Books

 Databases

- Relational
  - RQLite - Replicated SQLite using the Raft consensus protocol.
  - MySQL - The world's most popular open source database.
    - TiDB - A distributed NewSQL database compatible with MySQL protocol.
    - Percona XtraBackup - A free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®.
    - mysql_utils - Pinterest MySQL Management Tools.
  - MariaDB - An enhanced, drop-in replacement for MySQL.
  - PostgreSQL - The world's most advanced open source database.
  - Rivestack - Managed PostgreSQL with pgvector for AI workloads. HNSW indexing, sub-4ms latency, and built-in SQL editor with automatic embedding generation.
  - Amazon RDS - Makes it easy to set up, operate, and scale a relational database in the cloud.
  - Crate.IO - Scalable SQL database with the NOSQL goodies.
- Key-Value
  - Redis - An open source, BSD licensed, advanced key-value cache and store.
  - Riak - A distributed database designed to deliver maximum data availability by distributing data across multiple servers.
  - AWS DynamoDB - A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale.
  - HyperDex - A scalable, searchable key-value store. Deprecated.
  - SSDB - A high performance NoSQL database supporting many data structures, an alternative to Redis.
  - Kyoto Tycoon - A lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency.
  - IonDB - A key-value store for microcontroller and IoT applications.
- Column
  - Cassandra - The right choice when you need scalability and high availability without compromising performance.
    - Cassandra Calculator - This simple form allows you to try out different values for your Apache Cassandra cluster and see what the impact is for your application.
    - CCM - A script to easily create and destroy an Apache Cassandra cluster on localhost.
    - ScyllaDB - NoSQL data store using the seastar framework, compatible with Apache Cassandra.
  - HBase - The Hadoop database, a distributed, scalable, big data store.
  - AWS Redshift - A fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools.
  - FiloDB - Distributed. Columnar. Versioned. Streaming. SQL.
  - Vertica - Distributed, MPP columnar database with extensive analytics SQL.
  - Click

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON