High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Open-source data movement for ELT pipelines and AI agents โ from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
Flink CDC is a streaming data integration tool
The leader in Customer Data Infrastructure
Data processing for and with foundation models! ๐ ๐ ๐ฝ โก๏ธ โก๏ธ๐ธ ๐น ๐ท
A list of useful resources to learn Data Engineering from scratch
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
Where data access meets operational intelligence
Open-source inference server and production cluster for all the models your agent needs.
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.