Data Engineering

Pipelines, warehouses, transforms

4 tasks4 tools

Orchestrate data pipelines

Data Engineering

Apache Airflow

The leading workflow orchestration platform for data pipelines. Airflow lets you define, schedule, and monitor complex DAG-based pipelines in Python. The standard for data engineers and ML pipeline orchestration.

PythonOpen SourceSelf-hosted

Trust

Excellent

Compare:vs dbt

Transform & model warehouse data

Data Engineering

dbt

The leading data transformation tool for analytics engineers. dbt lets you write SQL SELECT statements and handles materialization, testing, documentation, and lineage. It transformed how data teams work.

PythonSQLOpen Source

Trust

Limited

Compare:vs Apache Airflow

Query a cloud data warehouse

Data Engineering

Snowflake

The leading cloud data warehouse. Snowflake separates compute from storage, scales elastically, and supports structured and semi-structured data. The primary target warehouse for most modern data stacks paired with dbt.

SQLSaaSPaid

Trust

Strong

—

Build a real-time stream pipeline

Data Engineering

Apache Kafka

The dominant distributed event streaming platform. Kafka handles high-throughput, durable message queues for real-time data pipelines and event-driven architectures. The backbone of modern data infrastructure at scale.

JavaOpen SourceSelf-hosted

Trust

Excellent

—