dbt

dbt (data build tool) is an open-source command-line application and development framework designed specifically for analytics engineering, enabling data teams to apply software engineering principles—such as version control, testing, documentation, and CI/CD—to data transformation workflows. At its core, dbt operates on a declarative paradigm: users write SELECT statements (in SQL) to define transformations, and dbt handles the execution, dependency resolution, incremental materialization (views, tables, incremental tables, ephemeral models), and lineage tracking. It integrates natively with major cloud data warehouses including Snowflake, BigQuery, Redshift, Databricks, and PostgreSQL, abstracting away warehouse-specific syntax through its adapter architecture. Key capabilities include robust testing (schema, uniqueness, not null, custom tests), auto-generated data documentation with interactive lineage graphs, modular project structure via packages and macros, environment-aware configuration (via profiles.yml and dbt_project.yml), and seamless Git integration for collaboration and code review. The dbt ecosystem spans dbt Cloud (a SaaS offering with scheduling, UI, IDE, and enterprise security features), dbt Core (open-source CLI), dbt Semantic Layer (for metric definitions and consistent business logic), and a thriving package registry (e.g., dbt_utils, dbt_date). Use cases span transforming raw ingestion layers into clean, modeled datasets; implementing slowly changing dimensions; building BI-ready marts; enforcing data quality across pipelines; enabling self-service analytics via documented, trusted models; and bridging the gap between data engineering and analytics by empowering analysts to own transformation logic. dbt does not extract or load data (it's ELT-native), nor does it replace orchestration tools—but it integrates tightly with Airflow, Dagster, Prefect, and others. Its growing adoption reflects a paradigm shift toward treating analytics code as production-grade software.