DLT vs Dunwich: A Comparison of Modern Data Ingestion Tools

By Admin — 2025-12-01

Python DLT On-Premises Data Ingestion versus Dunwich
In the evolving landscape of data engineering, choosing the right tool for data ingestion and pipeline management can make or break your data infrastructure. Today, we're comparing two distinct approaches to this challenge: dlt (data load tool) from dltHub and Dunwich from Starless.IO.

What Are These Tools?

dlt: The Open-Source Python Data Loading Library

dlt is an open-source Python library designed to load data from various messy sources into well-structured datasets. It offers schema discovery, versioning, and evolution capabilities that enable users to effortlessly load data with row and column-level lineage.
Key Philosophy: dlt takes a developer-first approach, positioning itself as a library rather than a platform. Unlike other solutions that require backends or containers, dlt can be imported directly into any Python environment, AI code editor, or Jupyter Notebook.

Dunwich: The Privacy-First On-Premises Data Warehouse Ingestion Platform

Dunwich is a comprehensive data warehouse ingestion platform built from modular components, engineered to streamline data ingestion processes while maximizing performance and ensuring reliable end-to-end delivery of data to its intended destination.
Key Philosophy: Dunwich focuses on on-premises sovereignty, GDPR compliance at the ingestion layer, and predictable flat-rate pricing. It's designed for enterprises with strict data residency requirements who need complete control over their data without the surprises of volume-based pricing.

Core Capabilities Comparison

Data Ingestion & Loading

dlt:
  • Works with JSON data, dataframes, generator functions, and other iterable objects
  • Supports three write dispositions: full load (replace), append, and merge with deduplication
  • Provides automatic schema migrations on destinations without requiring SQL knowledge
  • Supports a variety of destinations including DuckDB, BigQuery, Snowflake, and custom destinations
Dunwich:
  • Universal ingestion supporting REST APIs, gRPC endpoints, and Debezium CDC streams
  • Built-in integration with Debezium server for change data capture from PostgreSQL, MySQL, MariaDB, and other databases
  • Targets data warehouses specifically: AWS Redshift, PostgreSQL, and MySQL/MariaDB
  • Modular component architecture for reliable end-to-end data delivery

Privacy & Compliance

dlt:
  • Provides data governance features including data lineage and schema versioning
  • Supports data transformations before and after loading
  • Schema contracts to govern how schemas evolve
Dunwich:
  • Privacy-first architecture with GDPR compliance built into the data pipeline
  • Column-level privacy controls: flag columns for hashing, encryption, or complete omission directly at ingestion
  • No post-processing required for compliance - privacy by design
  • Ideal for regulated industries with strict data handling requirements

Deployment & Control

dlt:
  • Runs anywhere Python runs (Airflow, serverless functions, cloud deployments)
  • No separate backend infrastructure required
  • Local development with easy transition to production
Dunwich:
  • On-premises sovereignty: runs entirely within your infrastructure
  • Data never leaves your network - no third-party cloud dependencies
  • Perfect for government contractors, regulated industries, and enterprises with data residency requirements
  • Self-contained solution with no external cloud service dependencies

Architecture & Deployment

dlt

  • Deployment Flexibility: Can be deployed anywhere Python runs, including Airflow, serverless functions, or any cloud deployment
  • No Infrastructure Requirements: Runs as a library, no separate backend needed
  • State Management: Advanced state management allows storing and retrieving values across pipeline runs by persisting them at the destination

Dunwich

  • On-Premises First: Designed to run entirely within your infrastructure
  • Warehouse Agnostic: Supports AWS Redshift, PostgreSQL, and MySQL/MariaDB - switch targets without rewriting ingestion logic
  • Protocol Flexibility: REST for simplicity, gRPC for performance, Debezium CDC for real-time change capture
  • Observability: Real-time monitoring, detailed logging, and actionable alerts built-in

Ecosystem & Platform

dlt / dltHub

dlt exists in two forms:
  1. dlt (Open Source): The production-ready Python library for moving data, distributed under Apache 2.0 license
  2. dltHub Platform (In Development): An LLM-native data engineering platform that extends dlt with managed runtime, storage choices, transformations, and quality tooling
The platform offering includes:
  • Built-in multiple environment support through profiles that isolate configurations and data storage
  • Managed lakehouse options or bring-your-own-storage
  • AI agentic support through MCP servers to analyze pipelines and datasets

Starless.IO / Dunwich

Dunwich is a focused, boutique solution from Starless.IO that emphasizes:
  • Single-purpose excellence: Does one thing exceptionally well - on-premises data warehouse ingestion with privacy built in
  • No feature bloat: Focused on reliability over "résumé-driven development"
  • Battle-tested engineering: Proven approach for teams that value consistency and predictability
  • Transparent roadmap: Public binaries and updates, license required for production use

Use Cases & Best Fits

When to Choose dlt

dlt excels in scenarios where you need:
  • Rapid prototyping: Build and test locally before deploying
  • Python-native workflows: Seamless integration with existing Python data stacks
  • Schema evolution: Automatic handling of changing data structures
  • Governance & lineage: Built-in governance features including data lineage, schema versioning, and compliance adherence
  • Custom sources: The ability to write any custom data source and load from APIs, files, or databases
  • Flexible destinations: Support for diverse data warehouses, lakes, and databases
  • Cloud-native deployments: Running on serverless, containers, or managed orchestrators

When to Choose Dunwich

Dunwich is the right choice when you need:
  • On-premises sovereignty: Complete control with data that never leaves your infrastructure
  • GDPR compliance at ingestion: Privacy-first architecture with column-level hashing, encryption, and omission
  • Predictable costs: Flat yearly licensing without volume-based pricing traps
  • High-volume operations: Scale from 100 GB to 100 TB without cost surprises
  • Regulated industry requirements: Government contractors, healthcare, finance, or any sector with strict data residency rules
  • Warehouse flexibility: Ability to switch between Redshift, PostgreSQL, or MySQL/MariaDB without pipeline rewrites
  • CDC integration: Real-time change data capture via Debezium for PostgreSQL, MySQL, and MariaDB
  • No vendor lock-in: Freedom to migrate warehouses or negotiate better terms

Maturity & Community

dlt

  • Described as the most popular production-ready open source Python library for moving data
  • Active GitHub repository with comprehensive documentation
  • Growing community with Slack support
  • Backed by Anthropic funding and development
  • Regular updates and active development

Dunwich

  • Commercial product with a focused, boutique approach
  • Publicly available binaries for testing
  • Updates are public and downloadable by anyone
  • Community size and public adoption metrics not prominently featured
  • Emphasis on reliability and battle-tested engineering over hype

Pricing & Licensing

dlt

  • Core dlt library is open source under Apache 2.0 license and free to use
  • dltHub Workspace has a Free tier with most functionality, with some features requiring Basic tier licensing
  • dltHub Runtime is in private preview
  • No volume-based charges for the open-source library

Dunwich

  • Flat yearly licensing
  • Initial setup fee (one-time cost)
  • Optional support package available
  • No volume-based charges: Scale from gigabytes to terabytes without additional costs
  • No per-GB, per-row, or per-API-call charges
  • Trial version available: Download and test as long as needed, license required for production
  • One license per legal entity (prevents reselling to multiple clients)

The Verdict

These two tools serve fundamentally different markets and use cases, making direct comparison somewhat like comparing apples and oranges.
dlt is the developer-friendly, open-source choice for teams that want maximum flexibility, Python-native workflows, and the ability to deploy anywhere. It's perfect for startups, data teams building custom pipelines, and organizations that value the open-source model. The lack of volume-based pricing (it's free!) and the extensive destination support make it ideal for experimentation and growth.
Dunwich is the enterprise-grade, on-premises solution for organizations that can't use cloud services due to regulatory, compliance, or data sovereignty requirements. The flat pricing model becomes incredibly attractive at scale - what traditional SaaS vendors might charge $100K+ for at high volumes, Dunwich delivers for a predictable flat fee. The GDPR-first architecture and privacy controls at the ingestion layer are unique selling points for regulated industries.

Key Differentiators

dlt wins on:
  • Open-source model and community
  • Python ecosystem integration
  • Flexibility and broad destination support
  • Local development workflow
  • Getting started quickly with zero cost
Dunwich wins on:
  • On-premises sovereignty and data residency
  • Privacy-by-design and GDPR compliance
  • Predictable costs at high volumes
  • Enterprise-grade support model
  • Regulatory compliance requirements

Final Recommendations

Choose dlt if you:
  • Need a proven, production-ready open-source solution
  • Want comprehensive documentation and active community support
  • Value Python-native development workflows
  • Require flexible deployment options (cloud, on-prem, serverless)
  • Need support for diverse data sources and destinations
  • Want to start with zero cost and scale as needed
  • Prefer to build and customize your own pipelines
Choose Dunwich if you:
  • Cannot use cloud services due to regulatory or compliance requirements
  • Need GDPR compliance built into your ingestion layer
  • Process high volumes where volume-based pricing would be prohibitive
  • Require complete on-premises data sovereignty
  • Work in regulated industries (healthcare, finance, government)
  • Want predictable costs regardless of data volume
  • Need to support Redshift, PostgreSQL, or MySQL/MariaDB
  • Value CDC integration via Debezium
  • Prefer a supported commercial product over DIY solutions
Consider Both if you:
  • Have hybrid requirements (some pipelines on-prem, others in cloud)
  • Want to evaluate open-source flexibility against commercial predictability
  • Are in a transition period between deployment models

The Bottom Line

For startups, data teams, and cloud-native organizations, dlt offers unbeatable value with its open-source model, Python-native approach, and flexible deployment. The zero upfront cost and vibrant community make it an easy choice for most modern data teams.
For enterprises in regulated industries with on-premises requirements, Dunwich provides a compelling value proposition. It's significantly cheaper than volume-based alternatives once you hit moderate scale, and the privacy-first architecture solves compliance challenges that other tools treat as afterthoughts.
The good news? These aren't really competitors - they serve different markets. You're not choosing between them based on features alone, but based on your organization's fundamental constraints around deployment model, compliance requirements, and cost structure.
Back to Blog