Designing Data-Intensive Applications by Martin Kleppmann
A reference for the data and reliability decisions behind a complete product
- PostgreSQL
- Redis
- SQL
Connecting system behavior to the experience users see
The choices behind a database write or a background task often become visible in the interface. DDIA helps me reason about those choices and explain their trade-offs. These are my notes on the chapters I keep returning to while building features across APIs, data, and user-facing workflows.
The overview
Designing Data-Intensive Applications - DDIA, or 'the wild boar book' after its O'Reilly cover animal - is Martin Kleppmann's now-classic tour of how modern data systems actually work underneath the marketing. It covers reliability, scalability, maintainability, storage engines, encoding and evolution, replication, partitioning, transactions, consistency and consensus, batch and stream processing, and closes with a chapter on the future of data systems. The first edition shipped in 2017 and the heavily updated second edition - co-authored with Chris Riccomini - is now in O'Reilly's Early Release programme, folding in a decade of hard-won lessons from the cloud, streaming, and now AI era. It is roughly 600 pages, dense but unusually readable, and it is probably the single most recommended book in backend engineering.
Why I picked it up now
Working across the interface and backend makes data decisions visible in a different way. A delayed update becomes a confusing screen; a retry becomes a duplicate action; a migration can change what an existing workflow means. I return to DDIA when I need to reason about those boundaries. I read the chapters that match the problem in front of me rather than treating the book as something to finish once.
The author and why he is worth reading
Martin Kleppmann is a researcher at the University of Cambridge and previously an engineer at LinkedIn (where he worked on Samza and Kafka-adjacent infrastructure) and before that at Rapportive and Go Test It. He does not write from the position of a vendor evangelist - he writes from the position of someone who has read the papers, built the systems, and then spent years translating between those two worlds. The second-edition Early Release adds Chris Riccomini, formerly of WePay and LinkedIn, who brings a modern take on data platforms, lakehouses, and the cloud-native reality most of us now ship into. The combined voice is exactly what you want - academic rigor plus production scar tissue.
- Martin Kleppmann - Cambridge researcher, ex-LinkedIn distributed-systems engineer
- Chris Riccomini - ex-LinkedIn, ex-WePay, data platforms author and co-author of the 2nd edition
- The book credits an enormous cast of reviewers - this is one of the most peer-reviewed tech books in the last decade
- Website: dataintensive.net - with the famous map of distributed-systems terrain on the cover
Structure - three parts, each worth the price
The book is split into three big parts and a conclusion. Part I - Foundations of Data Systems - sets the conceptual baseline: what reliability, scalability and maintainability actually mean, how storage engines work, how encoding formats evolve, and how data models map to queries. Part II - Distributed Data - is the hard middle: replication, partitioning/sharding, transactions, the limits of distributed systems, consistency and consensus. Part III - Derived Data - unifies batch and stream processing into a single mental model (every database is a kind of cache of a stream of events) and closes on a genuinely philosophical chapter on the future of data. The Early Release of the second edition threads cloud-native architectures, lakehouses, and AI-assisted systems through all three parts, but the bones are the same.
- Part I - Foundations - reliability/scalability/maintainability, storage engines, encoding, data models
- Part II - Distributed Data - replication, partitioning, transactions, consistency, consensus
- Part III - Derived Data - batch processing, stream processing, the unifying view
- Conclusion - 'the future of data systems' - the most opinionated chapter and the one I reread most
Why it belongs in a full-stack engineer’s reference shelf
The interface depends on guarantees the rest of the system provides. Can a user see a change immediately after saving it? What happens if a request times out after a write succeeds? Can a background task run twice? DDIA gives me a vocabulary for asking those questions and comparing the costs of different answers. Its backend focus is valuable precisely because those choices reach all the way to the user experience.
- Connect data consistency to what the interface promises
- Discuss failure and retry behavior before adding a background workflow
- Compare complexity with the actual needs of the application
- Use the references to investigate a decision more deeply
The reliability/scalability/maintainability chapter is worth the price on its own
Before any databases or distributed systems, the book spends the first chapter defining three words that get thrown around constantly: reliable, scalable, maintainable. Reliable means 'continues to work correctly, even when things go wrong' - and Kleppmann is careful to distinguish faults from failures, and hardware faults from software faults from human error. Scalable is not a property a system has - it is a conversation about load parameters, throughput, and tail latency percentiles (you will never write p95 casually again after this chapter). Maintainable is split into operability, simplicity and evolvability. If someone read only Chapter 1 of DDIA and never went further, they would already argue about systems more precisely than most engineers do. It is the vocabulary chapter, and the whole rest of the book depends on it.
- Reliability - fault vs failure, fault-tolerance as a property you design for, not hope for
- Scalability - 'load parameters' and tail latency, not a hand-wavy adjective
- Maintainability - operability, simplicity (managing complexity), evolvability
- These three words become the axes you compare every architecture on for the rest of your career
Concepts I now see in the systems I build
Django, PostgreSQL, Celery, and Redis bring these questions into ordinary application work. A form submission may write data and start a task; an export may combine records that changed at different times; a retried upload may need to return an existing result. The book helps me separate the intended behavior from an accidental assumption about timing or delivery. I do not need to run a large distributed platform to benefit from that distinction.
- Log-structured storage - the mental model behind every append-only pipeline I have written
- Outbox pattern - the only sane way to keep a DB write and a published event in sync
- Idempotency and effectively-once - the thing that actually saves you, not 'exactly-once delivery'
- Read-your-writes and monotonic reads - the anomaly classes your users will report as 'it is broken'
- CDC / change data capture - how to treat a database as a stream without bolting on a second system
- Lambda / Kappa (now just 'unified batch+stream') - the shape most modern data pipelines converge on
Architecture decisions remain part of the job
AI-assisted implementation makes it especially useful to state the guarantees before generating the code. I need to explain which data must change together, how duplicate requests should behave, and what a user should see after a partial failure. Those requirements give both implementation and review something concrete to work against. The patterns in this book help me express the decisions, but I still need to check whether the application actually provides the intended behavior.
- State the data invariants before implementing the workflow
- Make retry and failure behavior explicit
- Review the database write and its external effects together
- Check what the interface shows while background work is incomplete
The diagrams and the map on the cover
DDIA is not as visually lush as Shvets's patterns book, but it has its own style. Every chapter opens with a fragment of a hand-drawn map of the 'distributed systems landscape' - the wild boar book cover extended inward - and every tricky concept gets a clean, minimal diagram. Replication topologies, partitioning strategies, serializable isolation levels, the happens-before relation - these are the kinds of things that live or die on a good diagram, and Kleppmann's are excellent. The map metaphor matters too: the book is explicit that the point is to give you a *map* of the terrain, not a set of directions, so that when a new database or stream processor shows up in five years you can place it somewhere you already understand.
How I am using it day to day
Honest truth - I am not reading DDIA cover to cover, and I do not think it wants to be read that way. It lives as an EPUB in my BookFusion library - usually on my trusty Boox Palma via the BookFusion Android app, sometimes on my phone, and straight in the browser when I am at the desk on Linux - and I treat it as a reference I pull from when the question matches a chapter. Before a non-trivial backend feature I read the relevant chapter - the transactions chapter before anything touching money or inventory, the replication chapter before a read-replica rollout, the stream processing chapter before I design a new pipeline. After a PR that touches data semantics I re-read the consistency sections to double-check which anomaly class I am paying for. The 'Summary' and 'Further Reading' sections at the end of each chapter are the densest parts - I often open those first to remember which trade-offs I should be thinking about before I even open the code.
- Read the relevant chapter *before* a feature, not after the outage
- Use the consistency chapter as a checklist during PR review on replicated systems
- Treat Chapter 1 (reliability/scalability/maintainability) as the vocabulary for all system-design conversations
- Open the 'Further Reading' list whenever a new datastore shows up - it places it on the map
What I would watch out for
Two caveats. First, DDIA is genuinely dense - this is a textbook, not a beach read, and certain sections (consensus, serializable isolation) reward a second and third pass with a notebook open. If you expect the ergonomics of a blog post, you will bounce. Second, the book is paradigm-broad but still biased toward the data world Kleppmann came out of - LinkedIn-era Kafka, log-structured everything, event-centric architectures. That bias is mostly a feature in 2025 (the industry bent his way) but if you work in a domain where request/response and a single Postgres are genuinely enough, some of the middle of the book will read as 'problems you do not have yet.' Read it anyway - the foundations chapters and the final 'future of data systems' chapter travel perfectly across contexts.
Who it is for
The book is useful to engineers responsible for the behavior of an application, including full-stack developers whose work crosses user interfaces, APIs, and background jobs. It rewards some experience with real data and failure cases. I use it to deepen the backend foundation of my work and to explain trade-offs more clearly when a feature spans several parts of the system.
Summary
DDIA is a reference I return to when a feature raises questions about data, consistency, or failure. I have worked through the parts that match my projects rather than reading every page in order. Its value in my full-stack work is the connection between a technical guarantee and the experience the product can honestly promise to its users.