Hands-On Large Language Models by Jay Alammar & Maarten Grootendorst

The overview

Hands-On Large Language Models is the O'Reilly book by Jay Alammar - the man behind 'The Illustrated Transformer' that taught most of the industry what a transformer actually is - and Maarten Grootendorst, the author of BERTopic. It is a visually-led, intuition-first tour of the entire language AI stack: tokens, embeddings, the transformer block, classification, clustering, prompt engineering, semantic search and RAG, multimodal models, and finally how these models are trained and fine-tuned. It shipped in late 2024 with O'Reilly, runs roughly 400 pages, and is genuinely one of the most beautifully illustrated technical books I own. If DDIA is the textbook for system design, this is the textbook for what is happening inside the model you are prompting all day.

• llm-book.com
• oreilly.com/hands-on-large-language-models
• jalammar.github.io/illustrated-transformer

Why I picked it up now

AI shows up in my work in two different ways: as coding assistance and as a feature inside an application. In both cases, understanding the model’s inputs and limitations helps me ask better questions. I wanted a clearer foundation for tokens, embeddings, retrieval, and multimodal models so I could reason about an integration rather than treating every unexpected output as a prompting problem.

The authors and why they are worth reading

Jay Alammar is a Director and Engineering Fellow at Cohere and the author of a series of blog posts ('The Illustrated Transformer', 'The Illustrated GPT-2', 'The Illustrated BERT, ELMo, and co.') that have collectively been the on-ramp into modern NLP for an entire generation of engineers. If you have ever seen those clean diagrams of attention heads and embedding lookups making the rounds on Twitter, that is him. Maarten Grootendorst is the author of BERTopic and KeyBERT - hugely popular open-source libraries for topic modeling and keyword extraction - and writes the 'Exploring Language Models' newsletter. Together they bring exactly the right pairing: Alammar's gift for visual explanation and Grootendorst's hands-on toolkit for actually using language models on real data.

• Jay Alammar - Cohere, illustrated-blog series, the visual-language explainer of modern NLP
• Maarten Grootendorst - creator of BERTopic and KeyBERT, practical NLP at scale
• Foreword by Andrew Ng - the book is endorsed by basically everyone in the field
• Companion site llm-book.com with all code on GitHub and runnable Google Colab notebooks

Structure - three parts, intuition before code

The book is split into three parts. Part I - Understanding Language Models - is the conceptual base: a recent history of language AI from bag-of-words to transformers, then the two central building blocks (tokens and embeddings) and finally a deep look inside the transformer architecture itself, essentially an updated, expanded version of Jay's famous 'Illustrated Transformer' blog post. Part II - Using Pretrained Language Models - is the hands-on middle: text classification, clustering and topic modeling, prompt engineering, advanced text generation with chains/memory/agents, semantic search and RAG, and a closing chapter on multimodal models. Part III - Training and Fine-Tuning - covers creating embedding models, fine-tuning representation models for classification, and fine-tuning generative models with techniques like LoRA and DPO. Each chapter is independent enough that you can skip around, which is exactly how I read it.

• Part I - Understanding - history of language AI, tokens, embeddings, the transformer block
• Part II - Using pretrained models - classification, clustering, prompting, RAG, multimodal
• Part III - Training and fine-tuning - embedding models, LoRA, DPO, preference tuning
• Companion code in Colab notebooks, all linked from llm-book.com

The intuition-first philosophy

The book opens by stating its philosophy outright: build intuition first, derive equations later. The pace of the field is brutal - papers obsolete each other in months - so chasing the latest model is a losing game. What does not change is the *shape* of the ideas: a token is a chunk of text the model actually sees, an embedding is a vector that captures meaning, attention is how the model decides which other tokens matter for the current one. Once those shapes are solid you can read any new paper, any new model card, any new framework, and slot it into a mental map. That is the book's real value. Alammar is famous for this style and it shows on every page - illustrations carry the weight, prose explains them, and code at the end shows you how to actually run it on your laptop or in Colab.

Tokens and embeddings - the chapter that pays for the book

If I could only keep one chapter from this book, it would be the one on tokens and embeddings. Tokens are the unit the model actually thinks in - not characters, not words, but subword chunks chosen by a tokenizer like BPE or SentencePiece. Once you have seen the same sentence tokenized by GPT-2 vs LLaMA vs a multilingual model, you understand instantly why prompts behave differently across providers, why some non-English languages cost dramatically more tokens, why typos and weird Unicode can wreck a system, and why 'token efficiency' is a real engineering metric, not a buzzword. Embeddings, then, are the geometry the model uses to reason: every token is a point in a high-dimensional space, and 'meaning' is *direction* and *distance* in that space. Word2vec, the king/queen analogy, contrastive training - this is where it all clicks. After this chapter, the phrase 'vector database' stops being magic and becomes obvious.

• Tokens are subword chunks, not words - the tokenizer is part of the model's identity
• Different tokenizers turn the *same* prompt into very different inputs - and very different bills
• Embeddings are vectors that encode meaning - similarity is just geometry
• Token embeddings vs contextualized embeddings vs sentence/document embeddings - three different jobs
• Word2vec's training trick (predict the neighbour) is the same trick still powering modern embedding models

Inside the transformer - finally a diagram I can hold in my head

Chapter 3 is the upgraded 'Illustrated Transformer.' It walks the forward pass through a generative LLM end-to-end: tokenizer turns text into IDs, an embedding lookup turns IDs into vectors, a stack of transformer blocks (each one a self-attention layer plus a feed-forward layer) refines those vectors, and a final linear+softmax head converts the last vector back into a probability distribution over the next token. Then the model picks one (greedy, top-k, top-p, temperature - the chapter walks through each), appends it, and runs the whole thing again. The KV cache, parallel processing, RoPE positional embeddings, and recent improvements to attention are all explained with the same visual style. The single most useful realization for me: the model does not 'know' the future, it just predicts one token at a time, and *every* trick we use - chain-of-thought, agents, structured output - is some way of getting it to take more useful steps in that one-token-at-a-time loop.

Embeddings are the secret protagonist of every AI pipeline

Half of the book is, in some sense, an extended argument for taking embeddings seriously. Classification with embeddings, clustering with embeddings, topic modeling with embeddings (BERTopic, Maarten's own library), semantic search with embeddings, RAG with embeddings, recommendation systems with embeddings - the same primitive, used six different ways. Once you internalize that an embedding is just 'meaning as coordinates,' a huge amount of AI engineering becomes legible. A semantic search system is: embed your documents, embed the query, return the closest documents by cosine distance. A RAG system is the same thing with an LLM stitched on the end. A clustering pipeline is the same thing with UMAP for dimensionality reduction and HDBSCAN for grouping. The chapter on text clustering and topic modeling builds exactly that pipeline step by step, and it is the cleanest 'this is how a real pipeline is shaped' explanation I have read.

• Classification - embed text, train a small head, done (no fine-tuning needed)
• Clustering / topic modeling - embed → reduce dimensions → cluster → label clusters with an LLM
• Semantic search - embed corpus once, embed query at runtime, return nearest by cosine
• RAG - semantic search + an LLM that grounds its answer in what was retrieved
• Reranking - use a slower but smarter model to reorder the top-k from the first stage

Multimodal embedding models - where this gets really interesting

Chapter 9 is the one I keep going back to. It explains how vision transformers work (an image is just a grid of patches, each patch is a token, attention does the rest), then walks through CLIP - the model that, by training on hundreds of millions of (image, caption) pairs with a contrastive loss, ends up with a *single shared embedding space for text and images*. That is the unlock. A photo of a dog and the string 'a dog' can land near each other in the same vector space. Once you have that, image search by text query, text search by image, zero-shot classification on arbitrary categories, and image-grounded chat all become variations on the same idea. The chapter then walks through BLIP-2 and how a frozen vision encoder can be glued to a frozen LLM with a small trainable bridge, which is the recipe most modern multimodal models still riff on. This is the chapter that turned 'multimodal' from a marketing word into a thing I can actually build with.

• Vision transformers - an image is a sequence of patches, treated like tokens
• CLIP - text and images embedded into one shared space via contrastive training
• Zero-shot image classification - just embed the candidate labels and pick the closest
• BLIP-2 - bridge a frozen vision encoder to a frozen LLM with a tiny trainable adapter
• Multimodal RAG - retrieve images by text, retrieve text by image, ground generation in both

Prompt engineering, chains, and agents - with the model demystified underneath

Chapters 6 and 7 cover prompt engineering and the agentic patterns built on top: in-context learning (showing examples), chain-of-thought (think before you answer), self-consistency (sample multiple answers and vote), tree-of-thought (explore branches), constrained decoding (force the output into a grammar), chains (LangChain-style multi-step pipelines), memory (buffer, windowed, summary), and finally ReAct-style agents that interleave reasoning and tool calls. None of this is novel if you have been using Claude Code or building agentic workflows for a while - but seeing it laid out *after* the chapters on tokens and the transformer block changes the texture of it. Chain-of-thought is not a magic spell, it is more useful tokens to attend to. An agent is not a separate thing, it is a generation loop with tools wired into the decoding step. Once you see it that way, you stop treating prompts as incantations and start designing them like the small programs they actually are.

How this connects to coding assistance

The book gave me a better vocabulary for thinking about the information a coding assistant receives. I try to provide relevant files, a clear task, and specific constraints, then review the changes against the intended behavior. Understanding language models helps explain some of their limitations, but it does not tell me that a particular generated change is correct. That still needs code review and verification in the application.

• Keep the task and relevant project context clear
• Distinguish a plausible response from a verified implementation
• Use model knowledge to investigate failures, not to assume correctness

Connecting model behavior to a useful feature

An AI integration includes the data it receives, the retrieval or processing around it, the response contract, and the interface where someone uses the result. The book helps me reason about the model-related parts of that flow. In a product, I also need validation, understandable failure states, and a way to assess whether the output is useful. Those responsibilities sit alongside the model choice and prompt.

• Inspect the source data and retrieved context when output is poor
• Choose compatible representations for the task
• Validate structured output before using it elsewhere in the application
• Evaluate the feature through the workflow it is intended to support

The illustrations and the visual style

This is the most beautifully illustrated technical book I own, full stop. There are over 250 custom diagrams - tokenizers, attention patterns, embedding spaces, transformer blocks, RAG pipelines, multimodal architectures - all in Alammar's now-iconic clean style. The diagrams are not decoration, they are the explanation; the prose is often a caption to the figure rather than the other way around. If you have ever read 'The Illustrated Transformer' on his blog, you already know the aesthetic - this book is the same visual language extended over 400 pages. For someone like me who reads on a Boox Palma in greyscale, the diagrams still work, which is a real testament to how carefully they were designed. On a phone or tablet in colour they are genuinely a pleasure.

How I am using it day to day

Same playbook as DDIA - this lives as an EPUB in my BookFusion library, on the Boox Palma when I am away from the desk and in the browser when I am at it, and I read chapters when the question matches them. Before building a new RAG-shaped feature I re-read the semantic search chapter. Before touching anything multimodal I re-read Chapter 9. When I am writing a tricky prompt for Claude Code I open the prompt engineering chapter and steal the structure. When something in a pipeline breaks, I trace it back to the relevant primitive (tokens, embeddings, attention, decoding) and ask myself what I was assuming about that primitive that turned out to be wrong. The Colab notebooks on llm-book.com are perfect for this kind of 'I just want to verify what this actually does' moment - I can run a tokenizer comparison, an embedding similarity check, or a small RAG demo in a browser tab and confirm my mental model before going back to the real code.

• Read the relevant chapter *before* the feature, not after the bug
• Use the Colab notebooks as a sandbox for sanity-checking assumptions
• Open the prompt engineering chapter when a long prompt is misbehaving
• Open the multimodal chapter before writing a single line of vision-related code
• Treat the tokens/embeddings chapters as the vocabulary chapter the rest of your AI work depends on

What I would watch out for

Two honest caveats. First, the field moves so fast that any book about it is partially out of date the day it ships - by the time I read it, several model names in the examples were already two generations behind. That is fine, because the *concepts* are the point and they have not changed, but if you are looking for the latest benchmark numbers, go elsewhere. Second, this is fundamentally a Python/PyTorch/HuggingFace book. If you live entirely in TypeScript and the Vercel AI SDK and have no intention of touching a notebook, you will still get enormous value from the conceptual chapters but you will skip half the code examples. That is what I did, and I do not regret it - the diagrams alone are worth the price.

Who it is for

I would see this as a foundation for developers who want to understand the language-model concepts behind the tools they use or the features they build. For my full-stack work, the useful bridge is between a model’s behavior and the application around it. The conceptual chapters remain useful even when a particular library or model in an example differs from the one in a project.

Summary

The illustrations and worked examples helped me connect tokens, embeddings, retrieval, and generation into a clearer picture. I use the book as a reference around specific questions rather than a promise that one model can solve every task. Its practical value is a better foundation for deciding how an AI feature should fit into an application and what needs to be checked around it.

• llm-book.com
• jalammar.github.io
• maartengrootendorst.com
• oreilly.com/hands-on-large-language-models