The Definitive Guide

Implementing LLMs:
Zero to Hero

From mathematical foundations to frontier techniques, build and understand large language models from scratch.

35Chapters

7Parts

676Exercises

Part I

Mathematical Foundations

Ch 1. Linear Algebra for ML Ch 2. Calculus & Optimization Ch 3. Probability & Statistics Ch 4. Information Theory Ch 5. Numerical Methods & Stability

Part II

Classical ML & Representations

Ch 6. Classical Machine Learning Ch 7. Word Embeddings & Representations Ch 8. Sequence Models & RNNs Ch 9. Language Model Fundamentals Ch 10. Tokenization

Part III

The Transformer

Ch 11. Attention Mechanisms Ch 12. The Transformer Architecture Ch 13. Positional Encoding Ch 14. Layer Normalization & Residuals Ch 15. Building a GPT from Scratch

Part IV

Pretraining at Scale

Ch 16. Scaling Laws Ch 17. Data Curation & Preprocessing Ch 18. Distributed Training Ch 19. Training Stability & Monitoring

Part V

Alignment & RLHF

Ch 21. Evaluation & Benchmarking Ch 22. Supervised Fine-Tuning Ch 23. Reinforcement Learning from Human Feedback Ch 24. Direct Preference Optimization Ch 25. Reasoning & Chain-of-Thought Ch 26. Safety & Constitutional AI

Part VI

Productionization

Ch 27. Parameter-Efficient Fine-Tuning Ch 28. Quantization & Compression Ch 29. Retrieval-Augmented Generation Ch 30. Multimodal Models Ch 31. Serving at Scale

Part VII

Frontier Techniques & Future

Ch 32. Mixture of Experts Ch 33. Long Context & Memory Ch 34. Agents & Multi-Agent Systems Ch 35. Open Problems & Future Directions