Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
1. Introduction to Building AI Applications with Foundation Models. . . . . . . . . . . . . . . . . . 1
The Rise of AI Engineering
From Language Models to Large Language Models
From Large Language Models to Foundation Models
From Foundation Models to AI Engineering
Foundation Model Use Cases
Coding
Image and Video Production
Writing
Education
Conversational Bots
Information Aggregation
Data Organization
Workflow Automation
Planning AI Applications
Use Case Evaluation
Setting Expectations
Milestone Planning
Maintenance
The AI Engineering Stack
Three Layers of the AI Stack
AI Engineering Versus ML Engineering
AI Engineering Versus Full-Stack Engineering
Summary
2
2
8
12
16
20
22
22
24
26
26
27
28
28
29
32
33
34
35
37
39
46
47
iii
2. Understanding Foundation Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Training Data
Multilingual Models
Domain-Specific Models
Modeling
Model Architecture
Model Size
Post-Training
Supervised Finetuning
Preference Finetuning
Sampling
Sampling Fundamentals
Sampling Strategies
Test Time Compute
Structured Outputs
The Probabilistic Nature of AI
Summary
50
51
56
58
58
67
78
80
83
88
88
90
96
99
105
111
3. Evaluation Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Challenges of Evaluating Foundation Models
Understanding Language Modeling Metrics
Entropy
Cross Entropy
Bits-per-Character and Bits-per-Byte
Perplexity
Perplexity Interpretation and Use Cases
Exact Evaluation
Functional Correctness
Similarity Measurements Against Reference Data
Introduction to Embedding
AI as a Judge
Why AI as a Judge?
How to Use AI as a Judge
Limitations of AI as a Judge
What Models Can Act as Judges?
Ranking Models with Comparative Evaluation
Challenges of Comparative Evaluation
The Future of Comparative Evaluation
Summary
iv
|
Table of Contents
114
118
119
120
121
121
122
125
126
127
134
136
137
138
141
145
148
152
155
156
4. Evaluate AI Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Evaluation Criteria
Domain-Specific Capability
Generation Capability
Instruction-Following Capability
Cost and Latency
Model Selection
Model Selection Workflow
Model Build Versus Buy
Navigate Public Benchmarks
Design Your Evaluation Pipeline
Step 1. Evaluate All Components in a System
Step 2. Create an Evaluation Guideline
Step 3. Define Evaluation Methods and Data
Summary
160
161
163
172
177
179
179
181
191
200
200
202
204
208
5. Prompt Engineering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Introduction to Prompting
In-Context Learning: Zero-Shot and Few-Shot
System Prompt and User Prompt
Context Length and Context Efficiency
Prompt Engineering Best Practices
Write Clear and Explicit Instructions
Provide Sufficient Context
Break Complex Tasks into Simpler Subtasks
Give the Model Time to Think
Iterate on Your Prompts
Evaluate Prompt Engineering Tools
Organize and Version Prompts
Defensive Prompt Engineering
Proprietary Prompts and Reverse Prompt Engineering
Jailbreaking and Prompt Injection
Information Extraction
Defenses Against Prompt Attacks
Summary
212
213
215
218
220
220
223
224
227
229
230
233
235
236
238
243
248
251
6. RAG and Agents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
RAG
RAG Architecture
Retrieval Algorithms
Retrieval Optimization
253
256
257
268
Table of Contents
|
v
RAG Beyond Texts
Agents
Agent Overview
Tools
Planning
Agent Failure Modes and Evaluation
Memory
Summary
273
275
276
278
281
298
300
305
7. Finetuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Finetuning Overview
When to Finetune
Reasons to Finetune
Reasons Not to Finetune
Finetuning and RAG
Memory Bottlenecks
Backpropagation and Trainable Parameters
Memory Math
Numerical Representations
Quantization
Finetuning Techniques
Parameter-Efficient Finetuning
Model Merging and Multi-Task Finetuning
Finetuning Tactics
Summary
308
311
311
312
316
319
320
322
325
328
332
333
347
357
361
8. Dataset Engineering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
Data Curation
Data Quality
Data Coverage
Data Quantity
Data Acquisition and Annotation
Data Augmentation and Synthesis
Why Data Synthesis
Traditional Data Synthesis Techniques
AI-Powered Data Synthesis
Model Distillation
Data Processing
Inspect Data
Deduplicate Data
Clean and Filter Data
vi
|
Table of Contents
365
368
370
372
377
380
381
383
386
395
396
397
399
401
Format Data
Summary
401
403
9. Inference Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
Understanding Inference Optimization
Inference Overview
Inference Performance Metrics
AI Accelerators
Inference Optimization
Model Optimization
Inference Service Optimization
Summary
406
406
412
419
426
426
440
447
10. AI Engineering Architecture and User Feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
AI Engineering Architecture
Step 1. Enhance Context
Step 2. Put in Guardrails
Step 3. Add Model Router and Gateway
Step 4. Reduce Latency with Caches
Step 5. Add Agent Patterns
Monitoring and Observability
AI Pipeline Orchestration
User Feedback
Extracting Conversational Feedback
Feedback Design
Feedback Limitations
Summary
449
450
451
456
460
463
465
472
474
475
480
490
492
Epilogue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
Table of Contents
|
vii
0
