Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
1. Exploring the Landscape of Artificial Intelligence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
An Apology
The Real Introduction
What Is AI?
Motivating Examples
A Brief History of AI
Exciting Beginnings
The Cold and Dark Days
A Glimmer of Hope
How Deep Learning Became a Thing
Recipe for the Perfect Deep Learning Solution
Datasets
Model Architecture
Frameworks
Hardware
Responsible AI
Bias
Accountability and Explainability
Reproducibility
Robustness
Privacy
Summary
Frequently Asked Questions
2
3
3
5
6
6
8
8
12
15
16
18
20
23
26
27
30
30
31
31
32
32
iii
2. What’s in the Picture: Image Classification with Keras. . . . . . . . . . . . . . . . . . . . . . . . 35
Introducing Keras
Predicting an Image’s Category
Investigating the Model
ImageNet Dataset
Model Zoos
Class Activation Maps
Summary
36
37
42
42
44
46
48
3. Cats Versus Dogs: Transfer Learning in 30 Lines with Keras. . . . . . . . . . . . . . . . . . . . 49
Adapting Pretrained Models to New Tasks
A Shallow Dive into Convolutional Neural Networks
Transfer Learning
Fine Tuning
How Much to Fine Tune
Building a Custom Classifier in Keras with Transfer Learning
Organize the Data
Build the Data Pipeline
Number of Classes
Batch Size
Data Augmentation
Model Definition
Train the Model
Set Training Parameters
Start Training
Test the Model
Analyzing the Results
Further Reading
Summary
50
51
53
54
55
56
57
59
60
61
61
65
65
65
66
68
68
76
78
4. Building a Reverse Image Search Engine: Understanding Embeddings. . . . . . . . . . 79
Image Similarity
Feature Extraction
Similarity Search
Visualizing Image Clusters with t-SNE
Improving the Speed of Similarity Search
Length of Feature Vectors
Reducing Feature-Length with PCA
Scaling Similarity Search with Approximate Nearest Neighbors
Approximate Nearest-Neighbor Benchmark
iv
|
Table of Contents
80
83
86
90
94
94
96
100
101
Which Library Should I Use?
Creating a Synthetic Dataset
Brute Force
Annoy
NGT
Faiss
Improving Accuracy with Fine Tuning
Fine Tuning Without Fully Connected Layers
Siamese Networks for One-Shot Face Verification
Case Studies
Flickr
Pinterest
Celebrity Doppelgangers
Spotify
Image Captioning
Summary
102
102
102
103
104
104
104
108
109
110
111
111
112
113
114
116
5. From Novice to Master Predictor: Maximizing Convolutional
Neural Network Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Tools of the Trade
TensorFlow Datasets
TensorBoard
What-If Tool
tf-explain
Common Techniques for Machine Learning Experimentation
Data Inspection
Breaking the Data: Train, Validation, Test
Early Stopping
Reproducible Experiments
End-to-End Deep Learning Example Pipeline
Basic Transfer Learning Pipeline
Basic Custom Network Pipeline
How Hyperparameters Affect Accuracy
Transfer Learning Versus Training from Scratch
Effect of Number of Layers Fine-Tuned in Transfer Learning
Effect of Data Size on Transfer Learning
Effect of Learning Rate
Effect of Optimizers
Effect of Batch Size
Effect of Resizing
118
119
120
123
128
130
130
131
132
132
133
133
135
136
137
138
139
140
141
141
142
Table of Contents
|
v
Effect of Change in Aspect Ratio on Transfer Learning
Tools to Automate Tuning for Maximum Accuracy
Keras Tuner
AutoAugment
AutoKeras
Summary
143
144
144
146
147
148
6. Maximizing Speed and Performance of TensorFlow: A Handy Checklist. . . . . . . . . 149
GPU Starvation
nvidia-smi
TensorFlow Profiler + TensorBoard
How to Use This Checklist
Performance Checklist
Data Preparation
Data Reading
Data Augmentation
Training
Inference
Data Preparation
Store as TFRecords
Reduce Size of Input Data
Use TensorFlow Datasets
Data Reading
Use tf.data
Prefetch Data
Parallelize CPU Processing
Parallelize I/O and Processing
Enable Nondeterministic Ordering
Cache Data
Turn on Experimental Optimizations
Autotune Parameter Values
Data Augmentation
Use GPU for Augmentation
Training
Use Automatic Mixed Precision
Use Larger Batch Size
Use Multiples of Eight
Find the Optimal Learning Rate
Use tf.function
Overtrain, and Then Generalize
vi
|
Table of Contents
149
150
152
153
154
154
154
154
154
155
155
155
157
157
158
158
158
159
159
160
160
161
163
164
164
165
165
166
168
168
170
172
Install an Optimized Stack for the Hardware
Optimize the Number of Parallel CPU Threads
Use Better Hardware
Distribute Training
Examine Industry Benchmarks
Inference
Use an Efficient Model
Quantize the Model
Prune the Model
Use Fused Operations
Enable GPU Persistence
Summary
173
175
176
177
178
180
180
183
185
186
186
187
7. Practical Tools, Tips, and Tricks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Installation
Training
Model
Data
Privacy
Education and Exploration
One Last Question
189
191
192
193
196
197
198
8. Cloud APIs for Computer Vision: Up and Running in 15 Minutes. . . . . . . . . . . . . . . 201
The Landscape of Visual Recognition APIs
Clarifai
Microsoft Cognitive Services
Google Cloud Vision
Amazon Rekognition
IBM Watson Visual Recognition
Algorithmia
Comparing Visual Recognition APIs
Service Offerings
Cost
Accuracy
Bias
Getting Up and Running with Cloud APIs
Training Our Own Custom Classifier
Top Reasons Why Our Classifier Does Not Work Satisfactorily
Comparing Custom Classification APIs
Performance Tuning for Cloud APIs
203
203
204
204
205
206
208
209
209
210
211
212
217
219
224
225
228
Table of Contents
|
vii
Effect of Resizing on Image Labeling APIs
Effect of Compression on Image Labeling APIs
Effect of Compression on OCR APIs
Effect of Resizing on OCR APIs
Case Studies
The New York Times
Uber
Giphy
OmniEarth
Photobucket
Staples
InDro Robotics
Summary
228
229
230
230
231
231
232
233
234
234
235
235
237
9. Scalable Inference Serving on Cloud with TensorFlow Serving and KubeFlow. . . 239
Landscape of Serving AI Predictions
Flask: Build Your Own Server
Making a REST API with Flask
Deploying a Keras Model to Flask
Pros of Using Flask
Cons of Using Flask
Desirable Qualities in a Production-Level Serving System
High Availability
Scalability
Low Latency
Geographic Availability
Failure Handling
Monitoring
Model Versioning
A/B Testing
Support for Multiple Machine Learning Libraries
Google Cloud ML Engine: A Managed Cloud AI Serving Stack
Pros of Using Cloud ML Engine
Cons of Using Cloud ML Engine
Building a Classification API
TensorFlow Serving
Installation
KubeFlow
Pipelines
Fairing
viii
|
Table of Contents
240
242
242
243
244
244
245
245
245
246
246
247
247
248
248
248
248
249
249
249
256
256
258
260
260
Installation
Price Versus Performance Considerations
Cost Analysis of Inference-as-a-Service
Cost Analysis of Building Your Own Stack
Summary
261
263
263
265
266
10. AI in the Browser with TensorFlow.js and ml5.js. . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
JavaScript-Based Machine Learning Libraries: A Brief History
ConvNetJS
Keras.js
ONNX.js
TensorFlow.js
TensorFlow.js Architecture
Running Pretrained Models Using TensorFlow.js
Model Conversion for the Browser
Training in the Browser
Feature Extraction
Data Collection
Training
GPU Utilization
ml5.js
PoseNet
pix2pix
Benchmarking and Practical Considerations
Model Size
Inference Time
Case Studies
Semi-Conductor
TensorSpace
Metacar
Airbnb’s Photo Classification
GAN Lab
Summary
268
269
270
270
272
273
275
277
277
278
279
280
282
283
286
290
295
295
296
298
298
299
300
301
301
302
11. Real-Time Object Classification on iOS with Core ML. . . . . . . . . . . . . . . . . . . . . . . . . 303
The Development Life Cycle for Artificial Intelligence on Mobile
A Brief History of Core ML
Alternatives to Core ML
TensorFlow Lite
ML Kit
305
306
308
308
309
Table of Contents
|
ix
Fritz
Apple’s Machine Learning Architecture
Domain-Based Frameworks
ML Framework
ML Performance Primitives
Building a Real-Time Object Recognition App
Conversion to Core ML
Conversion from Keras
Conversion from TensorFlow
Dynamic Model Deployment
On-Device Training
Federated Learning
Performance Analysis
Benchmarking Models on iPhones
Measuring Energy Impact
Benchmarking Load
Reducing App Size
Avoid Bundling the Model
Use Quantization
Use Create ML
Case Studies
Magic Sudoku
Seeing AI
HomeCourt
InstaSaber + YoPuppet
Summary
309
309
310
311
311
312
319
319
319
321
322
323
323
324
327
331
333
334
334
336
336
336
338
338
339
342
12. Not Hotdog on iOS with Core ML and Create ML. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
Collecting Data
Approach 1: Find or Collect a Dataset
Approach 2: Fatkun Chrome Browser Extension
Approach 3: Web Scraper Using Bing Image Search API
Training Our Model
Approach 1: Use Web UI-based Tools
Approach 2: Use Create ML
Approach 3: Fine Tuning Using Keras
Model Conversion Using Core ML Tools
Building the iOS App
Further Exploration
Summary
x
|
Table of Contents
345
345
346
349
350
350
355
361
361
361
363
363
13. Shazam for Food: Developing Android Apps with TensorFlow Lite and ML Kit. . . 365
The Life Cycle of a Food Classifier App
An Overview of TensorFlow Lite
TensorFlow Lite Architecture
Model Conversion to TensorFlow Lite
Building a Real-Time Object Recognition App
ML Kit + Firebase
Object Classification in ML Kit
Custom Models in ML Kit
Hosted Models
A/B Testing Hosted Models
Using the Experiment in Code
TensorFlow Lite on iOS
Performance Optimizations
Quantizing with TensorFlow Lite Converter
TensorFlow Model Optimization Toolkit
Fritz
A Holistic Look at the Mobile AI App Development Cycle
How Do I Collect Initial Data?
How Do I Label My Data?
How Do I Train My Model?
How Do I Convert the Model to a Mobile-Friendly Format?
How Do I Make my Model Performant?
How Do I Build a Great UX for My Users?
How Do I Make the Model Available to My Users?
How Do I Measure the Success of My Model?
How Do I Improve My Model?
How Do I Update the Model on My Users’ Phones?
The Self-Evolving Model
Case Studies
Lose It!
Portrait Mode on Pixel 3 Phones
Speaker Recognition by Alibaba
Face Contours in ML Kit
Real-Time Video Segmentation in YouTube Stories
Summary
366
368
371
372
373
382
384
384
385
391
397
397
397
398
398
399
402
402
403
403
403
404
404
404
405
405
406
406
408
408
410
411
411
412
413
14. Building the Purrfect Cat Locator App with TensorFlow Object Detection API. . . . 415
Types of Computer-Vision Tasks
Classification
417
417
Table of Contents
|
xi
Localization
Detection
Segmentation
Approaches to Object Detection
Invoking Prebuilt Cloud-Based Object Detection APIs
Reusing a Pretrained Model
Obtaining the Model
Test Driving Our Model
Deploying to a Device
Building a Custom Detector Without Any Code
The Evolution of Object Detection
Performance Considerations
Key Terms in Object Detection
Intersection over Union
Mean Average Precision
Non-Maximum Suppression
Using the TensorFlow Object Detection API to Build Custom Models
Data Collection
Labeling the Data
Preprocessing the Data
Inspecting the Model
Training
Model Conversion
Image Segmentation
Case Studies
Smart Refrigerator
Crowd Counting
Face Detection in Seeing AI
Autonomous Cars
Summary
417
417
418
420
421
423
423
424
425
427
432
433
435
435
436
436
437
437
441
445
446
448
450
452
453
453
454
456
457
458
15. Becoming a Maker: Exploring Embedded AI at the Edge. . . . . . . . . . . . . . . . . . . . . . 459
Exploring the Landscape of Embedded AI Devices
Raspberry Pi
Intel Movidius Neural Compute Stick
Google Coral USB Accelerator
NVIDIA Jetson Nano
FPGA + PYNQ
Arduino
A Qualitative Comparison of Embedded AI Devices
xii
|
Table of Contents
460
462
464
465
466
468
472
474
Hands-On with the Raspberry Pi
Speeding Up with the Google Coral USB Accelerator
Port to NVIDIA Jetson Nano
Comparing the Performance of Edge Devices
Case Studies
JetBot
Squatting for Metro Tickets
Cucumber Sorter
Further Exploration
Summary
476
478
480
483
484
484
486
487
488
489
16. Simulating a Self-Driving Car Using End-to-End Deep Learning with Keras. . . . . . 491
A Brief History of Autonomous Driving
Deep Learning, Autonomous Driving, and the Data Problem
The “Hello, World!” of Autonomous Driving: Steering Through a
Simulated Environment
Setup and Requirements
Data Exploration and Preparation
Identifying the Region of Interest
Data Augmentation
Dataset Imbalance and Driving Strategies
Training Our Autonomous Driving Model
Drive Data Generator
Model Definition
Deploying Our Autonomous Driving Model
Further Exploration
Expanding Our Dataset
Training on Sequential Data
Reinforcement Learning
Summary
492
493
496
496
498
501
503
504
509
510
512
518
521
522
522
523
523
17. Building an Autonomous Car in Under an Hour: Reinforcement Learning with
AWS DeepRacer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
A Brief Introduction to Reinforcement Learning
Why Learn Reinforcement Learning with an Autonomous Car?
Practical Deep Reinforcement Learning with DeepRacer
Building Our First Reinforcement Learning
Step 1: Create Model
Step 2: Configure Training
Step 3: Model Training
526
527
529
532
533
534
541
Table of Contents
|
xiii
Step 4: Evaluating the Performance of the Model
Reinforcement Learning in Action
How Does a Reinforcement Learning System Learn?
Reinforcement Learning Theory
Reinforcement Learning Algorithm in AWS DeepRacer
Deep Reinforcement Learning Summary with DeepRacer as an Example
Step 5: Improving Reinforcement Learning Models
Racing the AWS DeepRacer Car
Building the Track
AWS DeepRacer Single-Turn Track Template
Running the Model on AWS DeepRacer
Driving the AWS DeepRacer Vehicle Autonomously
Further Exploration
DeepRacer League
Advanced AWS DeepRacer
AI Driving Olympics
DIY Robocars
Roborace
Summary
542
544
544
548
551
552
553
558
558
559
559
560
563
563
563
564
564
565
566
A. A Crash Course in Convolutional Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . 567
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
xiv
|
Table of Contents