Analytics for IoT:
From Sensors to
Decisions
Tom Dietterich
Distinguished Professor
Oregon State University
1
The Usual View of IoT
Networks
Transducer
Array
Receive Beamformer Channels
420 µm
1000 µm
TS
∆Σ-M
Systems
TS
550 µm
TS
∆Σ-M
N
Channels
650 µm
TS
Σ
DSP
Processing
Processing
Elements
Receiver
TS
∆Σ-M
TGA
Transmitter
TS
Variable Length Delay Line
Devices
Materials
2
Sensors
123RF Limited
3
Spatio-Temporal Analytics
Hierarchy
Models
Events &
Activities
Trajectories
State
Variables
Cleaned
Data
Sensors
123RF Limited
4
Leadership in Analytics
Machine Learning, Data Mining, Data Science
Hector Cotilla-Sanchez
Tom Dietterich
Alan Fern
Xiaoli Fern
Thinh Nguyen
Raviv Raich
Scott Sanner (joining April 1)
Prasad Tadepalli
Arash Termehchy
Sinisa Todorovic
Weng-Keen Wong
5
Step 1: Data Cleaning and
Imputation
Broken
Sun
Shield
Cleaned
Data
o Detect bad data values and
broken sensors
o Interpolate good data values
as needed
Sensors
6
Step 2: Estimate State Variables
o
o
o
o
State
Variables
Location of each customer
Location of each employee
Total time customer in store
# of items
Cleaned
Data
Sensors
123RF Limited
7
Step 3: Record trajectories
Trajectories
o Trajectory of each customer
o Trajectory of each employee
o Trajectory of instrumented item
State
Variables
Cleaned
Data
Sensors
123RF Limited
8
Step 4: Event and Activity
Recognition
Events &
Activities
o
o
o
o
o
o
o
Trajectories
State
Variables
Cleaned
Data
Customer enters/exits store
Employee enters/exits store
Customer waiting at help desk
Customer is looking for item
Customer picks up item
Customer puts item in basket
Customer leaves store with
unpaid merchandise
Sensors
123RF Limited
9
Step 5: Predictive Models
Models
Events &
Activities
Trajectories
o Predict customer demand
per item
o Predict customer traffic
o Predict supplier delays
o Predict wholesale and retail
price trends
State
Variables
Cleaned
Data
Sensors
123RF Limited
10
Bridging from Sensing to Action
Events &
Activities
Trajectories
State
Variables
Cleaned
Data
Alert
Sensor Needs
Repair
Sensors
123RF Limited
11
Bridging from Sensing to Action
Events &
Activities
Trajectories
State
Variables
Enter Store
DB Update:
Increment Visit
Count
Cleaned
Data
Sensors
123RF Limited
12
Bridging from Sensing to Action
Events &
Activities
Trajectories
Alert: Employee
to greet
customer
State
Variables
Alert: Cashier
needed
Cleaned
Data
Alert: Employee
to help
customer find
item
Alert: Possible
shoplifting
Coupon Offer to
cell phone
Sensors
123RF Limited
13
Offline Analytics
Store Layout
 pinch points
 hot spots
 dead spots
Employee training
 conversion rate
 customer satisfaction
Inventory and staffing
 missed sales because
customer could not find item
 missed sales because out
of stock
 predictive inventory
management
 predictive staffing
 what predicts customer time
in store?
14
Beyond Retail:
Personalized Medicine
Actions
Medical devices
Smart walkers; Exoskeletons
Models
Normal state variables, gait
Normal events and activities
Events &
Activities
Unsteadiness, falls, breathing difficulties
heart attack, stroke
Running, walking, climbing stairs
Trajectories
Trajectory travelled
Glucose history
State
Variables
Patient state (glucose, heart rate, EKG)
Current location
15
Smart Buildings
Actions
HVAC automation
Adjust mix of energy sources
Time-shift predictable loads
Models
Building temperature in response
to external weather, HVAC controls
Events &
Activities
Holidays, Special events
Repair work, cold snap, heat wave
Energy price changes
Trajectories
State history
State
Variables
Room occupancy, CO2 level
Temperature, humidity, lighting
Exterior weather
16
Analytics for a
Research-to-Market IoT Center
Data Quality Control
Computer Vision
Modeling
User Interface and User Experience
Security and Privacy
System Executive and Control
17
Technology Needs: Data Cleaning
 Sensor diagnosis – Detect known sensor failure modes
 Anomaly detection – Detect novel sensor failures
 Imputation: learn and apply historical models to interpolate
missing or damaged data
 Data management
People:
o Tom Dietterich
o Alan Fern
o Weng-Keen Wong
o Xiaoli Fern
o Raviv Raich
o Arash Termehchy
18
Technology Needs:
Computer Vision
 Tracking
 Pinch points
 Dead regions
 Hot spot detection
 Event & activity recognition
 walking, standing, bending over,
looking down/up
 picking up item, setting down
item, placing item in basket,
placing item in
bag/backpack/pocket
 waiting, browsing, searching for
something
 successfully finding item
 frustration, impatience
 shopping “mode” (“on a specific
mission”, “browsing”, “long list”)
 Face recognition
 Item recognition
People:
o Sinisa Todorovic
o Alan Fern
19
Technology Needs:
Modeling
 Discovering Interesting Patterns in
Data
 Integration with External Data
Sources
 Web site visits
 Purchase History
 Third Party Customer Models, Social
Networks
 Shopping Apps
People:
o Weng-Keen Wong
o Xiaoli Fern
o Ron Metoyer
o Eugene Zhang
o Scott Sanner
 Optimizing Where and When RealTime Analytics are Computed
 Data Management
 Interactive Information Visualization
20
Technology Needs:
User Interface/User Experience
 Employee cuing (handheld? ear piece? smart watch?)
 Customer interaction via smart phone and kiosk
 Long-term customer relationship
 Long-term employee relationship
People:
o Margaret Burnett
o Chris Scaffidi
o Martin Erwig
21
Technology Needs:
Cybersecurity & Privacy
 Encryption
 Query-Specific Differential Privacy
 Software Quality and Testing
 Intrusion and Advanced Persistent Threat Detection
 Anomaly Detection
People:
o Rakesh Bobba
o Amir Nayyeri
o Attila Yavuz
o Mike Rosulek
o
o
o
o
o
Danny Dig
Alex Groce
Tom Dietterich
Alan Fern
Weng-Keen Wong
22
Technology Needs:
Control Executive / Integration
All components of the system must coordinate
to ensure
excellent experience for the customer
excellent experience for employees
minimize costs (power, bandwidth, analysis)
Algorithms for control
Optimization
Planning
Reinforcement Learning
People:
o Ted Brekken
o Alan Fern
o Prasad Tadepalli
o Tom Dietterich
o Thinh Nguyen
o Yue Zhang
23
Oregon State Advantages
Tightly-integrated School of Electrical Engineering
and Computer Science
Many EE⟺CS collaborations
Consistent Federal funding
Strong entrepreneurial culture
CASS: Center for Applied Systems and Software
Close ties to College of Business via the Division of
Engineering and Business
Easy routes to collaboration with industry
24