Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Automating Large-Scale Simulation Calibration
to Real-World Sensor Data
Richard E. Edwards
Distributed Intelligence Lab
Department of Electrical Engineering and Computer Science
University of Tennessee, Knoxville TN, USA
March 13, 2013
Funded by Whole Building & Community Integration Group,
Oak Ridge National Laboratory, Oak Ridge TN, USA
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
1
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Outline
Introduction
Preliminaries
Calibration Signal Estimation & Sensor Selection
Simulation Approximation
Learning Simulation Variable Relationships
Conclusion
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
2
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Outline
Introduction
Preliminaries
Calibration Signal Estimation & Sensor Selection
Simulation Approximation
Learning Simulation Variable Relationships
Conclusion
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
3
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Simulation Modeling
I
Nuclear Power
I
Climate
I
Buildings
I
Physics
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
4
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Simulation Problems
I
Simulation Model vs Actual Structure
I
I
I
I
Stated material properties vs Actual material properties
Estimated duty cycle vs Actual usage
Expected structure vs Built structure
Simulation Limitations
I
I
I
Computer limitations
Code-base maintenance
Simulating new technologies
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
5
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Simulation Problems
I
Simulation Model vs Actual Structure
I
I
I
I
Stated material properties vs Actual material properties
Estimated duty cycle vs Actual usage
Expected structure vs Built structure
Simulation Limitations
I
I
I
Computer limitations
Code-base maintenance
Simulating new technologies
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
6
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Previous Simulation Calibration Methodologies
I
Two Overall Methodologies
I
I
Manual procedural calibration
Semi-Automated statistical calibration
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
7
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Previous Simulation Calibration Methodologies
I
Two Overall Methodologies
I
I
Manual procedural calibration
Semi-Automated statistical calibration
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
8
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Manual Procedural Calibration
I
Core Components
I
Physical Audit Period
I
I
I
Yoon and Lee (1999)
Pendrini et al. (2002)
Sensitivity Analysis
I
Westphal and Lamberts (2005)
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
9
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Manual Procedural Calibration
I
Core Components
I
Physical Audit Period
I
I
I
Sensitivity Analysis
I
I
Yoon and Lee (1999)
Pendrini et al. (2002)
Westphal and Lamberts (2005)
Additional Components
I
Multi-phase Audit Procedure
I
Raftery et al. (2011)
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
9
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Manual Procedural Calibration
I
Core Components
I
Physical Audit Period
I
I
I
Sensitivity Analysis
I
I
Westphal and Lamberts (2005)
Additional Components
I
Multi-phase Audit Procedure
I
I
Yoon and Lee (1999)
Pendrini et al. (2002)
Raftery et al. (2011)
Problems
I
I
Entire process is slow
Non-transferable
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
9
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Previous Simulation Calibration Methodologies
I
Two Overall Methodologies
I
I
Manual procedural calibration
Semi-Automated statistical calibration
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
10
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Semi-Automated Statistical Calibration
I
Core Components
I
Surrogates (Approximations)
I
I
I
I
I
I
Ordinary Least Squares (OLS)
Multivariate Adaptive Splines (MARS)
Kriging
Radial Basis Functions (RBF)
Jin et al. 2001
Automatic Sensitivity Analysis (Feature Selection)
I
I
OLS (Helton et al. 2006)
MARS (Storlie et al. 2009)
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
11
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Semi-Automated Statistical Calibration
I
Core Components
I
Surrogates (Approximations)
I
I
I
I
I
I
Automatic Sensitivity Analysis (Feature Selection)
I
I
I
Ordinary Least Squares (OLS)
Multivariate Adaptive Splines (MARS)
Kriging
Radial Basis Functions (RBF)
Jin et al. 2001
OLS (Helton et al. 2006)
MARS (Storlie et al. 2009)
Problems
I
Scalability & Generality
I
I
Tian and Choudhary (2012) – Macro-scale E+ simulation calibration
(10 parameters, 1 output)
Zeng et al. (2013) – Crop simulation calibration
(100 parameters & ∼18 days )
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
11
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Problem Overview
I
Simulations require calibration
I
Manual method problems:
I
I
I
Slow
Not transferable
Semi-Automated method problems:
I
I
Scalability
Generality
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
12
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
cument
Calibration
Diagram
Calibration
Signal
Environmental
Sensors
Filter
Sensor
Data
Or
Relational Model
Map sensors to
Simulation
Output
Simulation
Approximation
Estimate
Simulation
Parameters
Select Best
Model
Delay
Select Best
Sensors
Simulation
Parameters
yes
no
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
13
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
cument
Calibration
Diagram
Calibration
Signal
Environmental
Sensors
Filter
Sensor
Data
Or
Relational Model
Step 1
Map sensors to
Simulation
Output
Simulation
Approximation
Estimate
Simulation
Parameters
Select Best
Model
Delay
Select Best
Sensors
Simulation
Parameters
yes
no
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
14
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
cument
Calibration
Diagram
Calibration
Signal
Environmental
Sensors
Filter
Sensor
Data
Or
Relational Model
Map sensors to
Simulation
Output
Simulation
Approximation
Estimate
Simulation
Parameters
Select Best
Model
Delay
Select Best
Sensors
Simulation
Parameters
Step 2
yes
no
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
15
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
cument
Calibration
Diagram
Calibration
Signal
Environmental
Sensors
Filter
Sensor
Data
Or
Relational Model
Map sensors to
Simulation
Output
Simulation
Approximation
Estimate
Simulation
Parameters
Select Best
Model
Delay
Step 3
Select Best
Sensors
Simulation
Parameters
yes
no
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
16
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
cument
Calibration
Diagram
Step 4
Calibration
Signal
Environmental
Sensors
Filter
Sensor
Data
Or
Relational Model
Map sensors to
Simulation
Output
Simulation
Approximation
Estimate
Simulation
Parameters
Select Best
Model
Delay
Select Best
Sensors
Simulation
Parameters
yes
no
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
17
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
cument
Calibration
Diagram
Step 5
Calibration
Signal
Environmental
Sensors
Filter
Sensor
Data
Or
Relational Model
Map sensors to
Simulation
Output
Simulation
Approximation
Estimate
Simulation
Parameters
Select Best
Model
Delay
Select Best
Sensors
Simulation
Parameters
yes
no
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
18
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
cument
Calibration
Diagram
Calibration
Signal
Environmental
Sensors
Filter
Sensor
Data
Or
Relational Model
Map sensors to
Simulation
Output
Simulation
Approximation
Estimate
Simulation
Parameters
Select Best
Model
Delay
Step 6
Select Best
Sensors
Simulation
Parameters
yes
no
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
19
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Summary
I
Step 1 — Select Best Prediction Model
I
Step 2 — Select Best Sensors (Step 1 result)
I
Step 3 — Map Sensors to Simulation Outputs (Step 2 result)
I
Step 4 — Build Simulation Approximation (Optional)
I
Step 5 — Build Simulation Relational Model
I
Step 6 — Calibrate Simulation (Step 3, 4, 5 results)
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
20
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Contributions
I
Step 1
I
Best predictor for hourly residential electrical consumption
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
21
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Contributions
I
Step 1
I
I
Best predictor for hourly residential electrical consumption
Step 2
I
I
Best sensors for predicting electrical consumption
Novel feature selection method
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
21
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Contributions
I
Step 1
I
I
Step 2
I
I
I
Best predictor for hourly residential electrical consumption
Best sensors for predicting electrical consumption
Novel feature selection method
Step 4
I
Large-scale E+ residential approximation
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
21
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Contributions
I
Step 1
I
I
Step 2
I
I
I
Best sensors for predicting electrical consumption
Novel feature selection method
Step 4
I
I
Best predictor for hourly residential electrical consumption
Large-scale E+ residential approximation
Step 5
I
Large-scale regression structure learning
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
21
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Outline
Introduction
Preliminaries
Calibration Signal Estimation & Sensor Selection
Simulation Approximation
Learning Simulation Variable Relationships
Conclusion
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
22
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
EnergyPlus (E+)
I
Requirements
I
I
I
Building model
Operation schedule
Weather data
I
Advantages
I
Disadvantages
I
I
I
Most general model
Runtime 3∼7 minutes
300+ adjustable variables
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
23
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Sensor Data Sets
I
Two Subdivisions
I
I
I
Wolf Creek
I
I
I
Wolf Creek
Campbell Creek
approximately 250 sensors
15 minute resolution
Campbell Creek
I
I
approximately 140 sensors
15 minute resolution
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
24
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
E+ Simulation Data
I
Markov Order 1 (MO1)
I
I
I
Markov Order 2 (MO2)
I
I
I
Adjust parameters independently
min,max adjustment
Adjust two parameters together
min,max adjustments
Fine Grain (FG – Brute Force)
I
I
Adjust 14 parameters
Small incremental adjustments
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
25
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
E+ Simulation Data
Markov Order 1
Markov Order 2
Fine Grain
# Outputs
95
95
82
# Inputs
156 (151)
156 (151)
162 (14)
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
# Simulations
299
29,727
11,989
Gigabytes
3.9
387.2
136.0
University of Tennessee
26
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Performance Metrics
I
Root Mean Squared
Error(RMSE):
v
u
N
u 1 X
RMSE = t
(yi − pi )2
N −1
I
Coefficient of Variance(CV):
I
Mean Absolute Percentage of
Error(MAPE):
MAPE =
i=1
i=1
I
CV =
RMSE
× 100
ymean
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
N
1 X |yi − pi |
N
yi
Mean Bias Error(MBE):
MBE =
1
N−1
PN
i=1 (yi
ymean
− pi )
×100
University of Tennessee
27
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Outline
Introduction
Preliminaries
Calibration Signal Estimation & Sensor Selection
Simulation Approximation
Learning Simulation Variable Relationships
Conclusion
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
28
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Step 1 – Select Best Prediction Model
I
Problem – Identify the best model for predicting hourly residential
electrical consumption
I
Data Sets:
I
I
I
ASHRAE Great Energy Prediction Shooutout
Campbell Creek
Explored:
I
7 Different models
Residential Model
I
Commercial Model
I
I
I
I
Selected Least Squares Support Vector Machines (LS-SVM)
Selected Feed Forward Neural Network (FFNN)
Results Published:
I
R. E. Edwards, J. New, L. E. Parker, Predicting Future Hourly
Residential Electrical Consumption: A Machine Learning Case Study,
Energy and Buildings, vol 49, pages 591-603, June 2012
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
29
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Step 2 – Sensor Selection
I
Problem – Identify the best sensors for predicting hourly residential
electrical consumption
I
Data Sets:
I
I
Campbell Creek
Explored:
I
3 Feature Selection Methods with Linear Regression Model
I
I
I
I
I
Stepwise
Information Complexity with Inverse Fisher Information Matrix
(ICOMP(IFIM)) with Genetic Algorithm (GA)
Novel voting method via estimating ICOMP(IFIM) distribution
Bruteforce selection up to Best 4
Conclusions:
I
I
I
ICOMP(IFIM) with GA performs well
Voting method is usually better than ICOMP(IFIM) with GA
Best bruteforce set performs worse than selection methods
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
30
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Outline
Introduction
Preliminaries
Calibration Signal Estimation & Sensor Selection
Simulation Approximation
Learning Simulation Variable Relationships
Conclusion
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
31
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Overview
I
Problem – Determine large-scale E+ surrogate effectiveness
I
Data sets:
I
I
I
I
Approaches:
I
I
I
MO1 (4 Gigabytes)
MO2 (10 Gigabytes)
FG (10 Gigabytes)
Feed Forward Neural Networks (FFNN)
Lasso regression with Alternating Direction Method of Multipliers
Layout:
I
I
I
I
FFNN learning details and results
Lasso regression learning details and results
Discussion
Summary
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
32
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Feed Forward Neural Network
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
33
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Network Structure
I
Number Hidden Units
I
I
We explored 5, 10, 15
Number of Output Units: 10
I
I
I
Output variables were grouped based on how they were stored
Similar outputs were stored next to each other within the dataset
Other pairings could produce different results
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
34
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Learning Process
I
Batch
I
I
Stochastic
I
I
Uses all examples to make an update
Updates network per training example
We use a hybrid approach
I
I
Data is divided into mini-batches
Network is updated per mini-batch
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
35
Preliminaries
Step 1 & 2
5
5
Step 4
Step 5 & 6
x 10
5
Sensible RMSE
Latent RMSE
Sensible MTR
Latent MTR
4
RMSE
5
Fine Grain Loads with 15 Hidden Unit FFNN
x 10
4
3
3
2
2
1
1
0
0
65
I
Conclusion
Mean Target Response
Introduction
66
67
68
69
70
71 72 73 74 75
E+ Load Variables
76
77
78
79
80
FFNN with 15 and 5 hidden units fit the Fine Grain loads best
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
36
Preliminaries
Step 1 & 2
5
5
Step 4
Step 5 & 6
x 10
5
Sensible RMSE
Latent RMSE
Sensible MTR
Latent MTR
4
RMSE
5
Fine Grain Loads with 10 Hidden Unit FFNN
x 10
4
3
3
2
2
1
1
0
0
65
I
Conclusion
Mean Target Response
Introduction
66
67
68
69
70
71 72 73 74 75
E+ Load Variables
76
77
78
79
80
Does not fit variable 77 and 79 as well
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
37
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Fine Grain with 5 Hidden Unit FFNN
50
RMSE
MTR
45
RMSE
40
40
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
0
I
45
4
8
12
16
20
24 28 32 36 40 44
E+ Non−Load Variables
48
52
56
60
Mean Target Response
50
0
64
FFNN with 5 hidden units has difficulty modeling non-loads
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
38
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Fine Grain with 10 Hidden Unit FFNN
50
RMSE
MTR
45
RMSE
40
45
40
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
0
4
8
12
16
20
24 28 32 36 40 44
E+ Non−Load Variables
48
52
56
60
Mean Target Response
50
0
64
I
Fits non-loads better than the 5 hidden unit model
I
The 15 hidden unit model is very similar to the 10 hidden unit
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
39
Preliminaries
Step 1 & 2
5
5
Step 4
Step 5 & 6
x 10
5
Sensible RMSE
Latent RMSE
Sensible MTR
Latent MTR
4
RMSE
5
Order 1 Loads with 10 Hidden Unit FFNN
x 10
4
3
3
2
2
1
1
0
0
74
I
Conclusion
Mean Target Response
Introduction
75
76
77
78
79
80 81 82 83 84
E+ Load Variables
85
86
87
88
89
MO1 results are similar to FG results
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
40
Preliminaries
Step 1 & 2
5
5
Step 4
Step 5 & 6
x 10
5
Sensible RMSE
Latent RMSE
Sensible MTR
Latent MTR
4
RMSE
5
Order 1 Loads with 15 Hidden Unit FFNN
x 10
Conclusion
4
3
3
2
2
1
1
0
Mean Target Response
Introduction
0
74
75
76
77
78
79
80 81 82 83 84
E+ Load Variables
I
Does not fit variable 82 and 86 as well
I
5 or 10 hidden units is best
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
85
86
87
88
89
University of Tennessee
41
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Order 1 with 15 Hidden Unit FFNN
50
RMSE
MTR
45
RMSE
40
45
40
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
Mean Target Response
50
0
0
4
8
12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72
E+ Non−Load Variables
I
Best non-load model
I
variables 28 to 36
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
42
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Order 1 with 5 Hidden Unit FFNN
50
RMSE
MTR
45
RMSE
40
45
40
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
Mean Target Response
50
0
0
4
8
12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72
E+ Non−Load Variables
I
Slightly worse than 10 and 15
I
variables 28 to 36
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
43
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
FFNN Result Summary
I
Loads are difficult to estimate for both datasets
I
FFNN perform better on the Fine Grain dataset
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
44
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Lasso regression
I
Model:
y = hw , xi + b
I
Optimization Criteria:
N
1X
(yi − hw , xi i)2 + λ|w |
2
i=1
I
Exact-Model Training Time:
O(n3 )
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
45
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Optimization Method
I
Alternating Direction of Method of Multipliers (ADMM)
I
I
Boyd et al. 2010
Benefits:
I
Completely decentralized
Defines methods for splitting across examples and features
Can solve Lasso, SVM, and Ridge Regression
I
Guaranteed to converge
I
I
I
By changing a single function
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
46
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
ADMM
I
I
Data splitting across examples:
 
A0
A1 
 
A= . 
 .. 
An
 
b0
 b1 
 
b=.
 .. 
bn
Lasso optimization process
I
I
xik+1 = argminxi ( 12 ||Ai xi − bi ||22 + ρ2 ||xi − z k + uik ||22 )
z k+1 = S λ (x̄ k+1 + ū k )
ρN
I
I
uik+1 = uik + xik+1 − z k+1
First step is the only heavy lifting
I
xik+1 = (ATi Ai + ρI )−1 (ATi bi + ρ(z k − uik ))
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
47
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
FG Results
x 10
5
Sensible RMSE
Latent RMSE
Sensible MTR
Latent MTR
4
RMSE
5
Fine Grain Loads with Lasso Regression
x 10
4
3
3
2
2
1
1
0
Mean Target Response
5
5
0
65
66
67
68
69
70
71 72 73 74 75
E+ Load Variables
76
77
I
Does not estimate FG loads as well as FFNN
I
Based on variable 65 and 67
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
78
79
80
University of Tennessee
48
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
FG Results
Fine Grain with Lasso Regression
50
RMSE
MTR
45
RMSE
40
40
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
0
I
45
4
8
12
16
20
24 28 32 36 40 44
E+ Non−Load Variables
48
52
56
60
Mean Target Response
50
0
64
Estimates non-load variables worse than FFNN
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
49
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
MO1 Results
x 10
5
Sensible RMSE
Latent RMSE
Sensible MTR
Latent MTR
4
RMSE
5
Order 1 Loads with Lasso Regression
x 10
4
3
3
2
2
1
1
0
Mean Target Response
5
5
0
74
75
76
77
78
79
80 81 82 83 84
E+ Load Variables
85
I
Estimates MO1 loads better than FG loads
I
Worse than FFNN
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
86
87
88
89
University of Tennessee
50
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
MO1 Results
Order 1 with Lasso Regression
50
RMSE
MTR
45
RMSE
40
40
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
0
0
I
45
Mean Target Response
50
4
8
12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72
E+ Non−Load Variables
Estimates non-load variables as well as FFNN
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
51
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Discussion Points
I
Prediction time
I
Data Clustering via Error
I
MO2 contains unobserved behavior
I
HVAC Operating Features
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
52
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Prediction time
Model
FFNN 5
FFNN 10
FFNN 15
Lasso
Training
Time (Hr)
∼2
∼8
∼24
∼0.2833
Total Training
Time (Hr)
∼18
∼72
∼216
∼25.50
I
Lasso regression method scales best
I
FFNN is only competitive when run in parallel
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
Prediction
Time (sec)
∼2.70
∼2.85
∼2.93
∼2.90
University of Tennessee
53
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Error Clustering – FFNN
FG Variable 65’s CV Error Clustering
40
Group 1
Group 2
Group 3
CV Error Metric
38
36
34
32
30
0
I
100
200
300
400
500
Simulation
600
700
800
Suggests multiple models would be best
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
54
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Error Clustering – Lasso
FG Variable 52’s CV Error Clustering
FG Variable 63’s CV Error Not Clustering
2
Group 1
Group 2
Group 3
4
CV Error Metric
CV Error Metric
5
3
2
1.5
1
0.5
1
0
100
200
300
Simulation
400
500
600
0
0
100
200
300
Simulation
I
Variables with linear relationships cluster
I
Variables with non-linear relationships do not cluster
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
400
500
600
University of Tennessee
55
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Error Clustering – MO1
Order 1 Variable 75’s CV Error Group
44
Group
Unkown
CV Error Metric
42
40
38
36
34
0
50
100
150
200
Simulation
250
300
350
I
Contains a single grouping
I
Implies sampling resolution directly impacts the number of groups
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
56
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
MO2 Scale Shifts
Dryer’s Total Heat Gain Rate
2000
Nominal
Abnormal
Watts
1500
1000
500
0
Jan
Feb Mar
Apr
May
Jun
Jul Aug
Time
Sep
Oct
Nov
Dec
I
Reason behind variable 10’s high variance (MO1 exp)
I
3 Simulations in sampled MO2 test data match the Abnormal shift.
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
57
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
MO1 HVAC ON/OFF vs Living Room (LR) Latent Load
HVAC Heating ON/OFF vs MO1 LR Latent Heating
HVAC Cooling ON/OFF vs MO1 LR Latent Cooling
2
2
ON
OFF
Latent
1.5
1.5
1
1
0.5
0.5
0
0
Jan Feb Mar Apr May Jun
I
ON
OFF
Latent
Jul Aug Sep Oct Nov Dec
Time
Jan Feb Mar Apr May Jun
Jul Aug Sep Oct Nov Dec
Time
The ON/OFF state temporally matches well
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
58
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
FG HVAC ON/OFF vs Living Room (LR) Latent Load
HVAC Heating ON/OFF vs FG LR Latent Heating
HVAC Heating ON/OFF vs FG LR Latent Heating
2
2
ON
OFF
Latent
1.5
1
1
0.5
0.5
0
ON
OFF
Latent
1.5
0
Jan Feb Mar Apr May Jun
Jul Aug Sep Oct Nov Dec
Time
Jan Feb Mar Apr May Jun
Jul Aug Sep Oct Nov Dec
Time
I
The ON/OFF state does not temporally match as well
I
FG Living Room Latent Heating distributed throughout the year
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
59
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
HVAC Results
I
FG HVAC Experiments
I
I
I
I
Improved performance – 6 loads
Diminished performance – 8 loads
Unchanged performance – 2 loads
MO1 HVAC Experimetns
I
I
I
Improved performance – 4 loads
Diminished performance – 6 loads
Unchanged performance – 6 loads
I
Latent loads are still difficult to estimate
I
Some sensible loads are equally difficult
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
60
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Summary
I
Both models approximate non-load variables well
I
FFNN is best over all
I
Lasso trains much faster, and scales better
I
Multiple models are most likely needed
I
Generalization may be hindered by scale shifts
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
61
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Outline
Introduction
Preliminaries
Calibration Signal Estimation & Sensor Selection
Simulation Approximation
Learning Simulation Variable Relationships
Conclusion
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
62
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Overview
I
Problem – Estimate building parameters via their relationships
I
Data sets:
I
I
I
I
Approaches:
I
I
I
I
MO1 (4 Gigabytes)
MO2 (10 Gigabytes)
FG (10 Gigabytes)
Direct Lasso regression structure learning
Bayesian Lasso regression structure learning
Both require Alternating Direction Method of Multipliers
Layout:
I
I
I
I
I
I
I
Problem forumlization
Structure learning review
Direct Lasso structure learning
Bayesian Lasso structure learning
Results
Discussion
Summary
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
63
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Variable Relationship Learning (Step 5)
I
Relational Learning
I
All Energy Plus inputs and outputs are random variables
Z = {X1 , X2 , . . . , Xn }
I
Then PDF, P(Z ), describes interactions
P(Z ) = P({X1 , X2 , . . . , Xn })
I
Objective:
I
Learn/Approximate P(Z )
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
64
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Variable Inference (Step 6)
I
Both are MAP Inference problems
I
Calibration
P(Z ) =
N
Y
P(X |Yi )P(X )
i=1
I
I
I
X = {Building Parameters}
Yi = {Weather Data, Operation Schedule, Sensor Data}
Approximation
P(Z ) =
N
Y
P(Yi |Xi )P(Yi )
i=1
I
I
Xi = {Building Parameters, Weather Data, Operation Schedule}
Yi = {Simulation Output}
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
65
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Variable Relationship Learning
I
Computing P(Z ) is intractable in the general case
I
Assume P(Z ) factorizes into simpler components
I
i.e. P(Z ) = P(X1 |X2 )P(X2 |X3 ) . . . P(XN )
I
Factorizations are best represented by a Probabilistic Graphical
Model (PGM), G
I
G = {V , E }
I
I
I
V are random variables
E , edges, represents conditional dependencies
Find G that best approximates P(Z )
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
66
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
PGM Structure Learning
I
Classical Approaches:
I
I
I
Score and Search
Constraint Based
Less Traditional:
I
Regression Based
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
67
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Score and Search
I
I
Search through possible G
Saving the G that best approximates P(Z )
I
I
Best G minimizes a criteria function
Most common criteria:
BIC(Data; G) = L(Data; G , θ) +
I
log M
∗ Dim[G ]
2
Common Algorithms
I
Greed-Hill Climbing
Sparse Candidate Algorithm (SCA)
I
Max-Min Hill-Climb (MMPC)
I
I
I
Friedman et al. 1999
Tsamardinos et al. 2006
I
Advantage
I
Disadvantage
I
I
I
I
Global Criteria Function
Applicable only to Bayesian Networks (DAG)
Too expensive for Markov Network (Undirected Graph).
Search space size
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
68
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Constraint Based
I
Searches through variable dependencies, via statistical test
I
Dependent variables are connected
I
Common Algorithms
I
SGS (named after authors)
I
Grow and Shrink
I
I
I
Margaritis et al. 1999
Advantage
I
I
Spirtes et al. 1993
Applicable to Bayesian and Markov Networks
Disadvantage
I
I
Scalability - not always guaranteed
Hard to apply to continuous random variables
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
69
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Regression Based
I
Assumes functional relationship between random variables
I
Uses variable selection methods to determine dependencies
I
Methods
I
Sparse Graphical Model for exploring gene expression data
I
Structure learning with large sparse undirected graphs
I
Modified Grow and Shrink
I
I
I
(Li, Fan. 2007)
(Jean-Philippe Pellet and Andre Elisseeff 2008)
Advantage
I
I
I
(Dobra, A., et al., 2004)
I
Applicable to Bayesian and Markov Networks
Highly Scalable
Disadvantage
I
Existing methods assume the joint probability P(Z ) is Gaussian
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
70
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Gaussian Graphical Model (GGM)
I
Assumes PDF, P(Z ), is Gaussian
P(Z ) =
I
n
1
1
exp(− (x − µ)T Ω(x − µ))
2
Requirements:
I
I
I
I
1
(2π) 2 |Σ| 2
estimating Σ
estimating µ
computing Ω, Σ−1
Exact Inference:
p(x) ∝ exp(x T Jx + hT x)
I
I
J=Ω
h=µ
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
71
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Learning Gaussian Graphical Model
I
Approach
I
I
I
Learn a Bayesian Network
Extract the undirected GGM
Estimating Ω:
Ω = (1 − Γ)T Ψ−1 (1 − Γ)
I
Γ Bayesian Network weights
I
I
I
Ψ−1
I
I
I
Upper triangle
Zero Diagonals
Diagonal matrix
Each entry is the estimated variance
Estimating h
I
I
Not required, if data is centered
Estimate centering parameters instead
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
72
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Directly Estimating Γ and Ψ−1
I
Approach
I
I
Impose an ordering over the variables
Perform N − 1 regressions
I
I
I
Γi = βi+1,n
I
I
Response is Xi
Candidate Parents are {Xi+1 , . . . , Xn }
βi+1,n computed via Lasso regression
Ψ−1 = diag (σ1 , · · · , σn )−1
I
σi is the estimated response variance
I
σi =
P|D|
n=1 (xni −
PN
j=i+1
βj xnj )2
|D|−2
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
73
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Variable Order
I
Order matters for Direct Method
I
I
A greedy algorithm can select good orders
Dobra et al., 2004
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
74
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Variable Order
I
Order matters for Direct Method
I
I
I
Order does not matter, proved by (Li, Fan. 2007)
I
I
A greedy algorithm can select good orders
Dobra et al., 2004
Dependent upon applying Wishart Prior to Ω
Prior provides iterative Lasso Regression method
I
Iteratively estimates Γ and Ψ−1
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
74
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Bayesian Γ and Ψ−1 Estimation
I
Wishart prior W (δ, T )
I
I
I
I
I
Γi = βi+1,n
I
I
I
δ defines shape
T defines prior covariance hyperparameter
T = diag (θi , · · · , θn )
i
)
P(θi ) = γ2 exp( −γθ
2
βi+1,n√computed via Lasso regression
λ = γψi
Ψ−1 = diag (ψ1−1 , · · · , ψn−1 )
I
ψi−1 =
PN
j=i+1
I
δ−1+N−2i+|D|
P|D|
PN
2
n=1 (xni − j=i+1 βij xnj )
P
−1
M
1
P(θj )
j=1 ψ̂j
M
βij2 θi−1 +θi−1 +
E[ψi−1 |θi , βi , D] =
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
75
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Small Scale Bayesian MO1 Result
Bayes Error
Variables 27 to 52
1.2
1
1
Difference in
Absolute Error
Difference in
Absolute Error
Bayes Error
Variables 1 to 26
1.2
0.8
0.6
0.4
0.2
0
0.8
0.6
0.4
0.2
0
5
10
15
Variables
20
0
26
25
31
1.2
1
1
0.8
0.6
0.4
0.2
0
52
62
67
Variables
73
98
103
149
154
0.6
0.4
0
78
78
83
88
93
Variables
Bayes Error
Variables 131 to 151
1.2
1
Difference in
Absolute Error
1
Difference in
Absolute Error
51
0.2
57
1.2
0.8
0.6
0.4
0.2
I
46
0.8
Bayes Error
Variables 105 to 130
0
104
41
Variables
Bayes Error
Variables 79 to 104
1.2
Difference in
Absolute Error
Difference in
Absolute Error
Bayes Error
Variables 53 to 78
36
0.8
0.6
0.4
0.2
109
114
119
Variables
124
129
0
130
135
139
144
Variables
139 good estimates, 8 poor estimates, 4 moderate estimates
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
76
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Small Scale Direct MO1 Result
Direct Error
Variables 27 to 52
1.2
1
1
Difference in
Absolute Error
Difference in
Absolute Error
Direct Error
Variables 1 to 26
1.2
0.8
0.6
0.4
0.2
0
0.8
0.6
0.4
0.2
0
5
10
15
Variables
20
0
26
25
31
1.2
1
1
0.8
0.6
0.4
0.2
0
52
62
67
Variables
73
98
103
149
154
0.6
0.4
0
78
78
83
88
93
Variables
Direct Error
Variables 131 to 151
1.2
1
Difference in
Absolute Error
1
Difference in
Absolute Error
51
0.2
57
1.2
0.8
0.6
0.4
0.2
I
46
0.8
Direct Error
Variables 105 to 130
0
104
41
Variables
Direct Error
Variables 79 to 104
1.2
Difference in
Absolute Error
Difference in
Absolute Error
Direct Error
Variables 53 to 78
36
0.8
0.6
0.4
0.2
109
114
119
Variables
124
129
0
130
135
139
144
Variables
139 good estimates, 13 poor estimates, 0 moderate estimates
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
77
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Small Scale Random MO1 Result
Bayes Error vs Rand Error
Variables 27 to 52
1
0.5
0.5
Difference in
Absolute Error
Difference in
Absolute Error
Bayes Error vs Rand Error
Variables 1 to 26
1
0
−0.5
−1
0
5
10
15
Variables
20
0
−0.5
−1
26
25
31
1
1
0.5
0.5
0
−0.5
−1
52
57
62
67
Variables
73
−1
78
78
0.5
I
Difference in
Absolute Error
Difference in
Absolute Error
0.5
0
−0.5
119
Variables
124
51
83
88
93
Variables
98
103
Bayes Error vs Rand Error
Variables 131 to 151
1
114
46
0
Bayes Error vs Rand Error
Variables 105 to 130
109
41
Variables
−0.5
1
−1
104
36
Bayes Error vs Rand Error
Variables 79 to 104
Difference in
Absolute Error
Difference in
Absolute Error
Bayes Error vs Rand Error
Variables 53 to 78
129
0
−0.5
−1
130
135
139
144
Variables
149
154
Sampling [0, 1] uniform distribution is competitive
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
78
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
FG Results
Rand FG Error
1
0.8
0.8
Difference in
Absolute Error
Difference in
Absolute Error
Bayes FG Error
1
0.6
0.4
0.2
0.6
0.4
0.2
0
0
9
27
28
29
30
66
67 68 69 157 158 159 160 181
Variables
9
27 28 29 30 66 67 68 69 157 158 159 160 181
Variables
I
Bayesian is statistically better – 4.05±0.98 vs 5.07±1.01
I
Statistically different variables 27, 28, 68, 157, 158, 159, 160, and
181
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
79
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
FG Results
0.8
0.6
0.4
0.2
Estimate
Actual
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.8
0.6
0.4
0.2
0
9
27 28 29 30 66 67 68 69 157 158 159 160 181
Variables
0
9
27 28 29 30 66 67 68 69 157 158 159 160 181
Variables
I
Random works best on 0.5 mean variables
I
Bayesian tracks means better
I
Appears to infer building parameters well
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
1
Parameter Values
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
Parameter Estimates
Random Parameter Estimation
Estimate
Actual
Parameter Values
Parameter Estimates
Bayesian Parameter Estimation
University of Tennessee
80
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
MO2 Results
10
15
Variables
20
25
26
31
1.5
1
0.5
0.5
83
88
93
Variables
98
103
Parameter Estimates
1
78
41
Variables
46
51
52
0.5
57
1.2
1
0.8
0.6
0.4
0.2
1.2
1
0.8
0.6
0.4
0.2
109
114
119
Variables
124
129
I
Appears to infer building parameters well
I
Tracks means well
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
62
67
Variables
73
78
Bayesian Parameter Estimates
Variables 131 to 151
Estimate
Actual
104
1.5
1
0.5
Bayesian Parameter Estimates
Variables 105 to 130
Estimate
Actual
Parameter Values
Parameter Estimates
Bayesian Parameter Estimates
Variables 79 to 104
1.5
36
Parameter Estimates
5
0.5
Parameter Values
0
0.5
Estimate
Actual
1
Parameter Values
0.5
1
1.5
Estimate
Actual
0.7
0.6
0.5
0.4
0.3
130
0.7
0.6
0.5
0.4
0.3
135
139
144
Variables
149
Parameter Values
0.5
1
1.5
Parameter Estimates
1
Bayesian Parameter Estimates
Variables 53 to 78
Estimate
Actual
1.5
Parameter Values
1
1.5
Parameter Estimates
Bayesian Parameter Estimates
Variables 27 to 52
Estimate
Actual
Parameter Values
Parameter Estimates
Bayesian Parameter Estimates
Variables 1 to 26
1.5
154
University of Tennessee
81
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Discussion
I
Inference optimization methods
I
Estimating Distant Values
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
82
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
GA vs Gradient
FG Building Parameter 2
Parameter Values
1.5
Actual
Gradient−Est
GA−Est
1
0.5
0
−0.5
0
I
I
I
5
10
15
20
25
30
Simulations
35
40
45
50
Gradient estimates near the mean often
GA introduces more variance
Gradient better for large parameter inference
I
I
Variance scales with number of parameters
MO1 and FG used GA, MO2 used Gradient
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
83
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Estimating Distant Values
MO2 Building Parameter 3
Parameter Value
2
One−Act
Zero−Act
Est
Zero−Est
1.5
1
0.5
0
0
50
100
150
Simulation
200
I
Values concentrate on the mean closely
I
Distant values hard to estimate
I
Changing γ and δ may introduce more variance
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
250
300
University of Tennessee
84
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Summary
I
Bayesian GGM estimates building parameters well
I
Best when using gradient optimization
I
However, all estimates always concentrate towards mean
I
Model needs more variability
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
85
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Outline
Introduction
Preliminaries
Calibration Signal Estimation & Sensor Selection
Simulation Approximation
Learning Simulation Variable Relationships
Conclusion
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
86
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Solution Overview
I
Step 1 — Select best Prediction Model
I
Step 2 — Select best Sensors (Step 1 result)
I
Step 3 — Map Sensors to Simulation Outputs (Step 2 result)
I
Step 4 — Build Simulation Approximation (Optional)
I
Step 5 — Build Simulation Relational Model
I
Step 6 — Calibrate Simulation (Step 3, 4, 5 results)
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
87
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Addressed Steps
I
Step 1 — Selected LS-SVM model
I
Step 2 — Selected best sensors from Campbell Creek
I
Step 4 — Built large-scale E+ approximation
I
Step 5 — Built large-scale E+ relational model
I
Step 6 — Partially demonstrated (E+ Ground Truth Simulations)
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
88
Introduction
Preliminaries
Step 1 & 2
Step 4
Step 5 & 6
Conclusion
Contributions
I
Step 1
I
I
Step 2
I
I
I
Best sensors for predicting electrical consumption
Novel feature selection method
Step 4
I
I
Best predictor for hourly residential electrical consumption
Large-scale E+ residential approximation
Step 5
I
Large-scale regression structure learning
Richard E. Edwards
Automating Large-Scale Simulation Calibration to Real-World Sensor Data
University of Tennessee
89