Joshua New, Ph.D.
Oak Ridge National Laboratory
newjr@ornl.gov
865-241-8783
Seminar 55
Simulation Calibration
Autotune Calibration
Learning Objectives
• Describe how ASHRAE Guideline 14 defines calibration criteria for
energy simulation
• Describe how high performance computing (HPC) resources can be
used to efficiently distribute simulation runs across multiple servers
• Describe how machine learning algorithms can be used to support
the development of efficient calibration techniques
• Describe the disadvantages of each of the three calibration
techniques presented
• Describe the advantages of each of the three calibration techniques
presented
• Describe realistic scenarios for model calibration that can be utilized
by practitioners today
ASHRAE is a Registered Provider with The American Institute of Architects Continuing Education Systems. Credit earned on
completion of this program will be reported to ASHRAE Records for AIA members. Certificates of Completion for non-AIA
members are available on request.
This program is registered with the AIA/ASHRAE for continuing professional education. As such, it does not include content
that may be deemed or construed to be an approval or endorsement by the AIA of any material of construction or any
method or manner of handling, using, distributing, or dealing in any material or product. Questions related to specific
materials, methods, and services will be addressed at the conclusion of this presentation.
Acknowledgements
•
•
•
•
•
•
•
•
•
Amir Roth – DOE’s BTO
Aaron Garrett – JSU
Jibonananda Sanyal – ORNL
Richard Edwards – UT
Mahabir Bhandari – ORNL
Som Shrestha – ORNL
Buzz Karpay – Karpay Associates
XSEDE
OLCF
Outline/Agenda
• Motivation
• What is Autotune?
– Calibration as search
• How does it work?
– Methods for speeding up the search
• How good is it?
– Calibration process and accuracy
• How can I use it?
– Deployment as web service
Motivation
3,000+ building survey, 23-97% monthly error
ASHRAE G14
Requires
Using Monthly utility
data
CV(RMSE)
15%
NMBE
5%
Using Hourly utility
data
CV(RMSE)
30%
NMBE
10%
5
The Autotune Idea
Automatic calibration of software to data
E+ Input
Model
.
.
.
6
Calibration as Search
Problem/Opportunity:
~3000 parameters per input file
2 minutes per simulation = 83 hours
7
Supercomputers for Buildings
•
•
•
•
EnergyPlus is a desktop app
Writes files during a simulation
Use RAMdisk
Balance simulation memory
vs. result storage
• Validate simulation output
• Bulk write data to disk
• Design of Experiments for
Uncertainty Quantification
• In-Situ data analysis
• Scalable Architecture for Big
Data Mining
• 270TB of simulation data
CPU
Wall-clock Data EnergyPlus
Cores Time (mm:ss) Size Simulations
16
18:14
5 GB
64
32
18:19
11 GB
128
64
18:34
22 GB
256
128
18:22
44 GB
512
256
20:30
88 GB
1,024
512
20:43
176 GB
2,048
1,024
21:03
351 GB
4,096
2,048
21:11
703 GB
8,192
4,096
20:00
1.4 TB
16,384
8,192
26:14
2.8 TB
32,768
16,384
26:11
5.6 TB
65,536
32,768
31:29
11.5 TB
131,072
65,536
44:52
23 TB
262,144
131,072
68:08
45 TB
524,288
8
Suite of Machine Learning
• Linear Regression
• Non-Linear Regression
• Feedforward Neural
Network
• Self-Organizing Map with
Local Models
• Regression Tree (using
Information Gain)
• Support Vector Machine
Regression
• Time Modeling with Local
Models
• K-Means with Local
Models
• Recurrent Neural Networks
• Gaussian Mixture Model
with Local Models
• Ensemble Learning
(combinations of
multiple algorithms)
• Genetic Algorithms
Integrated mixture of
Commercial, Research, and
Open Source software
9
MLSuite Architecture
Data Preparation
PBS
XML
Supercomputer
#1
MLSuite
Linux
#1
30x LS-SVMs
validation folds 1-10
input orders 1-3
…
Linux
#218
Supercomputer
#2
MLSuite Example
• EnergyPlus – 2-10 mins for an annual simulation
!- ALL OBJECTS IN CLASS
Version,
7.0;
!- Version
!- SIMULATIONCONTROL ===
SimulationControl,
No, !-Do Zone Sizing Calc
No, !-Do System Sizing Calc
…
• ~E+ - 4 seconds AI agent as surrogate model,
90x speedup, <5% error; “brittle” <156 input changes
11
Getting more for less
• EnergyPlus is slow
– Full-year schedule
– 2 minutes per simulation
• Use abbreviated 4-day schedule instead
– Jan 1, Apr 1, Aug 1, Nov 1
– 10 – 20 seconds per simulation
r = 0.94
Monthly Electrical Usage
r = 0.96
Hourly Electrical Usage
12
Evolutionary Computation
How are offspring produced?
Thickness
Conductivity
Density
Specific Heat
Bldg1
0.022
0.031
29.2
1647.3
Bldg2
0.027
0.025
34.3
1402.5
(1+2)1
0.0229
0.029
34.13
1494.7
(1+2)2
0.0262
0.024
26.72
1502.9
• Average each component
• Add Gaussian noise
• … “AI inside of AI”
13
Evolutionary Process
Island Hopping
1.
2.
3.
4.
4 of 19 experiments
Surrogate Modeling
Sensor-based Energy
Modeling (sBEM)
Abbreviated Schedule
Island-model evolution
14
Final Calibration Accuracy
XSEDE and DOE Office of Science
DOE-EERE BTO
Industry and building owners
Commercial Buildings
Monthly
utility data
Hourly
utility data
Features:
Works with “any” software
Tunes 100s of variables
Customizable distributions
Matches 1+ million points
Uses commodity hardware
ASHRAE
G14
Requires
Autotune
Results
15%
5%
30%
10%
0.32%
0.06%
0.48%
0.07%
CVR
NMBE
CVR
NMBE
Residential
home
Within
30¢/day
(actual use
$4.97/day)
Leveraging HPC resources to calibrate models on commodity for optimized building efficiency decisions
15
Working Internal Website
60+ fields (optional)
Determine Inputs to Calibrate
#Inputs
#Groups
#Inputs
#Groups
Restaurant
Hospital
49
49
227
146
Large Hotel Large Office
110
71
85
45
Secondary
Stand-alone
Small Hotel Small Office
School
Retail
231
282
72
59
128
136
61
56
Medium
Office
81
38
Strip Mall
113
89
Midrise
Apartment
155
82
Super
Market
78
73
Primary
School
166
113
Quick
Service
54
54
Warehouse
TOTAL
47
45
1809
1143
Provide Data
Calibrated Results
Metric
Input error average
Input error maximum
Input error minimum
Input error variance
IDF + CSV = XML
CV(RMSE)
CH4:Facility [kg](Monthly)
CO2:Facility [kg](Monthly)
CO:Facility [kg](Monthly)
Carbon Equivalent:Facility [kg](Monthly)
Cooling:Electricity [J](Hourly)
Electricity:Facility [J](Hourly)
…
NMBE
CH4:Facility [kg](Monthly)
CO2:Facility [kg](Monthly)
CO:Facility [kg](Monthly)
Carbon Equivalent:Facility [kg](Monthly)
Cooling:Electricity [J](Hourly)
Electricity:Facility [J](Hourly)
Electricity:Facility [J](Monthly)
143+ outputs
Value
24.38
66.12
0.09
228.53
9.95
15.42
20.40
14.42
1577.96
10.48
-9.57
-14.78
-19.52
-13.83
592.77
-9.52
-9.52
Performance and Availability
ASHRAE
Autotune
G14
Results
Requires
Monthly
utility data
Hourly
utility data
CVR
NMBE
CVR
NMBE
15%
5%
30%
10%
0.32%
0.06%
0.48%
0.07%
ASHRAE
Autotune
G14
Results
Requires
Monthly
utility data
Hourly
utility data
CVR
NMBE
CVR
NMBE
15%
5%
30%
10%
1.20%
0.35%
3.65%
0.35%
Results from 24 Autotune calibrations
Results from 20,000+ Autotune calibrations
(3 building types - 8, 34, 79 tuned inputs each)
(15 types – 47-282 tuned inputs each)
FY15 project to begin integration of
Autotune web service as OpenStudio application
Free to use. Pay for cloud computing.
Bibliography
•
•
•
•
•
•
•
Garrett, Aaron and New, Joshua R. (2014). "A Scientific Study of Automated Calibration applied to
DOE Commercial Reference Buildings." ORNL internal report ORNL/TM-2014/709, December 31,
2014, 114 pages
Ostrouchov, George, New, Joshua R., Sanyal, Jibonananda, and Patel, Pragnesh (2014).
"Uncertainty Analysis of a Heavily Instrumented Building at Different Scales of Simulation." In
Proceedings of the 3rd International High Performance Buildings Conference, Purdue, West
Lafayette, IN, July 14-17, 2014.
Sanyal, Jibonananda, New, Joshua R., Edwards, Richard E., and Parker, Lynne E. (2014).
"Calibrating Building Energy Models Using Supercomputer Trained Machine Learning Agents." In
Journal on Concurrency and Computation: Practice and Experience, March, 2014.
Garret, Aaron and New, Joshua R. (2013). "Trinity Test: Effectiveness of Automatic Tuning for
Commercial Building Models." ORNL internal report ORNL/TM-2013/130, March 7, 2013, 24
pages.
Edwards, Richard E., New, Joshua R., and Parker, Lynne E. (2012). "Predicting Future Hourly
Residential Electrical Consumption: A Machine Learning Case Study." In Journal of Energy and
Buildings, volume 49, issue 0, pp. 591-603, June 2012.
Bhandari, Mahabir S., Shrestha, Som S., and New, Joshua R. (2012). "Evaluation of Weather
Data for Building Energy Simulations." In Journal of Energy and Buildings, volume 49, issue 0, pp.
109-118, June 2012.
Garrett, Aaron and New, Joshua R. (2012). "An Evolutionary Approach to Parameter Tuning of
Building Models (Experiments 1-17)." ORNL internal report ORNL/TM-2012/418, April 2012, 68
pages.
Questions?
Joshua New
newjr@ornl.gov