Software vs. Reality
Bridging the Gap
For IBPSA-North Texas
Wednesday, 10/28/15
Joshua New, Ph.D.
865-241-8783
newjr@ornl.gov
http://bit.ly/autotune_papers
ORNL is managed by UT-Battelle
for the US Department of Energy
The Autotune Idea
Automatic calibration of software to data
E+ Input
Model
.
.
.
2
2
The search problem
Problem/Opportunity:
~3000 parameters per E+ input file
2 minutes per simulation = 83 hours
3
3
ORNL High Performance Computing Resources
Titan:
299,008 CPU cores
18,688 GPU cores
710TB memory, distributed
Jaguar:
224,256 cores
360TB memory
Nautilus:
1024 cores
4TB shared-memory
Kraken:
112,896 cores
Gordon:
12,608 cores
SSD
4
4
HPC scalability for desktop software
• EnergyPlus desktop app
• Writes files during a run
• Uses RAMdisk
• Balances simulation memory
vs. result storage
• Works from directory of input
files & verifies result
• Bulk writes results to disk
Acknowledgment: Jibo Sanyal, ORNL
R&D Staff
5
5
Computational complexity
Problems/Opportunities:
Domain experts chose to vary 156
Brute-force = 5x1052 simulations
E+ Input
Model
E+ parameters
main_Tot
1172.5
None_Tot( None_Tot( HP1_in_To HP1_out_ HP1_back HP1_in_fa HP1_compHP2_in_To HP2_out_ HP2_back HP2_in_fa
1)
2)
t
Tot
_Tot
n_Tot
_Tot
t
Tot
_Tot
n_Tot
0
0
6.75
18.75
0
0
0
6.75
18
0
0
LoKU
13.75 billion years
Need 4.1x1028 LoKU
6
6
MLSuite: HPC-enabled suite of
machine learning algorithms
• Linear Regression
• Feedforward Neural Network
• Support Vector Machine
Regression
• Self-Organizing Map with Local
Models
• Regression Tree (using
Information Gain)
• Non-Linear Regression
• Time Modeling with Local Models
• K-Means with Local Models
• Recurrent Neural Networks
• Gaussian Mixture Model with
Local Models
• Genetic Algorithms
• Ensemble Learning
Acknowledgment: UTK computer science
graduate graduate Richard Edwards,
Ph.D. (advisor Dr. Lynne Parker); now
Amazon
7
7
Applications of machine learning
• EnergyPlus – 2-10 mins for an annual simulation
!- ALL OBJECTS IN CLASS
Version,
7.0;
!- Version
!- SIMULATIONCONTROL ===
SimulationControl,
No, !-Do Zone Sizing Calc
No, !-Do System Sizing Calc
…
• ~E+ - 4 seconds AI agent as surrogate model,
90x speedup, small error, brittle
8
8
Evolutionary computation
How are offspring produced?
Thickness
Conductivity
Density
Specific Heat
Bldg1
0.022
0.031
29.2
1647.3
Bldg2
0.027
0.025
34.3
1402.5
(1+2)1
0.0229
0.029
34.13
1494.7
(1+2)2
0.0262
0.024
26.72
1502.9
• Average each component
• Add Gaussian noise
• … “AI inside of AI”
9
9
Getting more for less
• EnergyPlus is slow
– Full-year schedule
– 2 minutes per simulation
• Use abbreviated 4-day schedule instead
– Jan 1, Apr 1, Aug 1, Nov 1
– 10 – 20 seconds per simulation
r = 0.94
Monthly Electrical Usage
r = 0.96
Hourly Electrical Usage
10
10
Evolutionary combination
Island Hopping
1.
2.
3.
4.
4 of 19 experiments
Surrogate Modeling
Sensor-based Energy
Modeling (sBEM)
Abbreviated Schedule
Island-model evolution
11
11
Automated M&V process Autotune
calibration of simulation to measurements
XSEDE and DOE Office of Science
DOE-EERE BTO
Industry and building owners
Commercial Buildings
Monthly
utility data
Hourly
utility data
Features:
Works with “any” software
Tunes 100s of variables
Customizable distributions
Matches 1+ million points
CVR
NMBE
CVR
NMBE
ASHRAE
G14
Requires
Autotune
Results
15%
5%
30%
10%
0.32%
0.06%
0.48%
0.07%
Residential Tuned input
avg. error
home
Within
Hourly – 8%
30¢/day
Monthly – 15%
(actual use
$4.97/day)
10+ companies interested
Leveraging HPC resources to calibrate models for optimized building efficiency decisions
12
12
HPC-informed algorithmic reduction…
to commodity hardware
1 hour
LoKU
13.75 billion years
Need 4.1x1028 LoKU
13
Autotune
Implementation
Joshua New
Oak Ridge National Laboratory
How can I get/use Autotune?
• OpenSource GitHub - https://github.com/ORNLBTRIC/Autotune
• Demo
• Website
• Web service
• All loaded on vagrant image
Parameters
Tuned Models
Service
Response
User
Parameters
Tuned Models
Frontend
Insert Job
Get Status
Database
Get Next Job
Request
Scripted System
Insert Tuned Models
Website
Backend
Website
Website
Website
Website
Website
Website
Website
Website
Website
Web Service
• jsonString = tune(userdata, basemodel, schedule,
parameters, weather, email)
• Returns JSON containing tracking number and queue position
• jsonString = retune(trackingNumber)
• Returns JSON containing tracking number and queue position
• jsonString = terminate(trackingNumber)
• Returns JSON containing tracking number, queue position, total
runtime, and an array of model IDs and fitness values
Web Service
• jsonString = getOutput(trackingNumber)
• Returns JSON containing tracking number, queue position, total
runtime, and an array of model IDs and fitness values
• jsonString = getFullOutput(trackingNumber)
• Returns JSON containing tracking number, queue position, total
runtime, and an array of model IDs and fitness values, along with
all the inputs (base model, weather, etc.)
• jsonString = getModel(trackingNumber, modelNumber)
• Returns JSON containing tracking number, model ID, model file,
and model fitness
Example Web Service Output
{"autotune":
{"tracking":"8765",
"position":-1,
"weather":"USA_AL_Birmingham.epw",
"email":"tinman@oz.com",
"basemodel":"contents-here",
"parameters":"contents-here",
"schedule":"contents-here ",
"userData":"contents-here",
"terminate":0,
"runtime":12345,
"models":[
{"id":1, "fitness":0.76, "file":"contents-here"}
{"id":2, "fitness":1.23, "file":"contents-here"}]
}
}
Trinity Idea and
Implementation
Joshua New
Oak Ridge National Laboratory
Trinity Testing
• Question: How well does Autotune recover true inputs?
• Methods
• Use modified DOE reference buildings to generate E+ output
• Use Autotune to recover the modifications given the output
• Known to Autotune
• 3 DOE reference buildings (warehouse, medium office, stand-alone
retail)
• specific inputs that were modified (e.g. Infiltration Flow Coefficient)
• encompassing range of modification (+/- 30%, for instance)
• EnergyPlus output
• Specific input variable values known only to Josh New
Trinity Test – evaluates any calibration procedure
• Trinity Test (bit.ly/trinity_test) website and web service online
• Being used by other national labs and organizations
Test Creator
True
Model
Weather
Schedule
True
Model
Test Creation
Energy
Plus
True
Output
Base
Model
Weather
Schedule
Test Selection
Results Extraction
Model Evaluation
Base
Model
Evaluator
Calibrator
Predicted
Model
Base
Model
True
Output
Weather
Schedule
Discussion
Oak Ridge National
Laboratory
Joshua New, Ph.D.
newjr@ornl.gov
32