Heiko Koziolek, DECRC/I1 Ladenburg, Germany, 2014-11-13
6 Years of Performance Modeling
at ABB Corporate Research
© ABB Group
December 3, 2014 | Slide 1
My Story Today
© ABB Group
December 3, 2014 | Slide 2
2008: Performance Modeling with Palladio
Overview
1)
Measure
System instrumentation
& performance tests
using load drivers
(custom tooling)
2)
Model
Component-based
model with annotated
flow charts entered into
Eclipse-based tooling
3)
Predict
Running
analytic solvers /
simulators, varying
model parameters to
test different situations
© ABB Group
December 3, 2014 | Slide 3
Model derivation
Prediction
(manually)
(automatically)
2010: Q-ImPrESS & Industrial Control System
© ABB Group
December 3, 2014 | Slide 4
2010: Q-ImPrESS
Models & Tools
SoMoX
Q-ImPress Workbench

SoMoX / Sissy for Reverse Engineering Component Models from C++

Windows Performance Monitor for Performance Measurement

Self-implemented C#-Client as load driver

Q-ImPreSS Workbench for Modelling (meta model similar to Palladio)

LQN solver / Palladio SimuCom for Performance Prediction

PerOpteryx for Design Space Exploration
© ABB Group
December 3, 2014 | Slide 5
2010: Q-ImPrESS
Results
Koziolek, Schlich et al.
An industrial case study on
quality impact prediction for
evolving service-oriented
software.
In Proc. ICSE 2011 SEIP,
pp. 776-785. ACM, May 2011.
© ABB Group
December 3, 2014 | Slide 6
2010: Q-ImPrESS
Lessons Learned


© ABB Group
December 3, 2014 | Slide 7
Successes

Large performance model build with Q-ImPreSS tooling

Models validated through measurements (<30% error)

First experiments with design space exploration
Challenges

Not enough inputs on new ABB system available,
had to fallback to model older version
predictions for older system are not really actionable
as the older version will not be changed

Modeling tools disconnected from the tools currently
used during development (e.g., Enterprise Architect)
creating models with the tools from scratch required
high effort

Static code analysis challenged by Microsoft C++ code
2012: Performance Modeling for ABB Robotics
© ABB Group
December 3, 2014 | Slide 8
2012: Performance Modeling for ABB Robotics
Models & Tools
Palladio Workbench
LQN Solver

Dynatrace for distributed performance profiling

Neoload as load driver

Palladio Workbench for modelling ‚
(all manual no static code analysis)

LQN/SimuCom for performance prediction

PerOpteryx for design space exploration
© ABB Group
December 3, 2014 | Slide 9
2012: Performance Modeling for ABB Robotics
Results
© ABB Group
December 3, 2014 | Slide 10
Thijmen de Gooijer, Anton Jansen, Heiko Koziolek, and Anne Koziolek. An industrial case study of performance and cost
design space exploration. In Proc. 3rd Int. Conf. on Performance Engineering (ICPE'12), pp 205-216. ACM, April 2012.
2012: Performance Modeling for ABB Robotics
Lessons Learned


© ABB Group
December 3, 2014 | Slide 11
Successes

Due to performance fixes based on the measurements
the performance could be improved by 50%

Roadmap for extending the system cost-effectively was devised
based on the models

Large-scale industrial case study on design space exploration of
a distributed, component-based system

ABB Robotics integrated Dynatrace into their development
environment
Challenges

Information extraction for the models took a long time, lots of
calibration needed, several assumptions required for models

Expensive measurement & testing tools (>30K€)
2014: Automation Cloud
© ABB Group
December 3, 2014 | Slide 12
2014: Automation Cloud
Models & Tools
© ABB Group
December 3, 2014 | Slide 13

Amazon Web Services / Own Cloud Server as test
environment (up to 36 AWS m1.large instances)

KairosDB, OpenTSDB, Databus as time-series databases

Apache Cassandra and Hbase as distributed DBMS

Netflix Priam / Apache Whirr for quick deployment

Visual Studio Ultimate Web Load Test as load driver

[No modeling, only benchmarks!]
2014: Automation Cloud
Results
ABB Phasor Measurement Unit
used in Power Grids
ABB Smart Meter
Avg. Roundtrip Time: 193ms
for 15 customers
Linear scalability
for KairosDB
Limited overload WITH
AWS Autoscaling
© ABB Group
December 3, 2014 | Slide 14
Thomas Goldschmidt, Anton Jansen, Heiko Koziolek, Jens Doppelhamer, and Hongyu Pei-Breivold. Scalability
and Robustness of Time-Series Databases for Cloud-Native Monitoring of Industrial Processes. In Proceedings
7th IEEE Int. Conf. on Cloud Computing (IEEE CLOUD 2014) Industry Track. IEEE, July 2014.
2014: Automation Cloud
Lessons Learned


© ABB Group
December 3, 2014 | Slide 15
Successes

Showed technical feasibility for several scenarios from
industrial automation in a cloud computing environment

Created benchmarks for time-series databases based
on realistic workloads from ABB products

Created elasticity metrics and benchmark
Challenges

Better testing needed to improve robustness

Component-based model did not well fit with databases
/ cloud platform (e.g., auto-scaling?)

Limited insights expected from modeling due to focus
on initial technical feasibility
2015 Outlook: Collaboration with Uni Würzburg
Automatic Construction of Architectural Perf. Models

Kieker + C# adapter / JNBridge for distributed profiling

LibReDe for resource demand estimation

.NET Bookstore / Pet Shop (C#) for testing, later ABB system

Palladio / Descartes / PerOpteryx for modeling / prediction
© ABB Group
December 3, 2014 | Slide 16
6 Years of Performance Modeling at ABB
Conclusions

Performance modeling has matured over the last 6 years

But: to get wider adoption
lower costs and higher benefits
are required.
© ABB Group
December 3, 2014 | Slide 17
6 Years of Performance Modeling at ABB
Future Work


© ABB Group
December 3, 2014 | Slide 18
Future work for lower costs

Better integration between measurement and modeling tools

Faster modeling via more convenient software tools

Faster modeling via reusable model libraries
Future work for higher benefits

More performance questions to be answered

Decision support and incorporation of heuristics

Better integration into existing development processes & tools