Measurement and Evaluation
S38.4030
Measurement and Evaluation
Yang Qiu
Networking Laboratory
Helsinki University of Technology
{yangqiu}@cc.hut.fi
2nd Jun, 2006
1
Measurement and Evaluation
S38.4030
Autonomic Computing or
Communications
 Autonomic Computing or Autonomic Communications is
a concept that brings together many fields of computing
with the purpose of creating computing systems that are
reflective and self-adaptive
 When reviewing the current autonomic systems, the
concept of self management usually groups into having
four basic properties:
–
–
–
–
Self-configuration
Self-optimization
Self-healing
Self-protection
2nd Jun, 2006
2
Measurement and Evaluation
S38.4030
Software architectures for
autonomic computing
 The autonomic research activities in software
systems can broadly be categorized into four areas:
–
–
–
–
monitoring of components
interpretation of monitored data
creation of a repair plan (i.e. an adaptation of the system)
execution of a repair plan
And we can group the approaches to autonomic
computing systems into categories:
– intelligent multi-agent systems
– architecture design-based autonomic systems
– Hot swapping components
2nd Jun, 2006
3
Measurement and Evaluation
S38.4030
researches and the domain of
Autonomic computing
Main characteristics of Multi-agent systems
Domain
Group
Communication middleware based on CORBA
for monitoring and cooperation
Framework for multiagent systems
Kuo-Ming,
James,
Norman
Heartbeat or pulse monitor for monitoring
Autonomic
components
Sterritt,
Bustard
Self-organizing components with a global view
expressed as architecture description
Architectural
constraints for selforganizing
components
Georgiadis,
Magee,
Kramer
Broker agents used as to provide fault tolerance
to overlying problem-solving agents.
Adaptive Agent
Architecture
Kumar, Cohen
Framework for building multi-agent systems.
Working on including autonomic agents.
ABLE agent toolkit
Bigus et al.
2nd Jun, 2006
4
Measurement and Evaluation
S38.4030
researches and the domain of
Autonomic computing
Main characteristics of Architecture designbased autonomic systems
Domain
Group
Probes, gauges for monitoring running system,
architecture manager implements adaptive
behavior, based on architecture-model of
system.
Architecture model
based adaptation for
autonomic systems
Garlan,
Schmerl
Components considered as black-boxes.
Architecture for fault
tolerance in adaptive
systems
de Lemos,
Fiadeiro
xADL 2.0 architecture description language,
c2.fw development framework.
Framework for
architecture-based
adaptive systems
Dashofy, van
der Hoek,
Taylor
Autonomic behavior as a distributed multi-agent
infrastructure called Work flakes.
Adding autonomic
behavior to existing
systems
Valetto, Kaiser
2nd Jun, 2006
5
Measurement and Evaluation
S38.4030
researches and the domain of
Autonomic computing
Main characteristics of Hot swapping
components
Domain
Group
BARK tool as an extension to EJB to support
component replacement.
Reconfiguration in
EJB (Enterprise
JavaBeans) model
Rutherford et
al.
Adaptivity through replacement of bindings
between operations and invoked code blocks.
Model for
reconfigurable
software
Whisnant,
Kalbarczyk,
Iyer
High-performance hot-swapping of fine-grained
components in K42 OS.
Hot-swapping at OS
level
Appavoo et al.
DynamicTAO, a middleware for dynamically
reconfigurable software.
Reflective
middleware
Kon, Campbell
et al.
OpenORB, reflective middleware for self-healing Reflective
systems.
middleware
2nd Jun, 2006
6
G. S. Blair et
al.
Measurement and Evaluation
S38.4030
Metrics and evaluation
 Quality of Service (QoS)
–
–
QoS is the top-level means to compare modern systems – it should reflect the degree to
which the system is reaching its primary goal.
It is typically composed of a number of metrics:
•
•
•
Bandwidth
Delay
Jitter …
 Cost
–
–
–
Currently most performance studies of architecture-based autonomic systems have
measured its ability to reach its goal
agent-based systems typically compare the amount of communication, actions performed,
and cost of actions required to reach the goal
For many commercial systems the cost cannot be measured immediately
 Granularity/Flexibility
–
There is a chance of overhead in terms of the global system
•
•
The actions of adaptation are tolerable in a thick-grained component based architecture where the
overheads can be hidden in the system’s overall operation and potentially change is not that regular.
The actions of adaptation are potentially too much for a finer-grained architectures, such as an Hot
Swapping OS or Ubiquitous computing.
2nd Jun, 2006
7
Measurement and Evaluation
S38.4030
Metrics and evaluation (continue)
 Failure avoidance (Robustness)
–
–
Predictability of failure is an aspect in evaluation
The ability to cope with predicted failure
•
–
using a mean time before failure metric of hardware and others to cope with unpredicted environments
The ability to cope with unpredicted failure
 Degree of Autonomy
–
–
failure avoidance
AI and agent-based autonomic systems primarily as their autonomic process
•
•
–
Decreasing the degree of predictability in the environment and seeing how the system copes could
measure this.
Lower predictability could even reach it having to cope with things it was not designed to.
A degree of proactivity
 Adaptivity
–
Adaptivity is something simple as a parameter of begin changed in a system
•
some systems are designed to continue execution reconfiguring, while others cannot. So the
adaptation ( for example hot swapping ) of a node impacts the performance
2nd Jun, 2006
8
Measurement and Evaluation
S38.4030
Metrics and evaluation (continue)
 Time to adapt and Reaction Time
–
–
The time to adapt is the measurement of the time a system takes to adapt to a change in the
environment
Reaction time is the time between when an environmental element has changed and the
system recognizes that change, decides on what reconfiguration is necessary to react to the
environmental change and get the system ready to adapt
 Sensitivity
–
This is a measurement of how well the self-adaptive system fits with the environment it is
sitting in
•
If a system is highly sensitive to its environment potentially it can cause the system to be constantly
oscillating.
 Stabilisation
–
–
–
related to sensitivity
For open autonomic systems the sensitivity is learning how to best reconfigure the system
For closed autonomic systems the sensitivity would be a product of the static rule/constraint
base and the stability of the underlying environment the system must adapt to
 Benchmarking
–
Benchmarking is to bring above metrics together to form some sort of benchmarking tool.
2nd Jun, 2006
9
Measurement and Evaluation
S38.4030
Study case 1: Kendra audio server
 Kendra is a relatively simple system with closed selfadaptation
– set the top-level QoS goal to be that the audio quality was as
high as possible while avoiding periods of silence
– Also measured the general quality levels, unnecessary
adaptation, missed opportunities to adapt, and sensitivity to
environment
– the autonomic intelligence does not grow
 performance statistics were of a large volume and
difficult to interpret especially in terms of relating
behaviour to varying the many tuning parameters and
differing environment (networking) conditions
2nd Jun, 2006
10
Measurement and Evaluation
S38.4030
Study case 2: Measuring the
Effectiveness of Self-Healing
 It’s possible to quantify the self-healing capability with
two metrics:
– a measure of how effectively the System Under Test heals itself
in response to the injected disturbances
– a measure of how autonomic that healing response is
2nd Jun, 2006
11
Measurement and Evaluation
S38.4030
Study case 2: Measuring the
Effectiveness of Self-Healing
 Issues and discussion
– Quantifying autonomic maturity
• the benchmark’s maturity score to differentiate systems
• hope to ultimately map the quantitative measurement of maturity into benchmark index.
– Quantifying healing effectiveness
• Only the throughput of correctly-handled requests
• extended to capture broader impacts of disturbances
– Accounting for incomplete healing
• A complete self-healing cycle includes:
– bypassing the component affected by a disturbance,
– repairing the component,
– reintegrating them into the system Under Test
• A system Under Test might complete only some of these stages of healing in response
to a disturbance,
• A system Under Test might simply tolerate the disturbance without any active healing
process
2nd Jun, 2006
12
Measurement and Evaluation
S38.4030
Study case 2: Measuring the
Effectiveness of Self-Healing
 Issues and discussion (continue)
– Accounting for healing-specific resources
• A System Under Test may include resources dedicated to selfhealing capability. To prevent situations where a system is
intentionally over-provisioned in order to get a good self-healing
benchmark score, the cost of these extra resources must be
factored into the benchmark result
– Monitors
– hot standby cluster nodes
– spare disks
– Unified metrics
• Ultimately, the results of the self-healing benchmark being reported
on a 2-D space:
– one axis measuring effectiveness of self-healing as discussed above,
– one axis measuring the cost of the healing capability
2nd Jun, 2006
13
Measurement and Evaluation
S38.4030
Study case 3: QoS controller
Workload intensity variation vs. QoS Controller
Performance
Predictability of the workload
It is more important to correctly compare the
QoS-wise
– two points in the search space rather than knowing
their absolute QoS values
– AI is need to analysis and to generate a report or a
Benchmark
2nd Jun, 2006
14
Measurement and Evaluation
S38.4030
Study case 3: QoS controller


Picture above is the work load incoming.
Picture Right-upper is the evaluation of Average
QoS.
–
–
–


No Controller
Beam Search
Hill climbing
Picture Right-Lower is the evaluation of Average
QoS on different algorithm.
Average QoS will be higher if the predict algorithm
is accurate.
2nd Jun, 2006
15
Measurement and Evaluation
S38.4030
Conclusions
Autonomic computing is an engineering
concept that has found its way in a myriad
of computing fields
How to compose the benchmark is still a
question
2nd Jun, 2006
16