More for less
adam@metron.co.uk
More for less in a lean, mean, green,
virtualized ITSM world using capacity
management best practice
itSMF Norway
Sep 2010
Summary
1.
2.
3.
4.
5.
6.
itSMF Norway
Sep 2010
More systems
Less resources
Lean, mean & green
Virtualised
ITSM world – SDLC & ITIL
Capacity Management best practice
• Objectives; tasks
• I/O; CDB; activities
• Deliverables
• Monitor & analysis; publish, trend & advise
• Capacity Plan
• Mapping; distributions & queues
• Forecasting; modelling example
• CMMI; gap analysis
• Pyramid of needs.
3
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Survey – Do you use these acronyms?
• ITIL, ITSM, itSMF, itSMFI, itSMF UK, ITSML library
IT infrastructure, Service Management, Van Haren
• CMG, UKCMG, EuroCMG, CECMG, CMGAE, CMGI…
Computer Measurement Group
• SDLC, BPR
Software dev life cycle, Business Process Re-engineering
• CMMI, Six Sigma, Balanced Score Card, TQM, TQFM
Maturity model, lepto-kurtosis, process quality, ISO 9000
• Sox, Cobit, Basel II
Process audit and governance
• BCS, NCC, ISO/IEC 20000
Good practice, ethics, standards, certificates, pin badges.
4
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
More systems that have more:
• Critical business requirements
– IT service and web front end essential to delivery
• Users - Business customers, end-users and clients
• Services - Business customers’ view of applications
• Architectures - abstraction, virtualization,
consolidation
• Tiers & pools
– Web GUI, message handler, app server, warehouse…
• Applications
– Developments in-house, off-shore, packages…
• Servers
– Machines, nodes, CPUs, RAM, networks, storage
• But what is actually achieved in a business sense?
5
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Less resources and less:
•
•
•
•
•
•
•
•
Overhead – IT infrastructure and services
Finance –budgets cut
Sites – consolidated data-centres
Staff – cutbacks
Spare capacity – headroom, duplex, DR, non-stop
Physical servers – virtualised and consolidated
Expertise – less experience and less training
‘Inefficiency’, but at what cost to:
– Consistent, effective provision of essential services
– Increased risk of performance crises.
6
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Lean, mean & green
•
•
•
•
•
•
•
•
•
•
•
•
•
Lean - Focus on key objectives and core activities
Adopt 80/20 rule – identify busy services/users/servers
Exploit spare capacity, minimise headroom
Identify how much more traffic can be supported safely
Define and agree measurable SLA performance criteria
Virtualize infrastructure to avoid procurement by project
Mean - Optimise on current investment/reduce licences
Green - adopt solutions that are environmentally neutral
Centralise to minimise on wastage & reuse heat output
Use fewer, more powerful servers to reduce aircon needs
Reuse old machines – cascade to good causes
Grid and cloud computing…
Capacity on demand, use power when really needed…
7
#
of
30
Virtualised
Overhead
Difficulty of creation
& frequency (Hz)
Software-only:
VMware GSX
MS Virtual Server
OS-virtualization:
ESX, Xen, Hyper-V, paravirtualisation, grid, sysplex
LPARs (HP, IBM, Sun)
Fairshare, timeshare
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
z/VM, hyperthreading
(Intel VT, AMD V’n, IBM PowerPC
Software
Microcode
Hardware
8
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
SDLC – Development Life Cycle
Systems or service
or Software
Deliverables
Feasibility
Analysis
Design
Implementation
Testing
Deployment
Requirements
Architecture
Design
Implementation
Testing
Maintenance
TOR, Scope, PID, SOR
Functional Spec
SPE, System spec
Program spec, modules
system, a, b, load, trials
pilot, production, fix, retire
Steps: Review, Feedback, Prototype, Cyclic
Routes: Waterfall, Iterative, Scrum
Priorities: Project versus Process, Panic versus Procedure.
9
#
ITIL V2 & V3 Processes & Functions
of
30
Intro S Strategy S Design Transition Operation
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Improv’t
Financial M
Capacity M
Change M
Problem M
7 step CSI
Demand M
Availability M
Evaluation
Incident M
S Reporting
Continuity M
Validation
Event M
S Measure
Security M
Planning
Request M
Supplier M
Release M
Access M
SLM
Configuration
SPM/SCM
Asset M
Process?
Knowledge M
Service Desk
In V2 & V3
New in V3
Changed in V3
IT Ops M
Application M
Technical M
Function?
10
#
of
30
Six top entities/processes (6 steps)
Assets
Finance
Efficiency
Service Port/cat
Contracts
Availability, Capacity,
CMDB, CIS…
SLA, OLA,
Continuity…
CDB/CMIS
budgets…
Do it
Change
Start, deliver, stop
RFC, CAB, Release…
Operations service
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
N
Bug?
Repair
Y
11
#
6 steps applied to all of ITSM
of
30
Assets
Efficiency
Do it
Finance
Change
Bug?
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
ITSD Management of:
ITSS/O Management of:
Performance & Capacity
Incidents
Service Levels
Problems
Continuity
Events
Availability
Configuration
Supplier
Security
Finance…
Access…
12
#
of
30
Capacity management objectives
“The provision of a consistent, acceptable service level
at a known and controlled cost”
Objectives
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Right level of service performance
Right level of investment & resource
Resources matched to business need
Resolve bottlenecks, evaluate tuning
Forecast workload demands
Evaluate upgrade plans
Ensure timely procurements
Ensure effective SLA
Plan for growth, new apps, new sites
Provide performance assurance
Deliverables
Performance and capacity outputs
Demand management criteria
Business Workload forecasts
Service workload components
Event alarms & thresholds
Resource usage by domain
Capacity plans and acquisition plans
SLA targets and measures Application
sizing assessments
Performance test reviews
13 Capacity Management tasks
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
• Core
– Past performance trending & reviews
– Current analysis, alerting & reporting
– Future capacity planning & publishing
• Related
– Diagnostic monitoring, tracing & accounting
– Load balancing & system tuning
– Domain expertise and application
– Program optimisation
– Business requirement liaison
– SPE, application sizing, load testing
– Service Level Agreements and management.
14
#
of
30
Capacity Management I/O
Technology
SLA, SLR, OLA
Business plans
IT plans
Deployments
Development
Operations
Budgets…
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Outputs
Inputs
Business Capacity
Management
Service Capacity
Management
Performance reports
Capacity Plans
Baselines & profiles
Bottlenecks, patterns
SLA guidelines
Thresholds, alarms
New app sizing
Audit, costs & charges
Outcome
Resource/Component
Capacity Management
Control
Throughput
Feedback
Capacity Database
Economy
Standards
CDB/CMIS
Goals
Governance
15
#
of
30
Capacity management activities
Business CM
Service CM
Resource/
Component CM
Performance
activities:
Monitoring
Analysis
Tuning
Implement
Effective use of deliverables
= outcome
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Performance
Reports and
Capacity
Plan(s) to
intranet/email/
pager/paper
Reports
(regular, ad hoc &
exception) to and
from:
SLM
Thresholds
Alarms
Events
Charging
Audit
Demand
Management:
priorities,
quotas,
chargeback
Modelling
workload
forecasting
and
characteriz’n
Application
sizing,
performance
testing and
performance
engineering
Liaison with SLM & Development, test & QA,
Business: metrics, plans, KPIs
Capacity Database
Business and workload volumes
Platform and middleware statistics
Virtualization statistics
Hardware and RDBMS statistics
Detailed transaction statistics/ARM
Web/ intranet/ network traffic
ERP/User Application statistics
SLAs and new Systems sizing
16
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Monitor/analyse
17
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
William Playfair’s dual axis histogram 1821
Early use of correlation – wages vs. wheat over 250 years
18
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Publish, trend, advise
19
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Florence Nightingale’s polar pie-chart 1857
Crimean “coxcomb” in proportion to radius not arc
20 Mapping business needs to resource demand
#
of
30
Business
QoS, BMI
Weight
Dashboard
Drivers
SLA
Service
Service KPI
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Model
Plan
Trend/Alert
Publish
Applications(s)
Transaction(s)
App View
Server(s)
S
Metrics
Instrumentation
CPU RAM I/O N/W etc
2
y= (1/  * e-x /2 &  = 0 and  = 1
0.45
21
#
0.4
Distributions & queues
0.35
0.3
0.25
of
30
• Nature avoids normal distributions
0.2
0.15
• Computers abhor normal distributions
- Normal as derived by Gauss/de Moivre
0.1
0.05
0
-4
-3
-2
-1
0
1
2
3
4
6
7
y=e-x
• Computers like exponential distributions
- Typically a minimum value and a long tail
- Service times are typically exponential
- Inter-arrival gaps typically exponential
• Under typical enterprise conditions:
1.2
1
0.8
0.6
0.4
0.2
0
0
- Large populations & no priority/ class mix
- N queue server, FIFO and exponential applies
- “M/M/N” analytical models apply
- Utilization U and Service Time S
- Response Time R = S/(1-UN)
• And, in simple cases, if U=X% you will wait about
X% of the time
1
2
3
4
5
R=S/(1-U)
R=S/(1-U)&&S=1
S=1
12
12
10
10
88
R
R
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
66
44
22
00
00
0.2
0.2
0.4
0.4
0.6
0.6
UU
0.8
0.8
11
22
#
Knee?
of
30
R vs U plot for U=0 to U= 0.98 N=1,2,4,8,16
R vs U plot for U=0 to U= 0.98
R vs U plot for U=0 to U=0.9
N > 20 or so
Response Time
45
40
10
40
9
35
35
N=5
8
30
30
7
25
6
N=2
20
R
R
R
25
5
20
N=1
4
15
15
Service
Time
3
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
10
10
2
Utilization
5
5
1
0
0
0
0
0.1
0
0.2
0.3
0.4
0.2
N=1
0.5
0.6
0.4
U
N=2
N=4
0.6
UN=8
N=16
0.7
0.8
0.8
0.9
0.5
1
1
1.0
0
0
0.2
0.4
0.6
U
0.8
1
23 Forecasting options: mix and match
#
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
More potential accuracy and cost
of
30
Trust the
Vendor
Benchmarking
and application
trials
Workload
generation and
trials
Measurement &
Analytical
Modeling
Outsource it
Discrete event
Simulation
Trending
Behaviour
Patterns
Rules of
thumb
Do nothing
More Forecasting Effort
24
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Modelling example
25
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Charles Minard’s plot of Napoleon’s campaign
Tufte’s “top informative graphic”
1812 Russian campaign graphic done in 1869
Early use of line width to indicate bandwidth
26
#
of
30
The Capacity Plan –Sections
•
•
•
•
•
•
•
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
•
•
Management summary
Introduction
Assumptions
Business scenarios
Service summary
– List and brief description of business services supported
– Current and recent service provision and resource usage
– Service forecasts of business volumes
– Corresponding list of workloads & resource requirements
– Workload Forecast Scenarios
Resource summary
– Forecasts - account of impact of workload requirements
Options for service improvement
Capacity plans should be:
– Description of options for required upgrades
Short and clear
Cost model - quantifies finances for upgrades
Ad hoc, periodic, annual?
Per service, server or enterprise?
Recommendations
Based on QoS, BMI, KPI or OLA?
– Business benefits to be expected
Maximum 40 pages per service?
– Potential impact of carrying out recommendations
Max 2 page management summary?
– Risks involved
Right outputs on time to right people?
– Resources required
Targets met and metrics measured?
Produced as required and actioned?
– Costs, both set-up and ongoing.
27
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
CMP per app per site per stage
# CMMI
ITSM
CapMan
Task
%
5
Optimised bITa
Business level
Dashboard, CPM 2%
4
Measured ITSM
Service level
SLAM, cap plans 10%
3
Proactive Center
Resource level
J
CDB, Trends, web 30% L
2
Reactive
Tickets
Analysis
Utilization, uptime 55%
1
Ad hoc
Help calls Monitor
Ad hoc alerts
3%
28
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Summary of gap analysis procedure
• Agree scope of study and checklist of targets
– Enterprise Infrastructure, Service/App Portfolio…
– Mainframe, warehouse, UNIX, Linux, Windows…
– LAN, WAN, SAN, PAN, RDBMS, ERP…
• Define objectives, constraints, resources available
• Assess versus relevant checklists for Best Practices
• Review actual Capacity Management Practice (CMP)
• Talk to the Capacity Management Team(s) (CMT)
• Readily available reports collected for analysis
• Interviews within a week to assess current activities
• Review available evidence/ deliverables
• Reveal gaps, identify SWOT and next steps
• Snapshot report in a week for total focus on material.
29
#
of
30
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Risks, costs & benefits for CMP
• Benefits
• Risks
– Improved service provision at cost justifiable service quality
– Over expectation of IT by customers
– Services that meet Business, Customer and User demands
– Too much hardware vendor influence
– Increased efficiency – cost savings
– Lack of business information
Deferred
expenditure (cash flow)
– Lack• of
human upgrade
resources
• Consolidation
(maintenance/licences)
– Complex
environment
• Costs • Planned acquisition (discounts)
– Reduced risk
– Procurement of required tools
• Fewer
performance
disasters (customers)
– Project
management
as required
Fewerincluding
performance
problems
(users)
– Staff• costs
recruitment,
training
• Longer Application
Life-cycle (fewer abandoned apps)
– Accommodation
etc
• Learning
from previous
experience,
Good
– Liaison
and interface
with business,
dev, QA
etc.Practice
30 Pyramid of needs
#
Potential
benefit
to enterprise
of
30
Full
Control
Plan
Prediction
App size
Optimal usage of
available resources
Model
Basic pre-emption of Problems
Capacity
Management
for
Virtual
Systems
itSMF
Norway
Master
Sep
2010
Class
V001
Monitoring & Basic Control
Demand
Tuning
Analysis
Monitor
CDB
Acquisition of relevant metrics / Context
related knowledge
Disorder / Lack of control
More for less
adam@metron.co.uk
More for less in a lean, mean, green,
virtualized ITSM world using capacity
management best practice
itSMF Norway
Sep 2010