Benchmarking IT: The Cost/Quality
Balance
Chris Smith
Notes accompany this presentation. Please select Notes Page view. These materials can be reproduced only with written approval from Gartner. Such approvals must be requested via e-mail: vendor.relations@gartner.com. Gartner is a registered trademark of Gartner, Inc. or its affiliates.
Comparing IT Performance
•
•
While high level metrics are helpful and can be used to communicate the
value of IT - remember
•
They are database averages from surveys and
•
Do not necessarily reflect your environment’s intricacies
A Benchmark provides a more accurate view of your spending & staffing
levels as it compares your organization to a set of peers with similar
•
Service Levels
•
Environment complexity
•
Workload
•
Number of locations supported
•
Geographic location
•
Level of outsourcing
Specific Measures and Definitions
 Devices
 MIPS
 Minutes
 Volume
 Users
Cost
“Workload” represents a service
provided and is balanced with cost.
Workload
Gartner collects data in line with
its industry standard
“Consensus Models” to ensure
comparability
 Software
 Hardware
 Personnel
 Occupancy
 Transmission
To compare with actual spending, an
organisation’s workload is multiplied by
their peers’ average unit cost:
 Based on operational
expense
 Labour not depreciated
12,500
x £900
£11,250,000
The organisation’s
PC Count
Peers’ average
cost per PC
Peers’ cost for supporting
the organisation’s PCs
Key Principles of Benchmarking
Cost per Desktop
The diagram below serves to illustrate the fundamental flaw in the “database average” approach to
benchmarking: Company B appears to spend more than Company A when viewed at the simplistic “cost per
desktop” level. However, when a more contextual analysis is performed and the degree of complexity is factored
in (the red line represents the extrapolated trend line from a Gartner benchmark database), then it becomes
apparent that Company B is actually the more cost efficient, and Company A spends more than other
organisations with the same degree of complexity.
B
A
Low
Complexity / Workload
“Cost difference”
High
Government Trends in IT Spend to Run,
Grow and Transform the Business
• The distribution of
IT spending
between run, grow
and transform
activities has been
similar for the last 4
years
IT Spending to Run, Grow,
and Transform the Business
• The proportion of
spend across these
three categories is
very similar for
National and Local
government
Data Source: Gartner IT Key Metrics Data 2010
What if You Are Outsourced?
Determination of whether an overall deal is
good requires not just a comparison to
some high level spending numbers but also
some measures across other dimensions:
Is the deal generating the business outcomes
expected?
- Customer Satisfaction…measures
here can provide indications of how
the provider is doing and whether
services meet the needs and
priorities of the business.
- Relationship Assessment…provides
a view on the levels of trust and
control in managing the deal.
- Cost Assessment…in an open
relationship provide insight into
whether and how efficiencies in
service delivery can be gained;
across roles played by both service
provider and service recipient
- Business Alignment… focussing on
innovation and developments is the
deal delivering the business
improvements expected.
Is the service provider
delivering the service
outcomes expected, and is
pricing still competitive?
Are both operational and
management staff happy
with the day-to-day and
medium-term "outcomes"
being generated?
Is the working relationship healthy?
Is the service provider meeting the
contractual requirements?
Changing the Conversation from IT Cost
to IT Value
•
All CIOs must manage a diverse set of
stakeholders
•
The toughest stakeholders are those who
hold or control the IT budget directly and
indirectly
•
Obtaining clarity over which stakeholders
maintain this IT budget control is vital
•
IT is expected to play one of 4 specific and
different roles in any organization
•
The CIO does not always/often/ ever choose
the role IT is expected to play
•
Controlling the conversation about the value
of IT, within its chosen role, is critical to the
CIO’s survival and the strategic impact IT
can bring to the business through innovation
CIO Scorecard Results at a Glance
The overall performance of the IT
organization at XYZ is rated better
than the average organization.
Business
IT
RUN
GROW
TRANSFORM
=
EFFICIENCY
-
Overall
+
(relative to peer)
EFFECTIVENESS
Better than average
performance is found in the
areas of Cost Containment and
Infrastructure & Operations.
Average performance is found
in the areas of applications
organization and enterprise
architecture.
An improvement area is found
in business process
management.
INNOVATION
The effectiveness and innovation ratings for XYZ are slightly above average.
How to Compare
•
Peer groups are determined based upon the role that IT plays in the enterprise
•
The database demonstrates clear differentiation in the average performance in the underlying
maturity models for “Team Players” and “Grinders.” XYZ is compared to other “Grinder”
organizations based upon current role and “Team Player” based upon aspiration for each CIO
Scorecard evaluation component.
Evaluation Component
XYZ
Grinder
Team
Player
Infrastructure & Operations
2.83
2.39
3.11
Cost Containment
3.44
2.04
2.59
Applications Organization
2.68
2.69
3.88
Business Process
Management
1.40
1.28
1.60
Enterprise Architecture
2.20
2.38
3.12
Open Innovation Readiness
1.25
2.85
3.28
Effectiveness/Innovation
Enterprise Viewpoint
3.27
2.58
3.84
Effectiveness/Innovation IT
Organization Viewpoint
3.56
2.65
3.89
*Rating based upon 1 to 5 scoring with 5 as most mature
The Major Dimensions of IT Performance
•
There are a number of
dimensions in which IS
performance can be managed.
These dimensions are interrelated and over-focus on one
will adversely affect the others
- Process
- Value for Money
- Alignment
- Service Availability
- Service Quality
Service Level Attainment – Sample Output
 XYZ availability and
response service level
are competitive with
peer organisations
 End-user-facing service
levels, such as help
desk first call
resolution, tend to fall
below average peer
performance as well as
XYZ targets
Mainframe
Availability
Mainframe
Response
98.0%
98.5%
99.0%
99.5%
UNIX
Availability
98.0%
98.5%
99.0%
99.5%
Peer Average:
4 sec
100%
Wintel
Availability
2 sec
1 sec
0 sec
3 sec
2 sec
1 sec
0 sec
Wintel
Response
98.0%
98.5%
99.0%
99.5%
100%
Network
Availability
4 sec
3 sec
2 sec
1 sec
0 sec
250%
200%
150%
100%
50%
250%
200%
150%
100%
50%
4 days
3 days
2 days
1 day
0 days
Apps. % of
Time Plan
98.0%
98.5%
99.0%
99.5%
100%
App. % of
Budget Plan
Telecom.
Availability
98.0%
98.5%
99.0%
99.5%
100%
Voicemail
Availability
98.0%
4 days
XYZ Target:
3 sec
UNIX
Response
Apps. S1
Restore
98.5%
99.0%
99.5%
100%
MAC
Timeframe
XYZ Actual:
4 sec
100%
Apps. S2
Restore
3 days
2 days
1 day
0 days
Desktop S1
Restore
16 hrs
3 days
2 days
1 day
0 days
25%
50%
75%
100%
60 sec
30 sec
0 sec
First Call
Resolution
12 hrs
8 hrs
4 hrs
0 hrs
Desktop S2
Restore
16 hrs
4 days
0%
Mean Time
to Answer
12 hrs
8 hrs
4 hrs
0 hrs
120 sec 90 sec
Process Maturity – Sample Output
0
 XYZ exceeds database
average maturity levels in two
thirds of the assessed
disciplines.
 IT governance and relationship
management are areas of
particular strength, suggesting
a strong focus on business
value.
 In contrast, low maturity for
asset management and
business continuity may
indicate risk for higher cost and
greater downtime
Asset Management
Bus. Continuity and DR
Capacity Planning
Change Management
Configuration Management
Database Administration
Facilities Management
Incident Management
Inventory Management
IT Financial Management
Monitoring
Performance Management
Problem Management
Release Management
Security Management
Service Desk
Service Level Management
Storage Management
Task Management
Relationship Management
Client XYZ
Peer Average
IT Governance
1
2
3
4
5
Business Alignment – Sample Output
IMPORTANCE
5
4
3
2
SATISFACTION
1
1
2
3
4
5
•
Management assigns the
greatest satisfaction to
reducing operating cost—a
strong XYZ focus in years past
•
An increase in industry
competition, however, has
shifted focus from cost savings
to competitive advantage,
including improving time to
market and enabling revenue
growth
Faster Time to Market
Competitive Advantage
Enable Revenue Growth
Reduce Operating Cost
Improve Customer Support
Reduce Bus. Disruptions
Reduce Direct IT Cost
Enhance Business Agility
IT Service Satisfaction – Sample Output

Consistent with Process maturity findings, line of business managers are
concerned with XYZ’s preparedness in terms of disaster recovery

Managers gave Voice and Data communications their highest satisfaction
rating—4.44
Satisfaction
Importance
5.00
4.06
4.00
4.44
4.32
3.89
3.44
4.00
4.20
3.51
3.78
3.64
3.58
3.55
3.00
2.00
1.00
ni
x
U
ec
ov
er
y
Se
rv
ic
es
Se
rv
er
Pu
Pl
rc
at
ha
fo
se
rm
d
s
En
te
rp
ris
e
Ap
M
ps
ai
nf
ra
m
e
Se
rv
Vo
ic
es
ic
e
&
D
at
a
C
om
Ap
m
pl
s
ic
at
io
ns
S
W
up
in
po
te
lS
rt
er
ve
rP
la
tfo
D
rm
es
s
kt
op
P
H
C
el
p
Se
D
rv
es
ic
k
es
Su
pp
or
tS
Ap
er
pl
ic
vi
at
ce
io
s
ns
D
ev
el
op
m
en
t
IT
C
ha
rg
eb
R
ac
em
k
ot
e
PC
Se
rv
ic
es
D
is
as
te
rR
Se
cu
rit
y
Se
rv
ic
es
-
3.79
Help Desk Quality
Help Desk Productivity
Desktop Productivity
Benchmarking IT: The Cost/Quality
Balance
Chris Smith
Notes accompany this presentation. Please select Notes Page view. These materials can be reproduced only with written approval from Gartner. Such approvals must be requested via e-mail: vendor.relations@gartner.com. Gartner is a registered trademark of Gartner, Inc. or its affiliates.