Management of
Computer System
Performance
Chapter 9
Evaluation of the IT Function
Evaluation of the IT Function
Agenda:

Chapter 9
Objective:

Students should be able: to explain the
implications arising from the analysis.
2
Evaluation of the IT Function


The evaluation of IT services and functions
requires applying the proper metrics against
actual and perceived effectiveness.
There are several levels of this including
concepts previously discussed such as;


providing appropriate targets and measuring
them against actual data.
Measuring actual and targeted data against
industry and competitive data.

Ex-post and ex-ante evaluation methodologies.
3
Evaluation of the IT Function

To measure effectiveness of an IT Function
(department wide), it is necessary to view the IT
department from a fixed perspective. Does the IT
department;


Provide desktop support and workgroup server support or
Does it manage back-office systems, CRM, ERP/ERM,
SCM, POS E-Commerce and LAN, MAN, WAN and WAP
services while steadily increasing operational efficiencies
at lower departmental costs directly contributing to the
bottom line of the companies revenue base.
4
Evaluation of the IT Function


Effectiveness also entails the quantification of soft
and hard estimates or direct and indirect factors.
As part of that consideration of services offered.
Examples of this include:





Is it meeting its objectives?
Are its projects on-time, to budget and meet
specifications?
Is the IT department a key factor in reducing costs and
adding revenues?
Do they maintain a constant level of service and support?
If these are accurate, does executive management
recognize these traits?
5
Evaluation of the IT Function Goals

There are two generic approaches to these view of
IT Functions,






A Goal Centered view and,
A Service Centered view.
Goals are usually set at either strategic or tactical
level.
They are implemented by the successful
completion of a series of tasks or projects.
The Goal is usually associated with changing the
steady state of the business or IT Function.
Change is the primary approach to this concept.
6
Evaluation of the IT Function Goals

A Goals centered evaluation system is most
commonly associated with the Program or Project
Management perspectives.




Scope it – Define the requirements and understand what
you will need to do.
Plan it – design a solution and plan its implementation to
solve for the requirements identified in the scoping
activity.
Execute it – Execute the Plan.
Close it – measure the success against the original goals.
Do lessons learned, etc. Was the project successful?
7
Evaluation of the IT Function Services


A service centered view of the IT Function is
targeted toward maintaining or increasing the level
of service currently provided.
This requires that a minimum level of service be
maintained.



Service levels are agreed to in advance by the IT
department and representatives of the user community.
Metrics include tracking up-time and service availability.
This is reduced to writing in the form of an SLA or
Service Level Agreement.
8
Evaluation of the IT Function Services

An SLA is usually measured by quantifying the
time the system is available or service is
rendered and systems or service response time.



Both of these elements must be factored into the SLA
baseline. As an example, if a system that is “available”
but has a 30 second response time, it might as well be
down.
There are variations to this theme but a 99.5% SLA
on a system’s availability is common .
This means that within the 99.5 SLA, an EBusiness solution can be “down” for about 3.6
hours a month [(30 day x 24 hours x 60 minutes) x
.005]
9
Evaluation of the IT Function Services

3.6 hours can be a significant amount of time.




However, consider that a three tier UNIX solution may
require each server to be booted in series (First one, then
the next and then the last one.)
A UNIX box can take 20 minutes to boot if it was a clean
shut down. Therefore, each reboot can cost and hour.
There are many reasons for a system to be unavailable
over the course of a month. This include everything from
installing patches to fixing software problems.
3.6 hours is not enough time if a system is down
late Friday night and your technicians are not
available (been there and done that too).
10
Evaluation of the IT Function Services

DD
There are numerous levels of SLAs
Hours
Extension
Allowable
Downtime
Reliability
Downtime/
Hours Per
Month
Downtime/
Minutes Per
Month
Data Center
Classification
1
30
24
720
0.05
95.00%
36
2160
2
30
24
720
0.01
99.00%
7.2
432
3
30
24
720
0.005
99.50%
3.6
216
4
30
24
720
0.001
99.90%
0.72
43.2
5
30
24
720
0.0005
99.950%
0.36
21.6
30
24
720
0.00001
99.9990%
0.0072
0.432
11
Evaluation of the IT Function Services

Availability is the function of the users needs, the
system design and the negotiated baselines.


This approach to Services evaluation greatly
simplifies the quantification of the value of IT
Functions.


The minimum levels of service required are documented.
It either meets the requirements or it doesn’t
Another approach to evaluating IT Functionality is
through the use of user satisfaction surveys.
12
Evaluation of the IT Function Surveys

Surveys vary in form and content but if users
complete them, they can be valuable resources.




In either case, the establishment of goals or levels of
services, the satisfaction of the users needs to be taken
into consideration.
The most common of which is through the use of
the Satisfaction Survey.
These provide both hard data and direct responses
but assumes certain factors.
To be successful, there are some basic premises
that must be met.
13
Evaluation of the IT Function Surveys



Is the sample base of users providing data sufficient in
number to be statistically sound.
Is the raw data collected non-biased.
 There are many, many factors which can skew data
results.
These factors include



getting the users to fill out the complete survey in a
thoughtful manner without outside distractions.
That the questions do not skew the results.
That the survey monitors do not skew the results.
 How would users react when being interviewed for their
user satisfaction by out of work actresses and actors
(pretty people) who would be be hired to interview them to
document their satisfaction with the services received. 14
Evaluation of the IT Function Surveys


Several examples from the book state “If Yours
answer was Good or Excellent, please give
details”.



Do the surveys collect the correct data?
Any problems here? It depends on your perspective.
For corrective action, you also need to know what didn’t
work.
The use of surveys also includes gathering data on
performance (good/bad) or perceptions of the
services importance to the firm (Critical/Major).

This type of data provides both a performance
perspective and a value to the company perspective of
the user community.
15
Evaluation of the IT Function Surveys

In the case of data collected on Performance
or Importance, corrective action can be
undertaken from the data collected.


In these cases, the quantification of the results of
the surveys, SLA compliance or meeting goals
through effective project management can be
quantified.
Over time, and if the baseline for measurement
remains the same, trends can and should be
plotted.
16
Evaluation of the IT Function Surveys

The methodology includes
obtaining a statistically valid samples,
determining the mean, median and standard
deviation of that data set and,
 comparing that data with the appropriate
measures.
To obtain this data requires time and effort which
does cost money.


17
Evaluation of the IT Function Surveys



As a result of the cost to compile the data, Surveys
are often reduced to a level that generates less
than optimal statistical data sets.
Accordingly, the in-depth (data mining) analysis of
the data is not performed for a variety of sound
mathematical reasons.
The reviews that are performed provide a rating
that is similar to the surveys themselves. The users
are:

Very Satisfied, Satisfied,dissatisfied or very dissatisfied.
18
Evaluation of the IT Function Surveys


Taken alone, this can give some value but there
could be significantly more obtained.
It is appropriate to undertake complex analytical
reviews of collected data but the mathematical
requirements that apply to any other data set that
is to be analyzed should apply to IT evaluations as
well.


Don’t change the standards because the data is difficult
to obtain.
Within the parameters identified herein, there are
many subcomponents that can be examined the
result in the data collected.
19
Evaluation of the IT Function Data


The data collected should be reviewed and
quantified.
The relationship and weighting of each elements
should be reviewed.


Statistical normalization may be a valid approach as well.
Each approach to the valuation should be
documented and agreed to by all parties in
advance.

First get your facts; then you can distort them at your
leisure. - Mark Twain
20
Evaluation of the IT Function Data



An on going theme of this class is to
quantify the data for decision making
actions based on financial reasoning.
The compiled Survey data, the Service
performance metrics and/or the
achievement of Goals should then be
matched with an economic value of the
offering.
Measure the benefit and then the cost and
make the decisions from that analysis.
21
Summary

SLAs measure response time or
availability.


They serve to confirm the expected value
received from the project.
Surveys, if properly constructed, can
provide valid information on the
satisfaction of the IT system or services
customer.

The construction is critical.
22
Homework
23