Management of
Computer System
Performance
Chapter 12
IT Surveys and Data Collection
Techniques
IT Surveys and Data Collection Techniques

Agenda:





Studies, Surveys and Alternatives
Identifying Intangible Value
How collection can be done
When should data collection be done.
Objective:

Students should be able: to analyze and
describe IT surveys into a job sheets. .
2
IT Surveys for benefits
measurement


Survey creation is a complex process. A
sample survey is provided for reference
purposes only.
It is strongly suggested that a great deal of
research be done to properly prepare the
survey instrument. This will ensure that:


The results received meet the requirements.
That the survey does not skew the resulting
data (leading questions, etc.)
3
Identifying Intangible Value

The process of identifying soft or intangible
costs or benefits is one of the most difficult in
performing a cost or benefit analysis.


It may also be the most important.
The goal is to assign value.

This requires identifying and quantifying
something that is often ancillary to hard costs or
benefits.
4
On-line banking

On-line banking can be used as an example
of this problem.

Some hard benefits and costs are as follows:
Tangible Costs
Tangible Benefits
On-line consumer
systems
Lower Labor costs (Fewer Tellers)
Added system security
Increased automatic transactions
Testing
Greater fee generation
Additional IT labor
Fewer/Smaller facilities (lower rent)
Marketing & Advertising Lower Insurance costs
5
On-line banking

Intangible benefits are initially intuitive or
perceived.

These could include:
Intangible Costs
Intangible Benefits
Consumer Dissatisfaction with
Consumer Satisfaction with
System/GUI
Not having to go to the bank/No
Lines
No personal contact
No personal contact
No one available to ask
questions
Simple transactions/One stop shop
Need a fast computer
Easy access
Keeping track of passwords
Better lifestyle “fit”
6
Identifying Intangible Value


The tangible values can be calculated either
directly readily estimated on hard values.
Intangible valuation requires additional data
gathering.


Considering the alternatives that offer the
greatest flexibility, surveys must be addressed as
a valid means to acquire measurable data.


Measured data will yield the best results.
Guessing isn’t acceptable.
The corollary is that surveys are not cheap.
7
Data Collection Costs



Task Costs – Manual Surveys
Labor Costs
Cover letter and questionnaire




Cover letter and survey printing costs



Envelope costs (both ways + more)
Postage costs (both ways + more)
Incentives Data entry and verification


design & typing
Purchasing mailing list cost (if necessary)
Addressing mailing envelopes
Statistical analysis software
Distribution of the final report
8
Data Collection Costs



Task Costs – Automated Surveys
Labor Costs
Planning




Questionnaire Design



Notification System
Participant identification
Incentives Data entry and verification


Specifications
Applications
Hardware
Statistical analysis software
Distribution of the final report
9
The Survey ProcessMethodology

There is a sequential
methodology that can be
used to optimize the
creation and use of a
survey.


Knowing what you want to
achieve and planning how to
get there are the best way to
succeed.
This approach does take time
and costs money.
10
The Survey Process – Goals

Step 1: Identify what you want to
accomplish

This step identifies the goal you want to
achieve. In this case, a valuation of one or more
intangible elements of a project.



A goal is not the same as a result.
A goal should be a neutral and not drive towards a
predetermined result.
Have a trusted third party check your work.
Predeterminations sneak in sometimes.
11
The Survey Process - Planning


Step 1: Planning Questionnaire Research
Consider the advantages and disadvantages of
using questionnaires.
Prepare written objectives for the research.
Have your objectives reviewed by others.
Review the literature related to the objectives.
Determine the feasibility of administering your
questionnaire to the population of interest.
Prepare a time-line.
12
The Survey Process - Design


Step 2: Design Methodology
Is it written, on-line web based, verbal or
application driven?






Consider the advantages and disadvantages of using
questionnaires.
Prepare written objectives for the research.
Have your objectives reviewed by others.
Review the literature related to the objectives.
Determine the feasibility of administering your
questionnaire to the population of interest.
Prepare a time-line.
13
The Survey Process - Feasibility






Step 3: Determine the Feasibility
Is it written, on-line web based, verbal or
application driven?
Can a survey collect the needed
information?
Can it be done in the time provided?
Can it be done with the available budget?
Are the topics too amorphous?
14
The Survey Process - Instruments



Step 4. Develop the Instruments
Is it written, on-line web based, verbal or
application driven? Target Audience?
Conduct "think-aloud" with several people.




Carefully select individuals for think-aloud.
Consider asking about 10 individuals to write detailed
responses on a draft of your questionnaire.
Ask some respondents to respond to the questionnaire
for an item analysis.
For electronic surveys, consider Heuristics for
neutral design and navigation.

Consider use tracking or cookie based systems for
response validation purposes.
15
The Survey Process - Sampling


Step 5. Select a Sample Set
Validate the questionnaire prior to full role out.


Avoid using samples of convenience.



Identify the accessible population.
Simple random sampling is a desirable method of
sampling.
Systematic sampling is an acceptable method of
sampling.
Stratification may reduce sampling errors.

Consider using random cluster sampling when every
member of a population belongs to a group.
16
The Survey Process - Sampling

Step 5. Select a Sample Set (cont.)



Consider using multistage sampling to select
respondents from large populations.
Consider the importance of getting precise results
when determining sample size.
Remember that using a large sample does not
compensate for a bias in sampling.
 Consider sampling non respondents to get
information on the nature of a bias.

The bias in the mean is the difference of the population
means for respondents and non respondents multiplied
by the population non-response rate.
17
The Survey Process – Pilot
and Revise

Step 6. Pilot Project



Using the sample set previously identified, roll out the
survey to this group.
Validate the results using statistical and other means.
Determine if the survey is collecting the correct data
and that the information is being accurately presented.
Step 7. Post Pilot Project Revisions



Based on inaccuracies discovered as a result of the
Pilot project, the survey instrument may need to be
modified.
Make the perceived modifications and repeat Step 6.
Note: Steps 6 and 7 may need to occur several times.
With each major revision, the project should be run
again.
18
The Survey Process-Research

Step 8. Conduct Research
 There are many aspects of this.




Commitment of individuals to participate in the Survey.
Advanced notification of the Survey
Instructions to the participants.
Survey scheduling, processing and collection.
•Cover letter and questionnaire
•Design & typing
•Purchasing mailing list cost (if necessary)
•Addressing mailing envelopes
•Following up on non-respondents
•Cover letter and survey printing costs
•Envelope costs (both ways + more)
•Postage costs (both ways + more)
•Incentives
•Data entry and verification
•Statistical analysis software
•Distribution of the final report
19
The Survey Process – Analysis

Step 9. Analyze data




Prepare a table of frequencies.
 Consider calculating percentages and arranging them in
a table with the frequencies.
Use the median as the average for ordinal data.
 Consider using the mean as the average for equal
interval data.
Use the median as the average for highly skewed,
equal interval data.
Use the range very sparingly as the measure of
variability.
 If the median has been selected as the average, use
the inter-quartile range as the measure of variability.
 If the mean has been selected as the average, use the
standard deviation as the measure of variability.
20
The Survey Process – Report


Step 10. Prepare Report
The final report must contain several conclusions
as to the valuation of the intangible elements
being examined.




Statistical Tables and Figures
Averages and Variability
Relationships
Margins of Error
21
Class 8a Summary




VFM looks at all internal aspects of IT
department operations.
HCR looks at all external aspects of an IT
departments impact on the corporation,
Both require IT management buy in and
commitment to the process.
The results of these types of analysis will identify
numerous items that can be “optimized”

Range from staffing changes to outsourcing.
22
Summary

Use surveys appropriately.



Match benefits with costs
Follow a methodology in
developing the survey.
Remember:
"Statistics are no substitute for
judgment." -- Henry Clay
23
Homework
24

References:


Text
Dr. Hossein Arsham
http://www.ubmail.ubalt.edu/~harsham/statdata/opre330Surveys.htm#rvsd

Designing Surveys and Questionnaires

http://www.statpac.com/surveys/index.htm#toc
25