HOPE
Research Methods
Collecting, Processing
and Analyzing Data
HOPE
Aims of the Session
The purpose of this session is to:
• Alert you to the different types of
methodology available to you in your
research
• Make you aware of the different
techniques that you might use in
collecting, presenting and analysing data
• Discuss the different kinds of problems
that you might encounter in pursuing your
research.
Research Methods Tony Fleet
2
HOPE
Contents
1.
2.
3.
4.
5.
6.
7.
8.
Developing Your Research Questions
The Different Types of Research
Selecting Appropriate Research Methods
Robustness of Methods
Structuring Your Methodology
Data Analysis
Problems with The Research Process
Summary
Research Methods Tony Fleet
3
HOPE
1. Developing your
Research Questions
This section of the presentation
examines what you need to do in order
to focus your research.
HOPE
The Purpose of Research
The purpose of research is to contribute to a
current academic debate, and possibly to advance
knowledge in some manner. This means that the
research that you undertake has to be :
1. Embedded in a recognisable field of study, taking
account of, and drawing on, past research
2. Of interest to other researchers working in the
same field, and possibly to the wider community
3. Generalisable to more than one individual
experience or circumstance.
Research Methods Tony Fleet
5
Typical Research Structure
Conduct Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
Apply Methods & Instruments
Perform Statistical Analysis
The process of research is
well-documented. This
diagram more or less
describes the activities
you need to undertake.
What we will do in this
session, is to look at
some of the elements,
and how they fit together.
Test Hypotheses & Draw Conclusions
Research Methods Tony Fleet
6
Getting Started
Conduct Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
The first thing that you
will do is to make sure you
are well-informed, and to
pose pertinent questions
which your research will
answer.
Apply Methods & Instruments
Perform Statistical Analysis
Test Hypotheses & Draw Conclusions
Research Methods Tony Fleet
7
HOPE
Defining Your Field of Study (1)
• You undertake a Literature Review in a particular
field, in order to ensure that your research is
embedded within that field, and that you are
taking account of the methods, issues, results,
theories and conventions which apply.
• At the end of the literature review, you will have
narrowed the field down to a relatively small topic
within the field, and will have some ‘unanswered
questions’ which your research will try to
investigate.
• These focussed questions form the basis of your
Research.
Research Methods Tony Fleet
8
Defining Your Field of Study (2)
HOPE
Your Research Questions are crucial, as they effectively
define both the content of your research and the manner
in which you carry it out.
 Your research methodology is designed specifically to
attempt to answer these questions
 The data that you collect will be focussed on issues
relevant to these questions
 Your analysis of the data will seek to provide answers
to the questions
 Your conclusions will summarise the answers.
Research Methods Tony Fleet
9
Research Questions
HOPE
Typical research questions might be:
– To what extent is the business community in
India aware of the potential of Bluetooth?
– Is the software currently available for teaching
arithmetic to 5 year olds appropriate and
effective?
– Are there differences in the way that men and
women approach the task of writing software?
– How can historical events be modelled
effectively using VRML?
Research Methods Tony Fleet
10
Defining Your Population
HOPE
• When framing your Research questions, you need to be
clear about what set of objects, people or events forms
the ‘background population’ in your study.
• Are you saying, for example that the CAL software you
produce is designed for all English-Speaking people, all
men, all Afro-Caribbeans, all children under 5, all those
who have been diagnosed as dyslexic, or simply to AfroCaribbean boys under 5 with specific learning
difficulties?
• If you are investigating whether on-line learning is
effective, is your population: students, University
students, UK University students, Liverpool Hope
Students, Liverpool Hope Computing Students or
Liverpool Hope MSc. Computing Students?
Research Methods Tony Fleet
11
HOPE
Research Questions
References (SW Library)
• Lewis, Ian. - So you want to do research! : a guide
for beginners on how to formulate research
questions. - 2nd ed. - Edinburgh : Scottish Council
for Research in Education, 1997. - (SCRE
publication ; 2 ; .. - 1860030327
Research Methods Tony Fleet
12
HOPE
2. The Different Types of
Research
Here we look at the different
options available to us in carrying
out the research.
HOPE
Your Research Focus
The main focus of your research can be:
• Product-based Research, focusing on producing a piece of
hardware or software (or the designs for them) which is at
the ‘cutting edge of a discipline, drawing on other
researchers’ ideas, best practice and what is feasible. In
doing this, you may need to explore how the product will
‘enmesh’ with current systems, and existing and future
technologies.
• People-based research, focusing on the people who
interact with the hardware or software, looking at issues
such as usability, user behaviour, compatibility of software
with current user systems and other HCI issues.
Both of these approaches are legitimate, and it is possible
that in carrying out your research you might need to use
elements of each one.
Research Methods Tony Fleet
14
HOPE
Product v. People
Which focus?
• The focus of your research is decided by you.
• It will depend upon how confident you are in creating a
product at the ‘cutting edge’, or how comfortable you will be
as a researcher in dealing with people.
• When you frame your research questions, you need to
ensure that their focus leads you into the kind of research that
you want to do.
• There is no right answer, but you may find that your research
will be best carried out by using a ‘main’ focus of one
element, with a subsidiary focus of another. For example
producing a piece of software which securely encodes
personal data as a self-encrypting and decrypting file stored
on an ID card, may well need trialling with real people.
Research Methods Tony Fleet
15
HOPE
Approaches to Research
There are two main approaches to doing research:
• Quantitative Research looks for ‘hard’ numerical
data; it proceeds by counting, measuring and
summarising. The goal is to look for statistical
significance in the results.
• Qualitative Research takes a ‘soft’ approach to
data, and has a more descriptive ‘feel’. It attempts
to get to the heart of the matter, exploring
individual cases in detail, and seeking the reasons
for behaviour and explanations for events.
Both of these approaches are legitimate, and it is
possible to combine elements.
Research Methods Tony Fleet
16
HOPE
Quantitative v. Qualitative
Which approach?
• The approach you use will depend upon your topic
and your research questions.
• It will also depend upon how comfortable you as a
researcher feel about using these methods.
• There is no right answer here, and, as we shall see
in the rest of this presentation, there may be good
reasons for adopting a variety of methods, which
encompass both quantitative and qualitative
approaches
Research Methods Tony Fleet
17
HOPE
Types of Research
There are four main types of research that you
might consider:
 Experimental Research
 Survey Research
 Evaluative Research
 Observational Research
All four of these types can incorporate both
quantitative and qualitative approaches
Research Methods Tony Fleet
18
HOPE
Experimental Research
This is normally quantitative, but can take two forms:
• An attempt to produce a piece of hardware,
software or a combination of both, which is at the
‘cutting edge’ of a discipline.
• An attempt to investigate and document the
performance of a particular piece technology in
specific circumstances.
This might involve:
• Creating hardware or software applications
• Devising detailed tests and evaluation procedures
• Carrying out rigorous testing
• Evaluating performance or usability
Research Methods Tony Fleet
19
HOPE
Survey Research
•
•
•
•
•
This research can be qualitative or quantitative; in
the widest sense, you are interviewing people. This
might involve:
An unstructured interview
A semi-structured interview
An structured interview based on questionnaire (face
to face, or by telephone)
An administered questionnaire
A Postal Questionnaire
Research Methods Tony Fleet
20
HOPE
Evaluative Research
•
•
•
•
This is primarily qualitative. Here you are trying to
assess whether something is of value, whether it meets
its specifications, or whether it is fit-for-purpose. This
might involve:
Developing a list of criteria on which to make
judgements
Examining the object against each of the criteria to
judge to what extent it conforms to expectations
Weighing the positives and the negatives, coming to
overall conclusions
Matching these judgments against similar judgements
made elsewhere in the literature or in real life.
Research Methods Tony Fleet
21
HOPE
Observational Research
•
•
•
•
This research normally uses a qualitative
approach; in the widest sense, you are recording
people’s behaviour. This might involve:
Participating in a task or situation
Making field notes of experiences
Creating and using an Observation Schedule
Making a check-list of occurrences of particular
events or items.
Research Methods Tony Fleet
22
Research possibilities
HOPE
Qualitative
Quantitative
Experiment
Field- testing
Bench-testing and
simulations
Survey
Unstructured
Interviews
Written
Questionnaires
Evaluation
Using expert judges Evaluation criteria
and checklists.
Observation
Participant
Observation
Research Methods Tony Fleet
Observation
schedules
23
HOPE
Other Forms of Research
Historical & Documentary Research proceeds by
scrutinising existing materials, both written and
artefacts, using them as sources of evidence.
Action Research is normally conducted in an
educational or political context. Action is taken,
monitored, evaluated and then modified for the next
cycle.
Ethnographic Research consists of an in-depth study
of a cultural phenomenon, in order to generate new
theory.
Case Study Research selects a whole range of
research methods in scrutinising one particular
context or situation.
Research Methods Tony Fleet
24
HOPE
Research methods 1
References (SW Library)
• Crabtree, Benjamin F. - Doing qualitative
research. - London : Sage, 1992. - (Research
Methods for Primary Care ; 3). - 0803943121
• Creswell, John W.. - Research design : qualitative
and quantitative approaches / John W. Creswell. Thousand Oaks, Calif; London : Sage, 1994. 0803952554
• Creswell, John W.. - Qualitative inquiry and
research design : choosing among five traditions /
J. - Thousand Oaks, Calif.; London : SAGE, 1998.
- 0761901434
Research Methods Tony Fleet
25
HOPE
Research methods 2
References (SW Library)
• Miller, Delbert Charles. - Handbook of research
design and social measurement. - 3rd ed. - New
York : David McKay Co . Inc, 1977. - m0859739
• Research methods in education and the social
sciences / [Research Methods in. - Block 3B :
Research design. - Milton Keynes : Open
University Press, 1983. - (DE304, Block 3B ; 3B).
- 0335074235
• Yin, Robert K.. - Case study research : design and
methods. - Rev. ed. - Newbury Park; London :
Sage, 1989. - (Applied social research methods
series ; v.5). - 080393470x
Research Methods Tony Fleet
26
HOPE
3. Selecting Appropriate
Research Methods
The next few slides discuss how
you might go about selecting your
research methods from those
available
HOPE
Selecting Your Methodology
Your research methodology consists of :
• Research Methods (experiment, survey etc.)
• Research Instruments (questionnaire, tests etc.)
• Analytical Tools (statistics, inductive or deductive
methods)
When selecting the methodology, you need to be
aware of:
• The Research Questions you are trying to answer
• The Population you are trying to generalise to.
Research Methods Tony Fleet
28
Factors to Consider
Undertake Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
Apply Methods & Instruments
Perform Statistical Analysis
How has previous
research in this area been
done?
What methods have been
used?
What research instruments
have been devised?
How will the methods &
instruments be applied?
What Statistical tests can
be carried out?
What Research
Hypotheses can be tested?
Test Hypotheses & Draw Conclusions
Research Methods Tony Fleet
29
HOPE
Appropriate methodology
• Do your Research Questions involve
impressions, attitudes, opinions, beliefs or
knowledge held by people?
• If so, then survey research is appropriate
• Do your Research Questions involve
behaviour, actions, reactions to events,
circumstances or objects?
• If so, then observational research is
appropriate
Research Methods Tony Fleet
30
HOPE
Appropriate methodology
• Do your Research Questions involve the
reliability or robustness of hardware,
software, systems or infrastructure?
• If so, then an evaluative study is appropriate
• Do your Research Questions involve the
testing of hardware or software at the
technical level, speed, accuracy, security
etc?
• If so, then experimentation is appropriate
Research Methods Tony Fleet
31
HOPE
A Mix of Methods
• It may be that your research questions
overlap some of these categories, or
different questions address more than
category.
• If so, you should consider a mix of
methods, that ensure that you cover all
eventualities. This may bring added
benefits. (See Triangulation)
Research Methods Tony Fleet
32
HOPE
Mixed Methods
References (SW Library)
• Mixing methods : qualitative and quantitative
research / edited by Julia Bra. - Aldershot :
Avebury, 1995. - 1859721168
• Tashakkori, Abbas. - Mixed methodology :
combining qualitative and quantitative approaches
/ Abba. - Thousand Oaks, Calif.; London : Sage,
1998. - (Applied social research methods series ;
v.46). - 0761900705
Research Methods Tony Fleet
33
HOPE
4. Robustness of Methods
As well as ensuring that your questions
are well-focussed, and your methods
relevant and appropriate, you need to
ensure that your methods are also
Reliable and Valid
HOPE
Reliability & Validity
• Research is Reliable if different methods,
researchers and sample groups would have
produced the same results.
• Research is Valid if the results produced by the
research are accurate portrayals of the situation,
explanations are effective, and predictions from
the research are actually borne out by
observation.
Research Methods Tony Fleet
35
HOPE
Reliability
• Research can have poor reliability, if it on one or two
cases only, or if personal judgement or opinion is
included.
• Reliability can be improved if data collection
methods are made more precise, we have
controlled experimentation, and we can produce
statistical summaries.
Research Methods Tony Fleet
36
HOPE
Validity
• Research can have poor validity if the data
produced is too far removed the object under study,
or the respondent. Using detailed, highly structured
research instruments can lead to distortions, by
forcing observations into categories when they do
not fit. Data summaries and averaging can also lead
to distortions, and meaningless generalities.
Graphical representations and percentages can be
highly selective and produce biased findings.
• Validity can be improved by working directly with
individuals or objects, focusing on specific cases,
making detailed observations, conducting face to
face interviews, taking detailed measurements in
specific circumstances etc.
Research Methods Tony Fleet
37
HOPE
Reliability v. Validity
Valid, but
not reliable
Research Methods Tony Fleet
38
HOPE
Reliability v. Validity
Reliable, but
not Valid
Research Methods Tony Fleet
39
HOPE
Methodological Trade-Off
• If you improve validity, you will almost
certainly reduce reliability.
• If you improve reliability it will be at the
cost of reducing your validity.
• The trade-off is to balance the two so that
the benefits of using particular methods
outweigh the losses incurred.
Research Methods Tony Fleet
40
HOPE
Triangulation
• Triangulation takes its name from the navigational
method of positioning a ship at sea by making two
independent observations.
• The purpose here is to use two distinct
methodologies, independent of one another to
confirm that the effects which we are observing
are real, and not artefacts of the research
process.
Research Methods Tony Fleet
41
HOPE
Triangulation
• Triangulation attempts to counter the
methodological trade-off by using a mix of
methodologies.
• If you are using highly structured, statistical or
measurement-based research, you supplement
this with detailed observations or face to face
interviewing.
• If your research is mainly based on individuals or
on single items, you ensure that at least part of it
has some statistical summaries, structured
observations or questionnaires.
Research Methods Tony Fleet
42
HOPE
Validity & Reliability
References (SW Library)
• Kirk, Jerome. - Reliability & validity in qualitative
research. - Beverly Hills Calif : Sage Pubns, 1986.
- (Qualitative Research Methods Series ; 1). 0803924704
• Litwin, Mark S. - How to measure survey reliability
and validity. - London : Sage, 1995. - (The Survey
Kit ; 7). - 0803957041
Research Methods Tony Fleet
43
HOPE
5. Structuring Your
Methodology
This section looks in detail at
the techniques that you might
employ in the Research
Activity itself.
Research Structure
Conduct Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
Apply Methods & Instruments
After framing your
Research Questions,
and selecting your
methodology, you
should test that this
is going to work by
conducting a smallscale Pilot Study
Perform Statistical Analysis
Test Hypotheses & Draw Conclusions
Research Methods Tony Fleet
45
HOPE
5a Pilot Studies
• A pilot study is a set of preliminary
investigations & procedures carried out prior
to the main research, to ensure that the
research is possible and can proceed
without hitches.
Research Methods Tony Fleet
46
HOPE
Pilot Study (1)
• In almost every type of Research, you will
need to devise or adapt some sort of
Research Method or Instrument.
• This might be a measuring procedure, a
set of evaluation criteria, an interview
procedure or questionnaire, or an
observational method or schedule.
Research Methods Tony Fleet
47
HOPE
Pilot Study (2)
• Your procedure or instrument should be
based on best practice from previous
research.
• It is unlikely that you will find exactly what
you need; you will be forced to adapt or
amend it.
• This means that you will need to conduct a
Pilot Study to test whether the new
instrument is fit-for-purpose.
Research Methods Tony Fleet
48
HOPE
Pilot Study (3)
• The main purpose of a Pilot Study is to iron
out ‘bugs’ in procedures, or to check that
instruments work.
• The size of the pilot study will depend on
how ‘inventive’ you needed to be.
• If your procedures are almost entirely of your
own devising, then you will need a fairly
extensive pilot study to check them.
• If you have ‘lifted’ methods from the
literature, than your pilot can be quite small.
Research Methods Tony Fleet
49
Pilot Study (3)
HOPE
• With questionnaires & observation schedules,
you will need to check that individual items
are giving you expected results.
• With test procedures, you need to check that
you can actually do what you have said that
you are going to do.
• With evaluation criteria, you need to use these
in a limited context, to see that they are
workable and effective.
Research Methods Tony Fleet
50
HOPE
Pilot Study (4)
• As a result of the Pilot Study, you need to
evaluate procedures and instruments,
making amendments where necessary.
• The Pilot Study stage will be part of your
research; you will need to write this up,
reporting on how your methods were
adapted and improved as a result.
Research Methods Tony Fleet
51
Carrying Out the Research
Undertake Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
Apply Methods & Instruments
Here we will examine
how particular methods
and instruments can be
applied in the research
situation.
Perform Statistical Analysis
Test Hypotheses & Draw Conclusions
Research Methods Tony Fleet
52
HOPE
5b A detailed look at some
Research methods
•
•
•
•
Experimental Design
Evaluative Research
Observational Research
Survey Research
Research Methods Tony Fleet
53
HOPE
5c.1 Experimental Design
This section looks at the different ways in
which you can conduct experiments. Note
that you do not need to be doing pure
‘experimental’ research to adopt these
methods.
Research Methods Tony Fleet
54
HOPE
Experimental Designs
• You may need to think about experimental
design even if you are doing types of
research other than ‘Experiments’
• These designs occur where you have
made some change, and are trying to find
out its effect.
• The net result is that you are comparing
one thing, or one group with another thing
or group.
Research Methods Tony Fleet
55
HOPE
Experimental Designs
• The terminology for experimental designs
comes from agricultural experiments.
• We have different ‘treatments’ which we
apply to different groups
• We control the groups for different ‘factors’
Research Methods Tony Fleet
56
HOPE
Experimental Designs
• Experiments normally involve ‘independent’ and
‘dependent’ variables’
• Independent variables are factors that can be
controlled for, like temperature, file sizes, age,
gender etc.
• Dependent variables are those factors which will
change as a result of altering the independent
variables.
• For example, download times will increase as a
result of increasing file sizes.
– Download time = dependent variable,
– File size = independent variable
Research Methods Tony Fleet
57
Pre-Test/Post Test
Control Group Design
HOPE
• This is the classic
experimental design.
• It allows you to split your
sample into two distinct
parts (A & B)
• You give the same test to
the two groups before you
start
• You treat one of the groups
• You apply the same test
afterwards.
Group A
Group B
(treated)
(untreated)
Pre-test
Pre-test
Treatment
Post-test
Research Methods Tony Fleet
Post-test
58
HOPE
Control Group
• The Control Group (B), is one which is the
same in all respects, except for the fact that
we make no changes.
• We can use the test measurements on the
control group as a benchmark for any
changes we make to the treatment group
(A)
Research Methods Tony Fleet
59
Example 1: Server Testing
HOPE
• Suppose we wish to check whether a
firewall is effective in blocking external
attacks.
• Set up System A & System B on two
different servers, running ‘nearequivalent’ internal & external programs.
• Devise a test which includes a full range
of possible attacks; apply to both A & B.
Take series of measurements &
observations.
• Incorporate firewall into System A
• Reapply test to both systems, which are
again running ‘near-equivalent’
programs.
Research Methods Tony Fleet
System A
System B
(treated)
(untreated)
Pre-test
Pre-test
Apply Firewall
Post-test
Post-test
60
Example 2: Software Design
•
HOPE
•
Suppose we wish to find out whether
anthropomorphic agents are useful in
communicating with novice users of a
website.
Set up Website A & Website B with ‘nearequivalent’ structure, but different content.
Devise a procedure which asks two
equivalent sets of users to make explorations
of the websites; apply to both A & B. Make a
series of observations
Incorporate Agents into Website A
Re-apply procedure, asking two more
equivalent sets of users to make the same
explorations of both Websites. Again make
observations.
•
•
•
Research Methods Tony Fleet
Website A
Website B
(treated)
(untreated)
Pre-test
Pre-test
Apply Agents
Post-test
Post-test
61
Example 3: On-Line Learning
HOPE
• Suppose we wish to find out whether training is
effective in helping students cope with the
Group A
demands of on-line learning.
(treated)
• Set up group A & group B with ‘near-equivalent’
members.
Pre-test
• The two groups to undertake a short
programme of learning on-line, which
Apply Training
incorporates a short evaluation and a
knowledge test.
• Give Training to Group A
Post-test
• The two groups to undertake a further short
programme of learning on-line, again
incorporating a short evaluation and knowledge
test.
Research Methods Tony Fleet
Group B
(untreated)
Pre-test
Post-test
62
HOPE
Factorial Designs
• This is where we examine the effects of two or
more independent factors simultaneously.
• For two factors, we would need 4 groups
–
–
–
–
Group A (Control: no treatment)
Group B (Factor 1 treatment only)
Group C (Factor 2 Treatment only)
Group D (Factors 1 & 2 Treatments)
• Clearly, this is going to increase the complexity
and size of the research, but it has the added
benefit of producing verifiable results in cases
where two factors interact to produce interesting
effects.
Research Methods Tony Fleet
63
HOPE
Other Design Variants
Post-Test only Control Group Design
• Here we assume that both groups are the same
(but do not test that assumption). We simply apply
the treatment to one group, and apply the test to
both groups.
Matched Pairs Design
• The individuals we use in the test groups
(subjects) are matched for characteristics which
are likely to affect the outcome (gender, age, level
of education, ethnicity etc.)
Research Methods Tony Fleet
64
HOPE
Experimental Designs:
Reliability Issues
• For high reliability, we need to build in strict
controls over each of the independent variables
affecting the outcomes of the experiment.
• We will also need to ensure that any
measurements taken of the dependent variables
are as accurate as possible.
• We also need to ensure that our methodology is
clear and replicable.
Research Methods Tony Fleet
65
HOPE
Experimental Designs:
Validity Issues
• For high validity, we need to conduct the
experiment in as ‘natural’ a setting as possible,
and in as near as exact a match to the
circumstances in which the events or objects
would normally operate.
• If a sample is being used, then the sample
(whether time, events, people or objects) should
be as representative of the ‘background’
population as possible, and we should make
detailed observations of how the events unfold as
well as the final measurements.
Research Methods Tony Fleet
66
HOPE
Experimental Design
References (SW Library)
• Campbell, Donald T., Donald Thomas, 1916-. Experimental and quasi-experimental designs for
research / Donald T. Campbell. - Boston; London :
Houghton Mifflin, 1963. – 0395307872
• Field, Andy. - How to design and report
experiments / Andy Field, Graham Hole. - London
: SAGE, 2003. - 0761973826
• Miller, Steve. - Experimental design and statistics.
- London : Methuen, 1975. - (Essential
Psychology ; A8). - m0805407
Research Methods Tony Fleet
67
HOPE
5c.2 Evaluative Research
This section looks at the methods
and issues surrounding evaluation.
You may need to use such
techniques if evaluation is implicit in
your research questions
Research Methods Tony Fleet
68
HOPE
Evaluative Research
• Evaluative Research covers those cases where
you are attempting to compare whether one
procedure or object is better or more effective
than another procedure or object, or to determine
whether a particular procedure or object is fit-forpurpose.
• Evaluation needs to be done against a set of
criteria, which have been established as valid and
reliable in this context. You would normally
produce an evaluation form to be completed by
respondents.
Research Methods Tony Fleet
69
Evaluation Criteria
HOPE
• To establish criteria for evaluation, we need to break the
topic down into individual elements, and state the
different sub-topics on which the item is to be
evaluated.
• Alongside this, we will normally state specific questions
which will need to be answered in order to judge the
item against the criterion.
• The answers to the questions may involve
measurements, counts, assessments on subjective
scales, statements of fact or possibly even opinion.
Research Methods Tony Fleet
70
HOPE
Example:
Website Evaluation Criteria
Design
• Is the use of colour acceptable?
• Are the elements in harmony?
Navigation
• Do the links work?
• How many links per page?
Content
• How many words & graphics on page?
• Is the text interesting & informative?
Interactivity
• What interactive features are used?
• Do these improve communication with the user?
Coding
• What scripting is used? Is it clearly annotated?
Research Methods Tony Fleet
71
HOPE
Example:
Website Evaluation Form
1. Design
• Colour Safe Pallet used?
Yes/ No
• Harmony of elements on page:
good/ neutral/ poor
2. Navigation
• Number of links on the page:
____
• Number of working links
____
3. Content
• Total size of graphics files on page: ____ MB
• Is the text: boring/ dull/ neutral/ interesting/ fascinating
4. Interactivity
• Tick and/or name all interactive features used:
rollovers; dynamic images; image maps; ______, ______
5. Coding
• What DTD has been used? _______________________
72
Research Methods Tony Fleet
HOPE
Evaluation Criteria:
Validity Issues
• For criteria to be valid, each of the criteria
elements need to have face or content
validity; the questions posed by the
criterion needs to be relevant to the object,
and relates to a feature of it
• We also need to be able to answer the
questions in an objective manner, without
recourse to guessing, or giving an
‘impressionistic’ response.
Research Methods Tony Fleet
73
HOPE
Evaluation Criteria:
Reliability Issues
• For criteria to be reliable, each of the
criteria elements need to be clear and
unambiguous, so that different assessors
would interpret the criteria in the same
way.
• We also need to ensure that repetition of
the evaluation exercise will yield results
which are not dissimilar to one another.
Research Methods Tony Fleet
74
Evaluation Methods
References (SW Library)
HOPE
• Britain, Sandy. - A framework for pedagogical
evaluation of virtual learning environments. Manchester : Joint Information Systems Committee,
1999. - (JISC Technology Applications Programme r.. M0000712EL
• Broadbent, George Ernest. - The role of evaluation in
user interface design : a practical implementation. Liverpool : University of Liverpool, 1997. - p7270369
• Redmond-Pyle, David. - Graphical user interface
design and evaluation (GUIDE) : a practical process. London : Prentice Hall, 1995. - 013315193x
• Smeltzer, Nicholas. - Critical analysis of the design and
evaluation of a computer-based project. - Liverpool :
University of Liverpool, 2001. - M0002704LO
Research Methods Tony Fleet
75
HOPE
5c.3 Observational Research
This section describes what you need
to do if your research involves
making detailed observations of
people, events or objects.
Research Methods Tony Fleet
76
HOPE
Observational Research
In this case you are trying to determine
Either: How an individual or group of
individuals react, interact or behave in
particular circumstances
Or: How software or hardware performs
when used by particular groups of people
Research Methods Tony Fleet
77
HOPE
Participant Observation
In this case, the researcher becomes one
of the subjects, and works alongside the
subject, monitoring behaviour and
interacting with them.
Pros:Researcher can fully ‘understand’ what
the issues are. Can get real validity to
research.
Cons: Researcher can influence the
research, alter opinions. Highly subjective
& unreliable. Taking notes is difficult; relies
on good memory.
Research Methods Tony Fleet
78
HOPE
Non-Participant Observation
In this case, the researcher studies the
situation apart from subjects, taking notes,
monitoring behaviour and observing
interactions. The use of audio & video
recording is useful here.
Pros:Researcher can get overview of the
situation, and achieve objectivity
Cons: Researcher can only see resulting
behaviours, not what is causing it or why it
is happening.
Research Methods Tony Fleet
79
HOPE
Observation Schedules
• An observation schedule can be a
simple list of things to look for in a
particular situation
• It can be far more complex; a minute
by minute count of events such a
mouse-clicks or verbal interactions
between subjects.
Research Methods Tony Fleet
80
Observation Schedule
An Example
HOPE
Observation of subject using Information Portal
Subject: M / F
Date: _________
Age: 18-21 / 21-30 / 31-50 / 50+
Time: _________
Selected Navigation Tool : Mouse /Keyboard/ TouchScreen
First 5 pages visited in order: __ __ __ __ __
Time to obtain to obtain required information: ___ min ___ sec
Total Number of Pages visited: __
Feedback from subject:
very positive / positive / neutral / negative / very negative
Research Methods Tony Fleet
81
HOPE
Observation Schedules:
Some Reliability Issues
• For an observation schedule to be reliable,
it should require structured documentation
of events.
• This will involve such things as checklists,
minute-by-minute categorisation of activity
and numerical data such as frequencies of
occurrence and time intervals. Timings
should have a clear start and end points.
• The schedule should leave little room for
subjective judgement.
Research Methods Tony Fleet
82
HOPE
Observation Schedules:
Some Validity Issues
• For an observation schedule to be valid,
it should refer to events which actually
happen; each of the events on the sheet
should be possible, and likely to occur.
• Timings should be possible to take, and
not interfere with other observations which
should be made.
• There should be room for observations
which ‘enrich’ the data by adding detail to
the numbers, offering explanation and
illumination.
Research Methods Tony Fleet
83
HOPE
Observational Research
References (SW Library)
• Harding, Jacqueline. - How to make observations
& assessments / Jackie Harding and Liz MeldonSmith. - 2nd ed. - London : Hodder & Stoughton,
2000. - 034078038x
• Robertson, Kevin. - Observation, analysis and
video / editors: Anne Simpkin and Penny Crisfield.
- Leeds : National Coaching Foundation, 1999. 1902523164
• Simpson, Mary. - Using observations in smallscale research : a beginner's guide / Mary
Simpson. - Glasgow : Scottish Council for
Research in Education, 1995. (SCREpublications ; 16 ; 130). - 1860030122
Research Methods Tony Fleet
84
HOPE
5c.4 Survey-Type Research
This section describes what you need
to do in order to use human
respondents to provide you with
information.
Research Methods Tony Fleet
85
Survey methods
HOPE
There are two distinct elements here:
• The interview techniques that you adopt
in order to elicit information from people
• The sampling methods that you adopt in
order to select respondents for interview
You will need to make rational choices
for both of these elements, depending
upon the focus of your research and
the population under study.
Research Methods Tony Fleet
86
HOPE
Interviewing
• We interview people face-to-face in order
to try to find out exactly what they think.
• With the right questions, people respond
with high quality information.
• The data that you get can be of high
validity, since you have access to
respondents’ own words.
Research Methods Tony Fleet
87
HOPE
Types of Interviews
•
•
•
•
Unstructured Interview
Structured Interview
Semi-Structured Interview
Administered Questionnaire
Research Methods Tony Fleet
88
HOPE
Open or Closed?
• When interviewing respondents, the main choice
facing the researcher is whether to use open
questions, which leave the respondent free to
answer in any way they think fit, or closed
questions which force the respondent to make
particular choices, pre-determined by the
researcher.
• OPEN: What is your experience of chat rooms?
• CLOSED: Do you think chat rooms should be
monitored? (Yes/No/Maybe)
Research Methods Tony Fleet
89
Open or Closed?
Validity & Reliability Issues
HOPE
• In general, the data from open questions is richer,
more illuminating, and more valid, as it can illustrate
clearly why particular subjects think the way that they
do, or why they behave in particular ways.
• Data from closed questions however, is more
amenable to statistics. It is easier to summarise, spot
trend and make comparisons. In general the data is
more reliable.
• A good strategy is to use a mixture of both open and
closed questions
Research Methods Tony Fleet
90
HOPE
Unstructured Interviews
Unstructured Interview
• Interviewer has no set agenda; the object is to get
the respondent to talk freely about various topics.
Pros: Can be high quality data, often get real
insights into what people really think.
Cons: Can be difficult for the novice researcher to
carry out; need to be good at steering the
conversation without forcing it. Time consuming;
can only carry out a small number. Difficult to
analyse
Research Methods Tony Fleet
91
Semi- Structured
Interview
HOPE
• Interviewer has a formal list of topics, which sets the
agenda; however, the interviewer is free to take these in
different orders, or to return to topics at different points.
• Here we include the idea of ‘focus’ groups, where a
facilitator encourages the discussion of a particular
topic.
Pros: Data provided can be almost as good as
unstructured interview, but in a more focussed manner.
Cons: Can miss important ideas, because agenda set
beforehand; interviews can also be time consuming &
difficult to manage & analyse afterwards.
Research Methods Tony Fleet
92
HOPE
Structured Interviews
• Interviewer has a list of topics to be taken in a
particular order; questions are written, and read
out.
Pros: You can make good comparisons between
different respondents; you also will have access to
respondents’ thinking.
Cons: Unless you have done some preliminary
investigations, and extensively piloted the
interviews, you may miss lots of important data.
Takes time to do properly; can only carry out a few.
Research Methods Tony Fleet
93
HOPE
Administered
Questionnaire
• Interviewer has devised a list of questions which are
read out; some multiple choice, some open-ended.
Filled in either by respondent or interviewer.
• Pros: Interview can be quite brief; easy to make
comparisons, data amenable to statistical analysis.
• Cons: Data may be ‘warped’ by the choice of
questions. Respondent may be forced into giving
fabricated answers.
Research Methods Tony Fleet
94
HOPE
Questionnaires
There are many different types of
questionnaire, depending upon what you are
trying to find out.
As well as gathering factual data about the
person, you may be trying to explore their:
–
–
–
–
–
Knowledge
Beliefs
Attitudes
Opinion
Behaviour
If you need to design a questionnaire, see
for example:
http://www.leeds.ac.uk/iss/documentation/
top/top2/index.html
Research Methods Tony Fleet
95
HOPE
Question Types
•
•
•
•
•
•
•
•
•
Factual Data
Binary (Yes/No)
Single Selection from Categories
Multiple Selection from Categories
Attitude Scale items
Focussed items to elicit categories
Conditional Questions for routing
Open-ended Questions
Self-Reporting
Research Methods Tony Fleet
96
See: http://www.analytictech.com/mb313/principl.htm
Writing Questions
•
•
•
•
•
•
•
•
HOPE
Put Factual Data (demographics) at end
Put Explanations and Disclaimers at the start
Keep questions as brief as possible
Use non-technical language where possible
Avoid leading respondents towards particular answers
Use direct questions, no hypotheticals
Use simple questions, no portmanteaus
Use a mixture of positively and negatively worded
questions
• Verify data by asking same question in different ways.
Research Methods Tony Fleet
97
Questionnaire Example
1. Which operating system are you currently using:
Linux
Windows
Other
HOPE
2. How would you rate the operating system?
very poor
poor
good
very good
3. Circle the tasks which you use your computer to do:
Word processing
Internet Access
Program Development
4. Estimate the number of hours you use the computer
for each week:
______ hours
5. Do you use broadband?
Yes
No
Research Methods Tony Fleet
98
HOPE
Protocol
Whatever your interviewing method, as part of the
ethical constraints on Hope Researchers, you are
required to do the following:
 Explain to the respondents what the research is
about.
 Tell them that they will not be identified by name
in the final report, and that any views that they
express will be in confidence.
 Explain to them that if there are questions with
which they feel uncomfortable, they do not have
to answer.
Research Methods Tony Fleet
99
HOPE
Attitude Measurement
• You may wish to incorporate ‘attitude measurement’
into your questionnaire as either a major or a minor
feature.
• There are several different types of scales which can
be used to elicit numerical measurements of attitude:
– Likert Scaling
– Thurstone Scaling
– Guttman Scaling
– Semantic Differential Scaling
See: http://www.socialresearchmethods.net/kb/scalgen.htm
Research Methods Tony Fleet
100
Likert Scales (1932)
HOPE
• This consists of items like “I would not trust a bank’s
website to keep my details secure”
• Respondents are asked to use a scale such as
1=Strongly disagree; 2=disagree; 3=neutral; 4=agree;
5=strongly agree
• Scales can vary: 0-4, 1-7 etc.
• Some proponents suggest removing the neutral
category to force a choice, but this can reduce validity.
• Scores on individual items are totalled to obtain the
respondent’s score; this is often shown as an average.
Research Methods Tony Fleet
101
HOPE
Thurstone Scales (1928)
• This pre-ranks 11 statements with numerical values
1-11; each statement carries a numerical score, for
example:
•
•
•
•
1 = “Teleworking is an impossible concept”
2 =“Teleworking may be OK in exceptional cases”
8 = “If offered the opportunity, I would try teleworking”
11 = “In 20 years’ time, everyone will be teleworking”
•
Clearly, lots of preparatory work needs to be done to
generate and rank the statements.
• A respondent’s score, is the average numerical value
of all the items they agree with.
Research Methods Tony Fleet
102
HOPE
Guttman Scales (1944)
• This pre-ranks statements in order, very similar to
Thurstone scaling, except here we attempt to
construct the scale so that if a person agrees with
item 4 on the scale, they will also agree with items
1,2 and 3.
• When the questions are administered, the items
are muddled, but each retains a ranking.
• The respondent’s score is the sum of the ranks
associated with the items he or she agreed with.
Research Methods Tony Fleet
103
Osgood’s (1957)
Semantic Differential Scales
HOPE
• Respondents are asked to rate an idea or an object
against a series of opposing adjectives or
descriptions, for example:
Exciting
Hard
Frustrating
Using Learnwise
[] [] [] [] [] []
[] [] [] [] [] []
[] [] [] [] [] []
Dull
Easy
Stimulating
• The scale asks respondents to tick the box nearest
the descriptor that they agree with. Each box has a
numerical value: e.g. 1,2,3… 7
• The respondent’s score can be portrayed graphically,
or as a total of numerical values.
Research Methods Tony Fleet
104
HOPE
Questionnaires & Interviews:
Reliability Issues
• For questionnaire results to be reliable, you need
closed questions that are precisely framed,
unambiguous, with response categories that are
well-defined, exhaustive and exclusive. You need to
ask the same question in different ways, and you
need to collate the information into statistical
summaries and expose the data to rigorous
statistical testing.
• You need to use as large a sample as possible, and
even out any random fluctuations in the data by
using summary statistics and hypothesis testing.
Research Methods Tony Fleet
105
HOPE
Questionnaires & Interviews:
Validity Issues
• For questionnaire results to be valid, you need
response categories which enable the respondent
to express their views accurately; this can best be
done with open questions. You should take time
with the respondent and respect the data that they
provide for you.
• You need to ensure that the sample is
representative of the population, and large enough
for the results to be statistically significant.
Research Methods Tony Fleet
106
HOPE
Questionnaire References
On-Line
• http://www.tardis.ed.ac.uk/~kate/qmcweb/q
cont.htm
• http://www.leeds.ac.uk/iss/documentation/t
op/top2.pdf
• http://www.statpac.com/surveys/
• http://www.fao.org/docrep/W3241E/w3241
e05.htm#chapter%204:%20questionnaire
%20design
• http://www.surveysystem.com/sdesign.htm
Research Methods Tony Fleet
107
Questionnaire References (1)
(SW Library)
HOPE
• Foddy, William. - Constructing questions for interviews
and questionnaires : theory and practice. - Cambridge
: Cambridge University Press, 1993. - 0521467330
• Fowler, Floyd J.. - Improving survey questions : design
and evaluation. - London : Sage, 1995. - 0803945833
• Frazer, Lorelle. - Questionnaire design and
administration : a practical guide / Lorelle Frazer. Brisbane : Wiley, 2000. - 0471342920
• Gillham, W. E. C., William Edwin Charles, 1936-. Developing a questionnaire. - London : Continuum,
2000. - (Real world research). - 0826447953
Research Methods Tony Fleet
108
Questionnaire References (2)
(SW Library)
HOPE
• Oppenheim, A. N., Abraham Naftali, 1924-. Questionnaire design, interviewing and attitude
measurement / A.N. Oppenheim. - New ed. - London :
Continuum, 2000. - 0826451764
• Wengraf, Tom. - Qualitative research interviewing :
biographic narrative and semi-structured interviews London : SAGE, 2001. - 0803975007
• Young, Pauline V. - Scientific social surveys and
research : an introduction to the background,. Englewood Cliffs, N.J. : Prentice-Hall, 1966. - (PrenticeHall Sociology Series). - m0859969
• Youngman, Michael Brendan. - Designing and analysing
questionnaires. - Maidenhead, Berks : TRC Rediguide. (Rediguide ; 12). - m0891542
Research Methods Tony Fleet
109
Sampling
HOPE
• The question of how to select the respondents for
the sample is always tricky.
• The principle behind sampling, is that you should
ensure that the sample is appropriate in order for you
to generalise your results to the population under
study.
• There are essentially three ways in which this is
done:
– Random Sampling
– Quota Sampling
– Stratified Random Sampling
Research Methods Tony Fleet
110
HOPE
Random Sampling
• This method requires you to get a list of all the
population, as near complete as you can find,
then use some random selection method (such
as shutting your eyes and stabbing a pen at the
list, or allocating using random numbers)
• The idea is that every person in the population
has an equal chance of ending up in the sample.
• Most statistical methods assume that you are
sampling randomly, and it is the only method
which overall is guaranteed to ensure that
samples are free from bias, and therefore
provide validity.
Research Methods Tony Fleet
111
HOPE
Quota Sampling
• Here, you identify particular sectors of the population,
such as men, women , those under 21, over 21,
employed, unemployed etc., and put quotas on the
number of people in each category.
• For example, in a quota sample of 12, we might
have:
– 2 males under 21; 2 females under 21
– 4 males over 21 ; 4 females over 21
• The purpose here is not to get a sample which
represents the population in its entirety, but to ensure
that views from important sections of the population
are represented.
Research Methods Tony Fleet
112
Stratified Random Sampling
•
HOPE
•
•
•
•
•
One of the problems with Random Sampling, is that if you take
small samples, you can very well end up with a biased sample , by
chance.
In Stratified Random sampling, you would measure what
proportions of the population lie in each category or strata, and
select your sample so that you get precisely those proportions in
those strata
For example in your population you might have:
– 15% males under 21; 10% females under 21
– 30% males over 21 ; 45% females over 21
If you select a sample of 200 you would need stratified samples:
– 30 males under 21; 20 females under 21
– 60 males over 21 ; 90 females over 21
The allocation of subjects to samples should be done randomly.
The purpose here is to get a valid sample which which represents
the population in its entirety.
Research Methods Tony Fleet
113
HOPE
Other Sampling Methods
Snowball Sampling: This method is used whenever you are
dealing with a tricky subject, where people may be doing
devious or illegal activities (for example hacking, virus creation
etc.). In this case, you would use one respondent to suggest
the name of another who might be willing to be interviewed;
that one would lead to several others and so on.
Cluster Sampling: This method might be used if you have a
diverse population (such as those living in African urban
communities). Here you would randomly select particular cities,
and sample within neighbourhoods within those cities.
Convenience Sampling: This is used mainly for investigative
research. It relies on the fact that people appearing at a
particular location or time will do so, ‘at random’ (may not be
true). We use this fact to stand in one place and interview them.
Research Methods Tony Fleet
114
Sampling: On-Line References
HOPE
This is an easy introduction:
• http://www.csm.uwe.ac.uk/~pwhite/SURVEY1/node26.
html
This is more complex and technical:
• http://www.socialresearchmethods.net/kb/external.htm
Research Methods Tony Fleet
115
HOPE
6. Data Analysis
This section of the presentation
looks at the different types of data
available, and how this can be
analysed
Analysis of Research Data
Undertake Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
Apply Methods & Instruments
Here we examine how the
data produced by the
research can be presented
effectively and statistically
analysed.
Perform Statistical Analysis
Test Hypotheses & Draw Conclusions
Research Methods Tony Fleet
117
HOPE
Data Analysis
6a
6b
6c
6c
6d
6e
Types Of Data
Extracting Data for Analysis
Presentation of Data
Principles of Statistical Testing
Selected Tests
Analytical Tools
Research Methods Tony Fleet
118
HOPE
6a Types of Data
In this section we describe the
different types of data that you might
encounter, and what you might do
with it in your study.
Research Methods Tony Fleet
119
HOPE
Types of Data
• ‘Measurement’-type data, given in units measured on a
decimal scale. e.g. MB/sec, times, averages and other
statistical summary values.
• ‘Counted’ data: number of occurrences of events or objects:
e.g. number of pages visited
• ‘Ranking’ data: subjective impressions, such as marks out of
10, rankings etc. e.g. Interface rated as very poor (1), poor (2),
good (3) or very good (4).
• Categorical data: Names of different things or categories,
such as those named by an interviewee (NB these may be
counted) e.g. “Manager A discussed 3 concerns: security,
access and reliability”
• Descriptive data, such as that produced by interviews or
observational field-notes: e.g. “Subject A tried to click on all
the blue text, even though it was not underlined”.
Research Methods Tony Fleet
120
HOPE
‘Measured’ Data
• Such data is called “continuous”, “ratio scale”, or
“real number” data
• This type of data is easiest to use; it can be
arithmetically manipulated (added, subtracted,
multiplied, divided, summarised)
• it is also likely to have good, predictable statistical
features.
• There are lots of methods to pick from.
• The inclusion of valid measured data will add
reliability to your study.
Research Methods Tony Fleet
121
HOPE
‘Counted’ Data
• Such data is normally called “discrete”, “interval
scale”, or “integer” data
• It can be added, subtracted and averaged.
• ‘Counted’ data, has almost all the features of
measured data, especially if the numbers of the
count are large (above 30 or so).
• For small numbers (e.g. less than 30 observations
in total), methods are limited.
• The inclusion of valid counted data will almost
certainly improve the reliability of your study.
Research Methods Tony Fleet
122
HOPE
‘Ranking’ Data
• Such data is normally called “interval scale data”
data.
• In order to use it effectively, there are several
techniques; the usual one is to convert the data into
numerical values (e.g. “Yes” = 1, “No” = 0)
• When this has been done, it can be added,
subtracted and averaged.
• ‘Interval’ data needs care; the data ‘looks like’ counts
where the numbers are small, but are simply
subjective impressions. To deal with such data we
need to summarise large volumes of it data.
• The inclusion of valid counted data, suitably
summarised can add reliability to your study.
Research Methods Tony Fleet
123
HOPE
‘Categorical’ Data
• Such data is normally called “nominal” data.
• In order to use such data effectively, the objects, classes or
ideas that these categories represent should have the items
within them counted, evaluated on an interval scale and/or
summarised.
• A preliminary study may have found, for example that there are
4 main uses for spreadsheets in business: Finance, Timesheets,
Inventory and General Calculation. Businesses may be asked to
rank these in order of importance, or to rate their usefulness.
• ‘Categorical data is much more difficult to deal with. Processes
are time consuming and highly subjective; There are relatively
few statistical tools which deal effectively with categorical data.
• The inclusion of valid ‘Categorical’ data, suitably summarised
can add both validity and reliability to your study.
Research Methods Tony Fleet
124
HOPE
‘Descriptive’ Data
• Descriptive data is simply a verbal description of
something that happened, or what a respondent did,
or a quote from an interview.
• It is the most difficult to deal with effectively. There are
methods for extracting summary information from
such data (see Grounded Theory: Glazer & Strauss);
however, these are not for the novice researcher in IT,
as they are very time consuming.
• The best use for such data is for illustration and
supporting evidence elsewhere.
• The inclusion of appropriate ‘Descriptive’ data,
suitably selected can improve the overall validity of
your study.
Research Methods Tony Fleet
125
HOPE
Types Of Data
References (SW Library)
• Research methods in education and the social
sciences / [Research Methods - Block 5 :
Classification and measurement. - Milton Keynes :
Open University Press, 1979. - (DE304 : a third ..
- 0335074405
• Research methods in education and the social
sciences / [Research Methods. - Block 6 : Making
sense of data. - Milton Keynes : Open University
Press, 1979. - (DE304 : a third level course. 0335074413
Research Methods Tony Fleet
126
HOPE
Types of Data
References (SW Library)
• Chatfield, Christopher. - Statistics for technology :
a course in applied statistics. - 2nd. - London :
Penguin Books, 1978. - 0412157500
• Kranzler, Gerald. - Statistics for the terrified /
Gerald D. Kranzler, Janet P. Moursund. Englewood Cliffs, N.J. : Prentice Hall; London :
Prentice-Hall International, 1995. – 0131838318
• Pentz, Mike. - Handling experimental data. Milton Keynes : Open U.P., 1988. - 0335158242
• Salkind, Neil J.. - Statistics for people who (think
they) hate statistics. - Thousand Oaks, Calif.;
London : Sage, 2000. - 0761916210
Research Methods Tony Fleet
127
HOPE
6b Extracting Data For Analysis
Data produced by the research
process does not have to be
numerical in order to be of value.
However, statistical techniques work
best where the data uses numerical
values, or can be converted to
numerical scales.
Research Methods Tony Fleet
128
HOPE
Survey Data
• Questionnaire data should be the easiest
data to extract. Ideally the questionnaire will
have been designed with the methods of
analysis in mind, so that questions are
framed in such a way as to facilitate
graphical representation and statistical
testing.
• Interview Data can be more problematic; you
may need to ‘invent’ categories, and to count
up instances of different cases, both in the
same interview, and with different subjects.
Research Methods Tony Fleet
129
HOPE
Questionnaire Data Examples
Percentage of Respondents using Broadband: 82%
In the survey, 93% of people used their computer to
access the internet, compared to 37% who used it
for word-processing, and 7% who used it to develop
programs.
87% of Linux users rated their operating system as
good or very good, compared to 64% of Windows
users.
The average number of hours spent on the computer
each week by men was 9.3, compared to 3.7 for
women.
Research Methods Tony Fleet
130
HOPE
Interview Data Examples
• 5 out of the 6 interviewees were using Broadband.
• During the interview, all interviewees mentioned the positive
benefits of moving to broadband. For example one respondent
said “My daughter can use the phone while I am replying to
emails”. Altogether there were 25 instances in the interviews
where benefits were named, as opposed to only three cases
where drawbacks were noted.
• These cases broke down as follows:
– 20% of cases referred to accessibility issues
– 60% of cases referred to improved speed
– 20% of cases referred to improved compatibility
• However, it must be noted that all the drawbacks were cited by
a respondent who reportedly used the internet for over 70
hours per week. The other five respondents’ use was an
average of 15.7 hours, with totals ranging from 4 hours to 25
hours.
Research Methods Tony Fleet
131
HOPE
Observational Data
• Pure observational field notes are unlikely
to contain numerical data. As with
interview data, you may have to ‘invent’
appropriate categories and count up
instances or occurrences.
• Observation schedules are easier to deal
with. As with questionnaires, these should
have been designed with the methods of
analysis in mind, so that items are framed
in such a way as to facilitate graphical
representation and statistical testing.
Research Methods Tony Fleet
132
HOPE
Observational Data Examples
• Out of the 40 people observed in the study, 36 of
them (90%) used the mouse as the first point of
contact with the portal.
• After the entry page, 70% visited page 3, 20%
visited page 7 and the rest visited pages 2, 4 and 9.
• The mean time taken to access the information by
novice users was 163.5 seconds (SD = 24.8 secs);
however, those who were subjected to the brief
training session were able to access the information
in a mean time of 48.3 seconds (SD = 13.9 secs)
• 95% of the users rated the interface good or very
good.
Research Methods Tony Fleet
133
HOPE
Experimental Data
Experiments should have been designed
with specific comparisons in mind, where
independent variables are identified &
controlled, and dependent variables are
measured in some way. The methods of
analysis should have been identified
beforehand, and the whole is susceptible
to graphical representation and statistical
testing.
Research Methods Tony Fleet
134
HOPE
Experimental Data Examples
Using two identical machines, System A using
Windows XP and System B using Linux, accessing
the website using a 56Mb/sec modem.
Download times were:
System A
4.5MB:
25.3MB:
185MB:
387MB:
875MB:
1.56GB
4.76GB
System B
1.8 secs
7.2 secs
35.4 secs
71.8secs
165.3 secs
299.5 secs
timed out
4.5MB:
25.3MB:
185MB:
387MB:
875MB:
1.56GB
4.76GB
Research Methods Tony Fleet
1.6 secs
6.4 secs
36.2 secs
74.3secs
155.8 secs
287.5 secs
889.3 secs
135
HOPE
Evaluation Data
Evaluation is unlikely to yield lots of numerical
data. Categories on the Evaluation Sheet may
well involve numerical values: counts, times,
file sizes etc., and these are amenable to
graphical representation & testing. Other
categories may require judgement on
‘descriptors’, or on quality. It is possible to
convert some of these to simple scales
(0,1,2,3 etc), and to rank items in order, for
example. Yes/No can be encoded as 1 and 0.
Research Methods Tony Fleet
136
Evaluation Data Examples
The 5 evaluators rated the 4 packages as follows:
HOPE
Evaluator
Best
|
|
Worst
•
•
•
1
C
A
D
B
2
C
D
B
A
3
D
B
C
A
4
C
B
D
A
5
C
D
B
A
Package C performed best overall; it achieved higher grades on 6 out of
7 criteria, and received a total rating of 8.3, compared to ratings of 6.1,
4.5 and 2.8 for packages B, D and A
The reasons for Package A’s poor performance were cited by the
evaluators as: poor readability, confused navigation, and lack of
interactivity.
4 out of 5 of the evaluators rated the interface design for Package A as
poor or very poor.
Research Methods Tony Fleet
137
HOPE
Data Collection
References (SW Library)
• Research methods in education and the social
sciences. – Block 4 : Data collection procedures. Milton Keynes : Open University Press, 1979. (DE304 ; block 4). - 0335074251
• Research methods in education and the social
sciences / [Research Methods. - Block 6 : Making
sense of data. - Milton Keynes : Open University
Press, 1979. - (DE304 : a third level course. 0335074413
Research Methods Tony Fleet
138
HOPE
6c Presenting Data
• This section of the presentation looks
at some of the different ways in which
you might present data, and the
different ways in which you can draw
meaning from it.
Research Methods Tony Fleet
139
HOPE
Presenting Numerical Information
• Frequency Tables
• Cross Tabs
• Time Series Data
Research Methods Tony Fleet
140
HOPE
Frequency Tables
• In this example the number of errors cited when 3 different
websites are run through an html code checker are
tabulated.
• It is normal where percentages are given, to quote the actual
numbers involved, OR the total number of cases.
The percentages allow
an easy comparison to
be made on a ‘standard
scale’
Website
A
Website
B
Website
C
Errors
Reported
35
12
17
Percentage
54.69%
18.75%
26.56%
Research Methods Tony Fleet
141
CrossTabulations
HOPE
• The purpose of a Cross-tabulation is to examine the relationships
between two different variables.
• In this example, the number of men and women of different age
groups expressing an interest in Blue Tooth are cross-tabulated.
Here the information
does not really show
what is happening.
Here we have used
percentages, and it is
now clear that although
there is a reduction in
interest in both males
and females as they get
older, it is far more
dramatic for females.
Under 21
Men
87
Women
85
21-30
64
32
31-40
28
17
41-50
31
14
50+
17
4
Under 21
21-30
31-40
41-50
50+
Men
(n=227)
38.3%
28.2%
12.3%
13.7%
7.5%
Women
(n=152)
55.9%
21.1%
11.2%
9.2%
2.6%
Research Methods Tony Fleet
142
HOPE
Time Series Data
• A Time Series simply logs the value of a particular variable
over a period of seconds, days or weeks.
• Because of amount of numerical information, it is very difficult
to make sense of time series data unless it is summarised or
presented in graphical form.
Time
(min)
Unused
Sectors
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1028 985 972 946 889 825 748 729 719 716 714 693 676 658 643 628 599 579 570 561 545
This data shows the result of an experiment to randomly access websites, and download
any .jpg files on the first page onto a hard drive. All we can see here is that the numbers
are declining, but we cannot really get a sense of how this is happening
Research Methods Tony Fleet
143
HOPE
Presenting Graphical Information
•
•
•
•
Simple Bar Charts & Histograms
Comparative Charts
Scattergrams
Time Series Data
Research Methods Tony Fleet
144
Simple Bar Charts & Histograms
HOPE
• Simple graphs can be highly effective in making a point.
• This one shows the fact that Package A clearly has been
rated higher than the other two packages.
Average Rating for Packages
8
6
Rating 4
2
0
Package A
Research Methods Tony Fleet
Package B
Package C
145
Comparative Charts
Number of men and women expressing an interest in
Bluetooth
HOPE
• All three of these charts
display the same
information.
• Which makes it clearer?
100
90
80
70
60
Frequency 50
40
30
20
10
0
Men
Women
Under 21
21-30
31-40
41-50
50+
Age group
Number of men and women expressing an interest in
Bluetooth
Number of men and women expressing an interest in
Bluetooth
100
80
Men
60
Women
Frequency
40
20
0
Under 21-30
31-40 41-50
21
Women
Men
50+
100
90
80
70
60
Frequency 50
40
30
20
10
0
Men
Women
Under 21
21-30
31-40
41-50
50+
Age group
Age group
Research Methods Tony Fleet
146
Scattergrams
We could add a
‘regression line’ to
make the relationship
explicit.
File Size v. Download time (secs)
Download times (secs)
HOPE
• These are useful if you are trying to demonstrate
a relationship between two variables.
200.0
150.0
100.0
50.0
0.0
0
0.5
1
1.5
2
2.5
File Size (MB)
Research Methods Tony Fleet
147
Time Series
Here the data shows an
almost uniform rate of
loss of sectors, minuteby-minute.
Number of Sectors Remaining
1200
Unused Sectors
HOPE
• A variant on the scattergram with lines added,
records of what happened over a period of time to
the quantity measured.
1000
800
600
400
200
0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Minutes into Experiment
Research Methods Tony Fleet
148
HOPE
Simple Statistics
• Data needs to be
summarised using a variety
of standard tools:
 Proportions, expressed in
percentages;
 Averages, such as means,
medians etc.
 Measures of variability, such
as ranges, standard
deviations etc.
Expert
Website
A
Website
B
Website
C
I
II
III
IV
V
VI
8
4
5
7
9
9
5
4
5
6
6
7
2
3
3
2
2
1
Mean
St.Dev
7.0
2.10
5.5
1.05
2.2
0.75
In this example, 3 different websites
have been given gradings out of 10 by
a panel of 6 experts.
In the summaries, it is clear that A
outperforms the others, and C is the
worst. However, there is a lot more
inconsistency in A’s scores than any of
the others.
Research Methods Tony Fleet
149
HOPE
• St. John, P. R.. - Methods of
presenting fieldwork data / P.R. St.
John, D.A. Richardson. - Sheffield :
The Geographical Association, 1989.
- 0948512164
Research Methods Tony Fleet
150
HOPE
6d Statistical Testing
• This section of the presentation
explains the principles behind
statistical testing, and describes the
some of the different types of
statistical tests available.
• It also explores probability theory.
Research Methods Tony Fleet
151
Hypothesis Testing
Undertake Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
Apply Methods & Instruments
Perform Statistical Analysis
Test Hypotheses; Draw Conclusions
If possible, the final
section of your study
should include some
statistical tests to
provide rigour to your
arguments.
Research Methods Tony Fleet
152
HOPE
Probability Theory (1)
• If your data consists of
measurements,
measured then you
can probably assume
that you have a
‘Normal’ Distribution,
such as IQ scores.
• Lots of people with
average IQ scores in
the middle, but with
very low numbers
with High IQs or Low
IQs.
http://www.socialresearchmethods.net/kb/sampstat.htm
Research Methods Tony Fleet
153
HOPE
Probability Theory (2)
• Even if your data is not measurement data, you
can still make use of ‘Normal’ Distribution
techniques, provided that:
 Your data is counted, and you have high
numbers (over 50 observed in each category)
 You are working with averages of 10 items or
more.
Research Methods Tony Fleet
154
HOPE
Probability Theory (3)
• Probability tries to measure how likely a particular
observed event is, before it occurs.
• For example, if we picked an undergraduate
student at random, and asked what is the likelihood
of that student winning the lottery, it would be a very
low probability event (prob = 0.0000001)
• On the other hand, if we asked what is the
likelihood of that student graduating, the probability
would be very high. (prob = 0.95)
Research Methods Tony Fleet
155
HOPE
Random Chance
• In time, all things are possible. Mountains might
crumble, pigs might fly, you might win the lottery.
• If we wait around long enough these events will
happen, because each event has a small, but
finite probability.
• Random chance will dictate the day, the year, or
the millennium in which these occur.
Research Methods Tony Fleet
156
Making predictions
HOPE
•
•
•
•
Because of the way that statistics works, we can use
the features of data, and the idea of random chance to
make predictions.
If we were to walk into the dining hall, and select a
student at random, we could predict the likelihood that:
1. The student has an IQ between 85 and 115 (about 70%)
2. The student has an IQ over 130 (about 2.5%)
The first is a high probability event, and that’s what we
would normally expect to happen.
Clearly the second is a low probability event, and
therefore we would be very surprised if it occurred.
Research Methods Tony Fleet
157
Hypothesis Testing
HOPE
•
•
We all work on assumptions every day. Let us
suppose that you have come to college on what
you think is a normal day, only to find the rooms
deserted, no lecturers, no students.
What is the explanation?
Is it that:
1. You came in on a Sunday by mistake,
thinking it was Monday
2. The college has been evacuated due to a
leak at a nearby toxic waste incinerator.
Research Methods Tony Fleet
158
Hypothesis Testing
HOPE
• Most people would immediately suggest (1),
because (2) is a very low probability event.
• However, we can use the situation to reason thus:
If we assume that Today is Monday, and we
come in and find the college deserted, then the
reason must be (2).
However, this is such a low probability event
that we would not normally expect to witness it.
This means that something is wrong.
It is more likely that our original assumption is
incorrect, and Today is NOT Monday.
Research Methods Tony Fleet
159
HOPE
Hypothesis Testing
This is formalised into a process which goes like this:
• We set up two competing hypotheses (normally
called H0 and H1).
• We make observations in data, and calculate the
probability of these observations occurring by
random chance. We base this calculation on the
assumption that H0 is true.
• If the probability is very low, we reject H0 (the null
hypothesis), and accept H1, the alternative
hypothesis.
Research Methods Tony Fleet
160
HOPE
Hypothesis Testing
An Example
Let us set up two competing hypotheses:
 H0: MSc. Computing is equally attractive to men
and women.
 H1: MSc Computing is more attractive to one
gender than the other.
We now observe the number of students in a group,
and count the number of males and the number of
females, and ask what is the likelihood of observing
such, if the subject were really equally attractive.
.
Research Methods Tony Fleet
161
HOPE
Hypothesis Testing
An Example
Suppose in the class there are
8 men
2 women
• If H0 is true, then the probability of this event is
p=0.01; this is 1%, extremely low .
• We therefore reject H0, and accept H1,
concluding that MSc. Computing is more
attractive to one gender than the other.
Research Methods Tony Fleet
162
HOPE
How low can you go?
• The question now comes, how low does a
probability have to be in order for you to conclude
that the event is not simply occurring as a result of
random chance.
• For most circumstances, we would use p<.05 or
less than 5%, and we would say such a result is
statistically significant
• If p <.005, then the result is highly significant.
• If p< .0005, then the result is very highly significant.
Research Methods Tony Fleet
163
HOPE
What does this all prove?
• Your results may not be statistically significant,
that does not mean that they are not true; it simply
means that the data you collected was not
sufficient to decide things one way or another.
• If your results are significant, then you need to
decide whether the interpretation that you are
putting in them is actually the correct one; there
may be other explanations for the results than the
one that you have stated as H1.
• In other words, you should be cautious when you
write this up; be analytical, and question
everything.
Research Methods Tony Fleet
164
HOPE
6e Examples of Statistical Tests
• This section looks at some examples
of statistical tests that you can
undertake in different contexts.
• It distinguishes between Parametric
and non-Parametric testing
Research Methods Tony Fleet
165
HOPE
Parametric Testing (1)
• Parametric Testing assumes that the data you are
working with follows a Normal Distribution.
• You can use these techniques if
 Your data is measured
 Your data is counted, but there is a large number in the
count
 Your data is in numerical categories, or categories which
can be translated into numbers, and the overall shape of
the data is a ‘bell’ curve.
 You are working with averages, based on samples of 10
or more.
Research Methods Tony Fleet
166
Parametric Testing (2)
These are some samples of Parametric Tests
HOPE
• Z-Test
– Single sample: Null hypothesis is that the average of a
sample is equal to some theoretical value.
• Paired Sample T-Testing
– Two samples: Null hypothesis is that there is no
difference between the two samples
• ANOVA
– Three or more samples; null hypothesis is that there is
no difference between the samples.
• Correlation
– Two measurements taken on the same data; null
hypothesis is that the measurements are not related in
any measurable way.
Research Methods Tony Fleet
167
HOPE
Non-Parametric Testing (1)
• Non-Parametric Testing assumes that the data you
are working with does not follow a Normal
Distribution.
• You need to use these techniques if
 Your data is counted, with low numbers in each
category
 Your data is in categories which cannot be
translated into a numerical scale
 Data where the overall shape of the data is not a
‘bell’ curve.
 You are working with medians or other statistical
summary values which are not averages.
Research Methods Tony Fleet
168
Non-Parametric Testing (2)
These are some samples of Non-Parametric Tests:
HOPE
• Chi-Squared Test
– Crosstabs or Contingency Tables showing the effect of two
different factors: Null hypothesis that the two factors are
independent.
• Mann-Whitney U-test
– Two samples: Null hypothesis that there is no difference
between the two samples
• Phi Test
– 2 x 2 Crosstab; Null hypothesis is that the factors are
unrelated.
• Spearman’s Correlation
– Two judges asked to rank a series of items; null hypothesis
is that the rankings are statistically unrelated.
Research Methods Tony Fleet
169
HOPE
Selected Test Examples
•
•
•
•
t-Test
ANOVA
Correlation & Regression
Chi-Squared Test
Research Methods Tony Fleet
170
t-Test
HOPE
This is used to test whether a the mean of an observed
sample is significantly different from an hypothesised
value.
• In the example shown, 12 respondents were
given a Likert-scaled questionnaire, to determine
their attitude towards chat-room ‘behaviour
guidelines’.
• A neutral score would be 2.5
• A t-test can be used to test the hypothesis:
Ho: Respondents disagree with (or are neutral to)
the guidelines
• The t-statistic is calculated to be t = 2.72 at
p< .05; so we reject Ho, and conclude that the
respondents agree with the guidelines.
Research Methods Tony Fleet
Average of
Likert Scores
3.4
2.5
2.7
3.1
3.6
3.4
2.8
2.4
3.5
1.9
2.5
3.2
171
Package Package Package
A
B
C
8.4
4.7
5.1
7.5
5.1
4.7
9.1
6.2
6.2
6.7
7.5
6.3
6.2
3.8
5.4
7.8
6.2
5.6
HOPE
ANOVA
This is used to test whether a the means of three or more
samples are significantly different from one another.
The example above shows the average ratings from 6
different evaluators of three different web-authoring
packages, A, B & C on a variety of ten-point scales.
We can test Ho: There is no difference between the ratings of
the packages, using an ANOVA (Analysis of Variance) test.
The results below show a p value of 0.004881
We reject Ho, and conclude that the packages ARE different.
The result is
highly significant.
ANOVA
Source of Variation SS
Between Groups 16.81333
Within Groups
16.27167
Total
33.085
Research Methods Tony Fleet
df
MS
F
P-value
F crit
2 8.406667 7.749667 0.004881 3.682317
15 1.084778
17
172
HOPE
Correlation & Regression
This is used to determine whether there is a
relationship between two variables, and if so,
what?
The data collected shows the download times
for 10 files of different sizes.
There is a correlation of 0.998864 between the
two variables, showing that a very strong
relationship exists.
The regression analysis below shows that this
can be expressed as:
Download Time = 70.65 x File Size - 1.49
File Size Download
(MB)
time (secs)
0.018
1.3
0.056
3.4
0.094
5.7
0.132
7.1
0.271
15.0
0.408
28.4
0.846
60.3
1.284
87.1
1.922
128.3
2.362
170.8
Coefficients
Standard Error t Stat
P-value Lower 95% Upper 95%Lower 95.0%
Upper 95.0%
Intercept
-1.49155 1.410594 -1.05739 0.321215 -4.74438 1.76129 -4.74438 1.76129
X Variable 1
70.65 1.292702 54.65297 1.39E-11 67.66902 73.63098 67.66902 73.63098
Research Methods Tony Fleet
173
HOPE
Chi-Squared
Test
Under 21
21-30
31-40
41-50
50+
Men
87
64
28
31
17
Women
85
32
17
14
4
This is used to determine whether the effects of two or more
factors are independent.
The data above shows the number of respondents of each
gender in different age categories who expressed interest in
using Blue Tooth technology within the next 12 months.
A Ch-Squared test is used to test the hypothesis: Ho: There is
no relationship between age and gender in potential Blue Tooth
use.
The probability value of the Chi-Squared Statistic is p = 0.00893
We would reject Ho, and conclude that there are differences
between the age profiles of men and women potential users of
Blue Tooth.
Research Methods Tony Fleet
174
HOPE
Which test?
• http://research.med.umkc.edu/tlwbios
tats/choosetest.html
• http://staff.harrisonburg.k12.va.us/~g
corder/stats_Index_Page.html
Research Methods Tony Fleet
175
HOPE
Statistical Testing
• http://www.statsoft.com/textbook/stat
home.html
• http://faculty.vassar.edu/lowry/webtex
t.html
• http://www.tufts.edu/~gdallal/LHSP.H
TM
Research Methods Tony Fleet
176
HOPE
Statistical Testing
References (SW Library)
• Ferguson, George A. - Statistical analysis in
psychology and education. - 3rd. - Tokyo :
McGraw-Hill Kogakusha, 1971. - m0891134
• Peers, Ian S.. - Statistical analysis for education
and psychology researchers. - London : Falmer
Press, 1996. - 0750705051
• St. John, P. R.. - Methods of statistical analysis of
fieldwork data / P.R. St. John, D.A. Rich. Sheffield : The Geographical Association, 1989. 1899085165
Research Methods Tony Fleet
177
HOPE
6f Analytical Tools
• In order to analyse the data, you will need
a package that is capable of arithmetic and
other data manipulation, and will allow you
to display the data in the form of tables
and charts, as well as to perform statistical
tests.
• There are several packages that you can
use in order to analyse your data, but two
of the main ones are:
– Microsoft Excel
– SPSS
Research Methods Tony Fleet
178
HOPE
Excel or SPSS?
• Excel will be OK to use if all your data is
numeric, and that you have a clear idea of
what you want to do, and how to do it. Excel
is good at data display, but can only perform
limited statistical testing, mainly parametric.
• SPSS is better at statistical testing, is highly
versatile and can do lots of clever stuff. You
will need this if you intend to analyze
questionnaire data, or have lots of complex,
categorical data and need to do nonparametric testing.
Research Methods Tony Fleet
179
HOPE
Using Excel
• Excel is available on all computers, and
there is an extensive help facility
embedded in it.
• You will probably have worked with it
before, so your ‘learning curve’ is fairly
gentle.
• If you have only a few items to display,
and you are not going to do any heavy
statistical testing, then Excel is probably
the easiest tool to use.
Research Methods Tony Fleet
180
HOPE
Example of Excel Use (1)
• You can type data directly
into an Excel worksheet.
• Excel has formulae to
calculate statistics, such
as:
= Average(C3:C8)
= StDev(C3:C8)
Research Methods Tony Fleet
181
Example of Excel Use (2)
Expert Rankings
9
8
7
6
Ranking 5
4
3
2
1
0
W
eb
W site
eb
A
si
t
W
eb e B
si
te
C
HOPE
• This is an example of a chart easily produced from the
data on the previous slide using Excel.
Website A
Website B
VI
Research Methods Tony Fleet
V
IV
III
II
Website C
I
Expert
182
Example of Excel Use (3)
HOPE
• Excel can produce a range of
descriptive statistics very easily.
Website A
Website B
Website C
Mean
7 Mean
5.5 Mean
2.1667
Standard Error
0.8563 Standard Error
0.4282 Standard Error
0.3073
Median
7.5 Median
5.5 Median
2
Mode
9 Mode
5 Mode
2
Standard Deviation 2.0976 Standard Deviation 1.0488 Standard Deviation 0.7528
Sample Variance
4.4 Sample Variance
1.1 Sample Variance
0.5667
Kurtosis
-1.5496 Kurtosis
-0.2479 Kurtosis
-0.1038
Skewness
-0.5851 Skewness
0 Skewness
-0.3126
Range
5 Range
3 Range
2
Minimum
4 Minimum
4 Minimum
1
Maximum
9 Maximum
7 Maximum
3
Sum
42 Sum
33 Sum
13
Count
6 Count
6 Count
6
Research Methods Tony Fleet
183
HOPE
Using SPSS
• SPSS is “Statistical Package for the Social
Sciences” and is an industry standard tool
for data analysis.
• If you are doing lots of work with numerical
data (e.g. analysing questionnaires) then
you will need to use SPSS.
• SPSS is available via the F: drive, and it is
also available for £5 from the technician in
the Psychology Lab (GLB)
Research Methods Tony Fleet
184
HOPE
SPSS Tutorial
Research Methods Tony Fleet
185
HOPE
Example of SPSS Use
• Part of a study looks at file
upload times under 10
different conditions, using two
types of browser: Netscape
or Internet Explorer.
• The same circumstances
relate to each pair of
readings
Research Methods Tony Fleet
Netscape Int Exp
13.40
13.80
12.90
15.30
16.40
14.70
15.80
17.10
12.70
11.80
12.90
15.70
11.70
14.60
11.40
14.20
15.30
16.10
11.50
12.10
186
HOPE
Setting Up Variables
• First of all we set up descriptors for each of the
two variables:
– Netscape
– Internet Explorer
Research Methods Tony Fleet
187
HOPE
Entering Values
• Then we Enter the 10
values for each
variable, corresponding
to the different cases.
Research Methods Tony Fleet
188
HOPE
Analysing the data
• We can select
graphical and
numerical
summaries
Research Methods Tony Fleet
189
Summaries
HOPE
Results:
• It looks as if Netscape
generally takes longer to
upload files than IE: 14.39
secs for NS as opposed to
13.55 secs for IE
• We might try to test this as
a hypothesis.
Research Methods Tony Fleet
190
Results of a T-Test
We test:
H0: there is no difference
between sample means.
HOPE
H1: There is a difference
between sample means.
P = 0.158.
Unfortunately this is greater
than .05, and not low enough
to reject the null hypothesis;
the result is not significant.
Research Methods Tony Fleet
191
HOPE
Using SPSS
• http://www.utexas.edu/its/rc/tutorials/
stat/spss/spss1/index.html
• http://www.strath.ac.uk/IT/Docs/SPS
S/spss.html
• http://www.leeds.ac.uk/iss/documenta
tion/math.html
• http://www.indiana.edu/~statmath/stat
/spss/win/giant.html
Research Methods Tony Fleet
192
HOPE
Using SPSS
References (SW Library)
• Dancey, Christine P.. - Statistics without maths for
psychology : using SPSS for Windows / Christine. 2nd ed. - Harlow : Prentice Hall, 2002. - 0130336335
• Kinnear, Paul R.. - SPSS for Windows made simple :
release 10 / Paul R. Kinnear, Colin D. Gray. - [New
ed.]. - Hove : Psychology Press, 2002. - 1841691186
• Pallant, Julie. - SPSS survival manual : a step bystep guide to data analysis using SPSS for Windows.
- Buckingham : Open University Press, 2001. 0335208908
• West, Robert. - Computing for psychologists :
statistical analysis using SPSS and MINITAB. London : Harwood Academic, 1991. - 371865086x
Research Methods Tony Fleet
193
HOPE
7. Problems with The
Research Process
HOPE
Research Pitfalls
• Finally, we look at some problems
that you might encounter which
invalidate your research – and how to
prevent these.
Research Methods Tony Fleet
195
HOPE
Some Research Pitfalls (1)
Researcher Bias
• Problem: Introducing your personal
prejudices or preconceptions into the
research, and obtaining invalid results.
• Solution: Counter by an attitude shift: we
do not try to prove a hypothesis is true,
rather we test whether or not a hypothesis
is true, and keep an open mind as to the
result.
Research Methods Tony Fleet
196
HOPE
Some Research Pitfalls (2)
The Hawthorne Effect
• Problem: Subjects behave differently
because they know they are participating
in Research.
• Solution: Use of control group in
experiments; concealed observation
techniques; surveillance methods;
naturalistic data collection methods
Research Methods Tony Fleet
197
HOPE
Some Research Pitfalls (3)
The Halo Effect
• Problem: If Subjects rate something as
good or bad overall, they are likely to rate
all features of it as good or bad, without
discriminating between them.
• Solution: In questionnaires & evaluations,
use a variety of similar questions,
positively & negatively worded.
Research Methods Tony Fleet
198
HOPE
Some Research Pitfalls (4)
Pseudoscience
• Problem: Setting out to prove a
hypothesis by only looking for evidence
which confirms it, and disregarding
evidence which contradicts it.
• Solution: Explicitly stating what would be
disconfirming evidence for a hypothesis,
and setting out to look for it.
Research Methods Tony Fleet
199
HOPE
Some Research Pitfalls (5)
Sample Bias
• Problem: Getting misleading results,
because the sample on which you have
based your analysis is consistently
unrepresentative.
• Solution: Identifying the group of people,
objects or events you are trying to
generalise about, and selecting a sample
which has the same characteristics as that
group, or is a random selection from it.
Research Methods Tony Fleet
200
Some Research Pitfalls (6)
HOPE
Overgeneralisation
• Problem: Making sweeping claims that the
results that you have observed from one or two
cases will be present in the population at large.
• Solution: Be cautious in your claims, and use
moderate language, such as ‘likely’ or
‘probable’ causes; only claim that your results
are ‘significant’ when they are actually
statistically significant, after carrying out a
statistical test and rejecting the null hypothesis.
Research Methods Tony Fleet
201
Some Research Pitfalls (7)
HOPE
Mistaking Association for Causality
• Problem: Thinking that because one variable
increases (or reduces) as another increases (or
reduces) that we have found a ‘cause’ of an event. For
example, the variable ‘difficulty of navigation’ in a
multimedia package may be correlated with the
number of buttons. However, it is the underlying
navigational structure, which is at the root of the
problem, which coincidentally generates the number of
buttons.
• Solution: Understanding that correlation is about
variation, and that multiple factors often act together to
cause particular effects, and seeking such factors.
Research Methods Tony Fleet
202
Some Research Pitfalls (8)
HOPE
Ex Post Facto Hypothesizing
• Problem: Deciding on your research
hypotheses after you have seen the data, and
noticed some results which might be
significant. In any data set, there are bound to
be ‘blips’ which are freak results, and you are
in danger of basing your results on these.
• Solution: Decide on your research
hypotheses and the test you will do before you
collect data. That way the tests will be fair, and
your results valid.
Research Methods Tony Fleet
203
HOPE
8. Summary
HOPE
Summary
Things to consider
• Devising your Research Questions
• Selecting Your Methodology
• Deciding on Your Methods of Analysis
• Some Do’s & Don’ts
Research Methods Tony Fleet
205
Relationship between Elements
Conduct Literature Review
HOPE
Select Research Questions
Devise Methodology &
Research Instruments
Apply Methods & Instruments
Perform Statistical Analysis
• To a large extent,
this process is
iterative.
• You need a good
idea of the data
you intend to
collect, and how
you will use it,
before you
devise your
research
questions.
Test Hypotheses & Draw Conclusions
Research Methods Tony Fleet
206
HOPE
Devising Research Questions
• Your Research Questions should arise naturally
out of the Literature Review.
• However, in your Research Project you should
only devise questions that you are capable of
answering.
• That means when you ‘phrase’ them, you should
have a good idea of the methods that you will use
to answer them, the data that you will collect, and
how that data will be analysed.
• What you should not know,is the answer to those
questions!
Research Methods Tony Fleet
207
HOPE
Selecting your Methodology
• Your methodology should be devised specifically
to answer your Research Questions
• It should draw on the methods used by the
studies that have formed a part of your Literature
Review.
• It should also be created in such a way that the
data that it produces is amenable to analysis
• That means you should be able to take advantage
of the different forms of analysis, and the different
tools available, in order to enhance your research.
Research Methods Tony Fleet
208
HOPE
Deciding on the Analysis
• The method of Analysis needs to be appropriate
to the data collected.
• It also clearly needs to be sufficient in order to
answer the Research Questions .
• It should also use methods that have been
established by previous researchers that you
have highlighted in your Literature Review.
Research Methods Tony Fleet
209
HOPE
Advice – some Dos
• Plan ahead; think through your methodology and
your analysis before framing your Research
Questions.
• Include at least one ‘Research Hypothesis’, which
you know you are going to test; this adds
credibility to your dissertation.
• Spend time looking at how previous researchers
have pursued their research; draw on their
experience, adapt their methods and modify their
research instruments… but don’t forget to credit
them in the write-up!
• Consult with your supervisor, take advice.
Research Methods Tony Fleet
210
HOPE
Advice – some Don’ts
• Do not set out to ‘prove’; set out to ‘investigate’.
• Do not try to take on too wide a topic; this is an
MSc. Dissertation, and you should be working in a
confined area, but achieving true depth of
analysis
• Do not expect to have statistically significant
results; most Masters’ dissertations fail to prove
anything one way or the other. However, you are
expected to know whether the results are
statistically significant or not.
Research Methods Tony Fleet
211
HOPE
… and Finally
• If you are doing a questionnaire, or data analysis,
you may need to have specialist advice.
• You can sign up to see me, even though I am not
your supervisor.
• However, if you come for a consultation, I expect
you to know what you want to do, and have a
good idea of how to do it; you should come to me
to check that you are on the right lines, not to find
out what to do.
• Contact Tony Fleet in FML 213, telephone: 0151291-3525 or email me at fleeta@hope.ac.uk
Research Methods Tony Fleet
212
HOPE
References and Bibliography
Comprehensive accounts of Research methods:
• http://www.socialresearchmethods.net/kb/
• http://www.socialpsychology.org/methods.htm
Huge list of resources & links:
• http://www.geocities.com/Athens/3238/bookmark.htm
Undertaking a Research Project: Guidelines
• http://dec.bournemouth.ac.uk/staff/scrowle/Teaching/
FinalYear/ProjectWorkshops/index.html
Research Methods Tony Fleet
213
HOPE
Research Methods Books in
the Sheppard-Worlock:
• Fowler, Floyd J.. - Survey research methods. 2nd ed. - Newbury Park, Calif.; London : Sage
Publications, 1993. - (Applied social research
methods series ; v.1). - 0803950489
• Lehman, Richard S. - Statistics and research
design in the behavioural sciences. - Pacific
Grove, Calif. : Brooks/Cole, 1991. - 0534138780
Research Methods Tony Fleet
214
Other Research Design Books
in the Sheppard-Worlock:
HOPE
• Handbook of applied social research methods / edited by
Leonard Bickman and. - London : SAGE, 1997. 076190672x
• Human centred methods in information systems : current
research and practice. - Hershey; London : Idea Group
Pub, 2000. - 1878289640
• Qualitative research : theory, method and practice /
edited by David Silverman. - London : Sage, 1997. 0803976666
• Research design. - A. - Milton Keynes : Open University
Press, 1979. - (Social sciences, a third level course
research methods in education and Social Sciences ). 0335074227
Research Methods Tony Fleet
215
Past MSc. Dissertations
HOPE
• You should examine what other students have done,
especially in an area which is akin to yours.
• Remember: if it is on the shelves, it passed!
Examples:
• Broadbent, George Ernest. - The role of evaluation in user
interface design : a practical implementation. - Liverpool :
University of Liverpool, 1997. - p7270369
• Lomas, Joanne. - The research and design of a computer
based training package for children. - Liverpool : University
of Liverpool, 2001. - M0002708LO
Research Methods Tony Fleet
216