Research Methods II
Method Design
"When you change how you see the world, your whole world changes.”
Aman Motwane
Game Plan

Designing your research



Collecting Data




Quantitative
Qualitative
Primary and Secondary Data
Sampling
Measurement
Qualitative Data Collection



Interviews
Surveys
Observation
Method design in the thesis process
Formulate and clarify your
research question
Chapter 1
Review the Literature/Theory
Chapter 2
Formulate your research
design
Chapter 3
Answering research question
Sampling
Measurement
Analyze your data
Chapter 4
Qualitative Methods
Quantitative Methods
Write your Results and
Discussion
Chapter 5
The methods process

Research Design





How are research questions answered?
Ex: A mixed method study investigating the relationship between cultural acuity
and practitioner effectiveness
Data Collection / Sampling / Setting/

Who did you study and how did you select them?

Ex: Population is all employees of Intel – sample is Finance
Department
Measurement

How are numbers assigned to concepts?

Ex: Quality of Work Life – measured by the QOL index
Data Analysis

How are data manipulated and interpreted?

Ex: Regression Analysis
Alternative Research Designs
Creswell (2003)
Quantitative
research
Qualitative
research
Mixed methods
research
Experimental designs
Non-experimental
designs, such as
surveys
Predetermined
Instrument based
questions
Performance data,
attitude data,
observational data and
census data
Statistical analysis
Study is exploratory
Case studies
Ethnographies
Emerging methods
Open-ended questions
Interview data
Observation data
Document data
Audiovisual data
Text and image
analysis
Elaborate and expand
findings of one method
with another
Concurrent – converge
quantitative and
qualitative data
Both predetermined
and emerging methods
Both open-ended and
closed-ended questions
Multiple forms of data
drawing on all
possibilities
Statistical and text
analysis
Your design will drive your data collection
The Research Onion
Techniques and
procedures
Time horizons
Choices
Strategies
Approaches
Philosophies
Positivism
Realism
Data
collection
and data
analysis
Interpretivism
Deductive
Experiment
Survey
Objectivism
Case
CrossMono-method studyAction
Subjectivism
sectional
research
Mixed
methods
Grounded
Multi-method
Longitudinal
Pragmatism
Ethnography
Archival research
Constructivism
theory
Inductive
Functionalist
Interpretive
Radical
humanist
Positivism versus Constructionism
Constructionism




Knowledge is subjective
It’s the product of the context
in which its constructed
(ex. Copernicus)
Scientific knowledge is not a
“direct representation of the
natural world”
It’s based on systems of
thought that are culturally
and historically bound
Positivism






All knowledge is derived from
experience whether of the
mind or of the senses.
Evidence by what we can see,
hear, touch, taste, or smell
Is the basis of “The Scientific
Method”
Emphasizes the importance of
testability, reliability, validity,
and repeatability
Asserts that the researcher is
“bias-free”
Claims “objectivity”
Your theoretical perspective leads to your research design
Qualitative vs Quantitative Design
Quantitative







Tests or verifies theories or
explanations
Identifies variables to
study
Relates variables
Observes and measures
information numerically
Uses unbiased approaches
Uses statistical procedures
Largely deductive
reasoning
Qualitative







Takes place in the natural
setting
Uses multiple methods
that are interactive and
humanistic
Is emergent
Fundamentally interpretive
Researcher views social
phenomena holistically
Researcher reflects on who
he or she is in the inquiry
and is sensitive to how
that shapes the study
Largely inductive
reasoning
Collecting Primary Data

Survey Research

Interviews
Observation Methods
 Experimental Research

Collecting Secondary Data

Consider who would have collected
data


Governments, NGOs, Association or
interest group etc
Good Sources







Statistical Universe
Statistical Abstracts
Historical Statistics
WDI – World Development Indicators
IMF – International Financial
Statistics
United Nations – UN Stat
Economic Report of the President Historical tables
Secondary data: data that has been collected for some other purpose
Sampling: who AND how many to ask when you
can’t study the entire population


You are conducting a study of professional success factors
among CC alumni who are external consultants. What is
your best approach?
Two key questions:


Sampling is often a major limitation


Is the sample biased? Is the size adequate?
Failure to identify the population, sample of convenience,
volunteerism,
Sample size – bigger is not always better – “it all
depends…”
Sampling Techniques
Sampling
Probability
Simple
Random
Non-Probability
Stratified
Random
Systematic
Quota
Cluster
Snowball
Purposive
Convenience
Self-Selection
Your goal is to produce valid and credible sample data and
statistics that match the precision needed for the study
Probabilistic Sampling





Simple Random: every member of the population has an
equal probability of being sampled (use random # tables)
Systematic Sampling: select at intervals (e.g,. Starting at
18, every 50th listing is selected)
Stratified Sampling: grouping the study population into
strata and selecting a random sample within each stratum
(e.g., dividing the population by education – less than high
school, high school, no college diploma, college diploma,
then selecting a random sample)
Cluster: random selection of groupings from which all
members are chosen for the sample (e.g., geographic
groupings, intact groups)
Multi-stage: select a cluster, then sampling members of
the selected clusters
Non-probability
Non-probabilisticSamples
Sampling





Quota: ensures that certain characteristics of a population
sample will be represented (e.g., need 20 small business
owners, 10 minority, etc.)
Purposive: (Judgment) experienced individual selects
sample based on judgment about particular characteristics
(e.g,. Political polls)
Snowball: initial respondents are asked who else would
be appropriate for the study
Self-Selection: participants self-select into the study
Convenience: selecting people (units) who are most
conveniently available (e.g., college professor using
students)
How many interviews, surveys?

It all depends



Survey



Research Question
Variance of the population
Minimum of 30 for most statistical tests
More is better – to a point!
Interviews

Ask until you start to get repetition
Remember – the intent in qualitative research is not to generalize,
but to elucidate the particular, the specific
Measurement:
Variable
DV:
IV:
IV:
IV:
Variable definition
Conceptual Definition
(include source)
Operational
Definition required
for measurement
(include source
where appropriate)
Measurement: Research questions and variables

Q1: What is the relationship between faculty
sleep deprivation and clarity of faculty
communication in a thesis workshop?



Dependent variable => clarity of communication
Independent variable => sleep deprivation
Q2: What is the relationship between clarity of
faculty communication and student learning
during a thesis workshop?


Dependent variable => student learning
Independent variable => clarity of communication
Faculty sleep deprivation, communication clarity
and student learning
Variable
Conceptual Definition
(include source)
Operational
Definition required
for measurement
(include source
where appropriate)
Clarity of faculty
communication
Faculty stays on topic, uses
appropriate examples, invites
and answers questions – what,
how, who, when and where
Content analysis of lecture
Sleep deprivation
Number of nights over a one
week period where sleep falls
below 5 hours a night
# of hours of sleep per
night
Faculty knowledge of
subject matter
(moderating variable)
Demonstrated level of skill,
ability and experience with
subject matter
# years experience,
Education
Student learning
Degree to which students
demonstrate the ability to apply
concepts from the workshop to
their project
Application of concepts in
projects (content analysis)
Measuring Variables: Scale construction


Scale – any measuring instrument (questionnaire,
interview, test) composed of one or more items
(questions, observations, etc…) that have logical
relationships with one another.
Standardized measures – don’t reinvent the
wheel. Source for standardized measures –


If you have to create your own


Journal articles, books, other thesis or dissertation
projects
KISS
Measure twice and cut once… the value of beta
testing your approach
Scale Example: Job Characteristics Hackman
and Oldham, 1975

Measures the degree to which a job is designed to be
motivating (the motivating potential of a job)

Five dimensions: skill variety, task identity, task
significance, autonomy and feedback





My job requires me to use a variety of skills.
My job allows me to see the finished products of my work.
Doing my job well affects others in some important way.
My job is designed so that I know when I have performed well.
My job allows me freedom to work with minimum supervision.
Scale Examples

Participation/Involvement Scale





I feel free to suggest new and better ways of doing
things.
I am asked how we can improve the way my work
group operates.
Sufficient effort is made to get the opinions and ideas
of people in this work unit.
Suggestions made by unit personnel are implemented
in our daily work activities.
Satisfaction




In general, I am satisfied with my job.
I have a sense of personal fulfillment at the end of the
day.
I am a valued member of my unit.
I would recommend an assignment in my unit to a
friend.
Validity and Reliability

Validity – does it measure
what it is supposed to
measure?


Many types of validity that
you need to consider
Reliability – is it
consistent?

Increase reliability by:




Sample size
Multiple items
Multiple raters
Multiple measures
Qualitative Data Collection
Interviews
 Surveys
 Documents
 Audiovisual materials
 Observations

Surveys / Interviews
Interviews
 Telephone Interviews
 Group Interviews/ Focus Groups
 Self-Administered Questionnaires Mail,
Fax, Internet


Survey Monkey
Interview Question Design: Good
Questions
Sound conversational
 Use words participants would use
 Are easy to say
 Are clear
 Are short
 Are open-ended
 Are one-dimensional

Question Design: Probing Questions
How do you mean that?
 Would you give me an example of what
you mean?
 Tell me more about that…
 I don’t understand
 Anything else?

Example of a Questioning Route
Opening Questions
 Introductory Questions
 Transition Questions
 Key Questions
 Ending Questions

Other questioning techniques
Listing things
 Rating things
 Picture sort
 Drawing a picture

Questioning Pitfalls

Avoid asking “why”?

Can put respondents on the defensive
Be cautious about giving examples
 Double-barreled questions
 Abbreviations / Jargon / Slang
 Double-negatives

Interview template
•Discusses setting
•Transcribes interview
•Adds reflection notes where
appropriate
Designing Questionnaires/Surveys



Survey Monkey
Zoomerang
InstantSurvey
CC has a Survey Monkey account. See
Library for account name and
password.

Major Decisions in
Questionnaire Design





What should be asked?
How should each
question be phrased?
In what sequence
should the questions be
arranged?
What questionnaire
layout will best serve
the research objectives?
How should the
questionnaire be pretested?
A survey is only as good as the questions it asks
Phrasing Questions
Simple-Dichotomy
 Determinant Choice
 Frequency determination
 Attitude rating scale
 Checklist

Observational Studies

What can be
observed?






Human behavior and
action
Verbal behavior
Expressive behavior
Spatial relations
Temporal patterns
Physical objects
Observational Studies

Content of Field Notes








Space
Actor
Act
Activity
Objects
Time
Goal
Feeling
Observation Field notes
Observation Exercise
Partner with someone in class
 Decide on what you could observe to
answer a question about the CC
experience
 Go to the place for 5 minutes


Observe and take notes
For purposes of this exercise, please do not disturb other classes
2 people can go into room 117 or room 128if you would like
to observe a classroom
Observation Follow-up

General comments



Most challenging? awkward?
Easy to record?
Impact of observer

How could you manage that?
Documents
E-mails, Letters, Memos, reports and
minutes of committees
 Annual reports
 Could ask participants to keep journals
(case studies and narrative research)

Administration
No class tomorrow – Aju and I will be
available to meet with you (Julie after
10:30)
 Research Question presentation Monday.
Please bring your presentation on a
memory stick.
