Research Design
Procedures
1
Research Design
 Forming your action plan
 Deciding on the Who and When
 Defining all concepts and terms
2
Research Design
 Three purposes for research:



Exploration
Description
Explanation
 Or



Descriptive- existing conditions
Normative- observed vs. intended
Impact- can it be attributed to programs?
3
Exploration
 To gain familiarity with a topic
 Typically done to:



Satisfy a researcher’s curiosity and desire to
understand
“Test the water” for a more extensive study
Develop methodology to be used in a
subsequent study
4
Description
 Provide context for situations and events
 Typically based on observation and reporting
 Observation is systematic
 Example: U.S. Census
5
Explanation
 Attempts to address the question of “why”
 Tries to get at reasons and underlying causes
 Example: Not “would you vote for McCain”
but “why would(n’t) you vote for McCain?”
6
GAO- Designing Evaluations

Considerations
1.
2.
3.
4.
5.
6.
7.
Kind of information to be acquired
Sources of information (i.e. types of respondents)
Methods of sampling (i.e. random)
Methods of collecting data (i.e. interviews, surveys)
Timing and frequency of information collection
Basis for comparing outcomes
Analysis plan
7
Research Design: Experiments
 Experiments aim to control one variable or
set of variables in order to determine their
relationship to and/ or impact on another set
of variables.
 Types of studies:

Experiment


Uses a random sample
Quasi-experiment

Does not use a random sample, must try to
correct for error through statistical tests
8
Impact of a Program/Service Involving
an Experimental Design
IMPACT =
Outcome of experimental group receiving the
treatment compared to control group
Information Literacy Instruction
9
Classic Design
 Two group pre- and post- test
 One experimental group
 One “untreated” control group
 Compare outcomes to assess impact
 Problems with this?
10
Solomon 4 Group Test
 Classic Design expanded to include two sets:


One set has experimental group and control
group who both receive pre- and post-tests
One set has experimental group and control
group who receive only post- tests.
 Advantages over classic model?
11
 Solomon Four Group Design
Group 1
Control 1
Group 2
Control 2
Before Treatment
After
no
no
yes
yes
yes
yes
yes
yes
yes
yes
yes
no
12
Time Series Design
 Repeats testing twice (or more) to establish a
trend in the data independent of the
experiment
Pretest Treatment Posttest Pretest Treatment Posttest
Experiment
yes
Control Group yes
yes
no
yes
yes
yes
yes
yes
no
REPEAT
yes
yes
13
EXAMPLES
 Pretest/posttest design with control group
Experiment
Control Group
pretest treatment
posttest
yes
yes
yes
yes
yes
no
Pretest/posttest design without control group
Experiment A
Experiment B
pretest treatment posttest
yes
yes
yes
yes
yes
yes
14
Case Study
 Basis of selection: representative, typical,
cluster, probability, etc.
 Multiple methods of data collection
15
Populations and
Statistical Sampling
16
Populations
 Population- the entire group/ universe under
study
 Sample- a portion of a population of possible
information sources
 Sampling- methods for selecting these
sources
17
Research Design: Action Plan
(continued)
 Who is studied


population
Sample

Is sample reflective of population?
 Where

Sampling?
 When

Sampling?
18
Who (or What) is being studied?
 Units of analysis: the what or whom being studied. In social
research the most typical units of analysis are individual people.
 Can be: individuals, groups, organizations, social interactions,
social artifacts
 Examples:
 Library Users or Non-Users
 First-year Students
 Senior Citizens
 Public Libraries
 Also
 ILLs
 Biographies, Mysteries, Audiobooks (i.e. collections)
 Web sites
19
Beware: Pitfalls of Analysis
 Ecological Fallacy:

Something learned about a group says
something about the individuals making up
that group.
 Reductionism

An attempt to explain phenomena in terms of
limited or lower-order concepts.
20
Who is being studied
 How to select a sample?
21
Sampling- Three Options
 Census- collecting information from the entire
group making up a population

Like the decennial census
 Judgment sampling- making conscious
choices

Convenience Sampling- what’s available
 Probability/ Statistical Sampling- left to
chance, each member of a population has an
equal chance of being chosen
22
Sampling: Purpose
 Representativeness
 Sample has roughly the same distribution of
characteristics as the population from which it
is drawn.
 Nevertheless, each sample will differ from
each other, as well as from the population
 Can determine the amount of error
23
Probability Sampling (1)
 Random sampling: Each member of the
population has an equal and known
probability of being selected
 Systematic sampling: Each member of the
population is either assembled or listed, a
random start is designated and then
members of the population are selected at
equal intervals… nth intervals
24
Probability Sampling (2)
 Stratified: Each member of the study
population is assigned to a group or stratum,
then a simple random sample is selected
from each group or stratum
25
Probability Sampling (3)
 Cluster: Each member of the study population
is assigned to a geographically-defined group
or cluster. Clusters are then selected at
random, and members of a selected group
are represented in the sample.
 http://www.claritas.com/MyBestSegments/Default.jsp
 Role of GIS and TIGER files
 http://www.census.gov/geo/www/tiger/index.html
26
Non-probability Sampling
 Convenience: selecting cases based on their
availability
 “Typical” cases: selecting cases already
known and not considered “extreme”
 Snowball: group members identify additional
members to be included in sample
 Quota: sample is in same proportion as
population
27
The Sample
 How selected
 Sample size
 Determine the actual
individuals or “things”
included
28
Sample Size
 A larger sample does not necessarily mean
better results, but
 Too small a sample can lead to error
29
Sample Size- 3 Considerations
 Precision (sampling error)- the range in which
the “true value” is estimated to be: ±5
 Confidence Level (Central Limit Theorem)when a population is repeatedly sampled, the
average value is = to the true value, and
values in each survey will be normally
distributed: 95% confidence level
 Degree of Variability- distribution of attributes
within a population. The more homogenous
the population, the smaller the sample size.
30
Sample Size
http://edis.ifas.ufl.edu/PD006#TABLE_1
31
Sampling Customers (Example)
 Present
 Lost
 Never-gained
 Nonuser
32
Users/Uses of Electronic
Resources (More Examples)
 Home page users in general
 Users of a database
33
Questions of When
 How might time effect our study?
 How do we choose a time frame?
 What is an appropriate time frame based on
the research problem?
 Address time through:
 Cross-sectional studies
 Longitudinal studies
34
Cross-Sectional Studies
 A study based on observations representing a
single point in time.
 A “snapshot”
 Best for exploratory and descriptive studies
 U.S. Census
 Explanatory cross-sectional studies aim at
drawing causal relationships over time based
on observations made at one time.
 Issues?
35
Longitudinal Study
 Permits observations of the same
phenomena over an extended period.
 Researcher may “follow” a group over time
 Researcher may become part of a group
 Researcher may study artifacts developed
over time
36
Types of Longitudinal Studies
 Trend Studies

Type of longitudinal study that examines
changes within a population over time
 Cohort Studies

Examines a specific subpopulation (cohort) as
they change over time- often based on age.
 Panel Studies

Examines the same set of people over time
37
The Learning Organization
 http://www.lib.umd.edu/groups/learning/learni
ngorg.html
38
Group Activity
Selecting a Sample







Archive/ Special Library
Public Library
Academic Library/ School Library
Identify a research question
Define your population
Describe how you would select a sample
Could you design an experiment around this
project?
39