Session 3-Experimental &
Quasi-experimental Research
This Evening’s Agenda
• Review Course Requirements &
Upcoming Assignments
• Ethics in Research
• Review for, Take, & Correct Quiz
• Discussion & Lecture on Experimental
Research
• Activity
• Dismissal

Discussion Guides- Please submit to dropbox
folder and label
◦ 3 main points for all of the readings for that
session.
◦ No need to double space
◦ Address all of the headings (even the difficulties,
concerns, questions section).
◦ Use complete sentences
◦ These are designed to guide your discussions.
◦ Please put your questions for ME on the feedback
guide, so if they aren’t answered by your discussion
group, type them on the feedback guide!

Today NIH/CITI training modules
August 5th : Annotated Bibliography (individual assignment)

August 5th : Conceptual Framework (group assignment)

August 7th : Article Review Assignment (group assignment)

August 12th : Written Research Proposal (group assignment)

August 14th : Presentation of Research Proposal (group
assignment)

◦ Each individual reviews 3 research articles regarding their topic
◦ See Example:
http://rxsped596.pbworks.com/w/file/fetch/54804527/Example%20Anno
tated%20Bibliography.pdf
◦ Group submits short summary of literature and presents a conceptual
framework for theories that drive their proposal.

Briefly and clearly state how each research question will be
addressed.

For example, “This research question will be answered by

Briefly present the proposed theoretical and practical
implications of the findings.

e.g., “The results of this study may have implications for
comparing the end-of-year state wide test scores of
students who received the intervention and those who did
not receive the intervention.”
the use of the evidence-based Super-Duper Reading
Intervention by elementary special education teachers…”

Framed based to operationalize (clearly define so that
it can be replicated) the objectives of the proposed
research project.

Mention the IV and DV and how they will affect each
other

Framed based on methodology:
◦ Experimental/Quasi-experimental/Single-subject: “Is there
a causal of functional relationships between IV and DV”
◦ Correlational: “Is there a correlational relationship between
IV and DV”
◦ Descriptive/Qualitative: Describe a phenomenon or issue
better, “What is the prevalence of intellectual disabilities in
African-American middle school students?”


What did you learn from doing the CITI
course?
Regarding…
◦
◦
◦
◦
Research misconduct
Data Management
Responsible authorship
Collaboration in Research
◦ http://www.youtube.com/watch?v=W7sfIA1dIGQ
◦ http://www.youtube.com/watch?v=yr5cjyokVUs
◦ http://www.youtube.com/watch?v=b8McGyYAwcU
1. Respect for Persons-
◦ Required to: obtain informed consent & respect privacy
of participants
2. Beneficence-
◦ Required to: use best possible design to maximize
benefits & minimize harms, show they can perform the
procedures and handle the risks, prohibit research that
that is without a favorable risk-benefit relationship
3. Justice
◦ Required to: select participants equitably & avoid
exploitation of vulnerable populations or populations of
convenience.



http://www.rsp.pdx.edu/compliance_human.php
Portland State University (PSU) is responsible for
the rights and welfare of human subjects
involved in research sponsored or conducted by
the university. In order to meet this
responsibility, the University established the
Human Subjects Research Review Committee.
Members are charged with reviewing all research
conducted under the auspices of PSU that
involves human subjects to ensure adequate
protections are in place.
In-text Citations:
Formatting Quotations
When quoting, introduce the quotation with a signal
phrase. Make sure to include the author’s name, the year
of publication, the page number, but keep the citation
brief—do not repeat the information.
Caruth (1996) states that a traumatic response
frequently entails a “delayed, uncontrolled
repetitive appearance of hallucinations and other
intrusive phenomena” (p.11).
A traumatic response frequently entails a
“delayed, uncontrolled repetitive appearance of
hallucinations and other intrusive phenomena”
(Caruth, 1996, p.11).
In-text Citations:
A Work with Two Authors
When citing a work with two authors, use “and”
in between authors’ name in the signal phrase yet “&”
between their names in parenthesis.
According to feminist researchers Raitt and Tate
(1997), “It is no longer true to claim that women's
responses to the war have been ignored” (p. 2).
Some feminists researchers question that “women's
responses to the war have been ignored” (Raitt &
Tate, 1997, p. 2).
In-text Citations:
A Work with Three to Five authors
When citing a work with three to five authors,
identify all authors in the signal phrase or in
parenthesis.
(Harklau, Siegal, & Losey, 1999)
In subsequent citations, only use the first
author's last name followed by "et al." in the
signal phrase or in parentheses.
(Harklau et al., 1999)
In-text Citations:
A work with 6 or more authors
• When citing a work with more than 6
authors, you do NOT have to identify all
authors in the signal phrase or
parenthesis.
• You identify the first author and use et al.
after his/her name with the date of the
publication
(Carr et al., 1999) OR
signal phrase in text e.g.:
“Carr et al. (1999) conducted a meta-analysis
of behavior support practices…”
Steps in the Research/Scientific
Process





1. Identify socially
important issue
2. Review current
literature
3. Define conceptual
model
4. Define specific
hypothesis(es) and
research question(s)
5. Define dependent
variable(s)/measure






6. Identify independent
variable(s)/measures
7. Select appropriate
research design
8. Obtain consents
9. Collect data
10. Analyze data
11. Communicate
results


Written presentation
Oral presentation
Conceptual Models
(or theoretical models)




Theory that drives the research.
Guides our thinking and provides “rules”,
“principles” that guides the research and
practice.
Structure of assumptions, principles, and rules
that holds together the ideas of a broad
concept.
Outlines your research
“Logical Flaws” of FBA use in
public schools
(Scott etal.,2005)
FBA is used mainly as a reactionary approach.
opportunity is lost to utilize FBA technology to develop
interventions that address minor behaviors that usually
precede more serious problems.
FBA is restricted to set of procedures used by “experts”
The rich supply of information from people with whom the
student interacts with the most is lost.
FBA is restricted to rigorous procedures that are unrealistic for
public school settings.
Disincentive for using FBA technology.
Cynicism as to the practicality of FBA .
Proactive…Parsimonious…Practical FBA in schools
FBA conceptualized by Scott et al. as a proactive prereferral routine that uses the most parsimonious
procedures required to create an effective behavior
support plan.
Given the time & resource constraints in schools, we
must encourage schools to “work smarter” to develop
capacity to implement technology to effectively
support more students.
Use Practical FBA procedures to develop capacity
within a school to utilize FBA technology.
Practical FBA Logic Model
Individualized
Supports
5% of
Students
Secondary
Group
Supports
10-15% of
Students
School-wide Positive
Behavioral Supports
80% of Students
Behavior Specialist
responsible for 25
FBAs in school of 500
Personnel with “flexible” roles conduct
proactive Practical FBA to expand the
scope of FBA, prevent intensive problem
behaviors, & decrease reliance on
specialist.
A) You should educate readers about the topic and
provide a clear rationale as to why the study is
important and necessary based on the previous
research and writing on the topic.
B) Within your literature review you will present the
logic or conceptual framework as to why and how your
current study (topics, methods, designs) is organized
the way it is.
C) Make this section compelling. Concisely explain the
social importance of what you are studying.
e.g., Start with a powerful statement or statistic


Get together with 2-3 other people and use
your discussion guides to “guide” your
discussion.
Spend more time on the chapter reading and
the Quality Indicator Article as we will be
doing an activity on the other two articles
later!

Experimental & Quasi Experimental Research
 Research Designs & Threats to Validity


Attempt tonight to apply the principle of
teaching “less more thoroughly”….
https://www.youtube.com/watch?v=qtLnBz6l
bRQ

“Gold Standard” in Research (Moore &
McCabe, 1993; Feuer, Towne, & Shavelson,
2002; Slavin, 2002).

Rooted in postpositivist paradigm.

Seeks to make causal conclusions.

Difference between experimental design and
quasi-experimental design is the use of
random selection of participants.




Emphasis is on operationally defining the
variables (dependent & independent) and the
context of the research.
Dependent variable(s)- outcome variables (e.g.,
reading scores)
Independent variable(s)- variable that is
manipulated (the intervention or practice; e.g.,
reading curriculum)
Context- defined clearly so replication can occur
(e.g., K-3 school with 200 students, etc.)



Experimental (or treatment group)- receives
the intervention
Control group- business as usual.
For true “experimental” research, participants
are randomly assigned to each group.
◦ In order for to be considered random, every person
must have an equal chance of being in either group

Direct manipulation of an independent
variable (intervention)


Refers to whether a study is able to
scientifically answer the questions it is
intended to answer.
Extent to which your test (or study) measures
what it intends to measure.



Changes observed in the dependent variable
(outcome) are due to the effect of the
independent variable (intervention)…..& not
to some other unintended variables
(extraneous, alternative explanations)
12 threats to internal validity (noted by
Mertens, 2010)
https://www.youtube.com/watch?v=_UPUtlH
DM0A
1.
2.
3.
4.
5.
6.
History- events other than IV affected results?
Maturation- changes in participants?
Testing- participants became “test-wise”?
Instrumentation- difference between pre- and
posttests?
Statistical regression- extreme groups used?
Differential selection- groups differed in ways
other than exposure to IV?
7. Experimental mortality- drop outs of study?
8. Selection-maturation- was selection a
problem based on the characteristics of the
sample? (e.g., participants in one group may
have been older)
9. Experimental treatment diffusion- treatment
& control groups shared information?
10. Compensatory rivalry by the control group
11. Compensatory equalization of treatments –
extra resources given to control group?
12. Resentful demoralization of the control
group



External Validity= extent to which findings in
one study can be applied to another situation.
AKA: ecological validity, generalizability
10 threats posed as questions (noted by
Mertens, 2010)
1.
2.
3.
4.
5.
Were the variables, context, and treatment
described in sufficient detail?
Were multiple treatments used? Did they
interfere with each other?
Was the Hawthorne effect (special attention
as part of study affecting results) operating?
Was the treatment influenced by being novel
or disruptive?
What was the influence of the individual
experimenter?
6. Were the participants sensitized by taking a
pretest?
7. Were the participants sensitized by taking a
posttest?
8. What was the influence of the type of
measurement used for the dependent variable?
9. Was there an interaction of history and
treatment effects?
10. What was the influence of the time period that
elapsed between the treatment and
administration of the dependent variable?

Treatment fidelity- implementer of the
independent variable follows the exact
procedures specified for administering the
treatment(s).


May not be reasonable to expect participants
to improve given the “strength” of the
intervention.
Intervention may not have been tried long
enough or delivered with adequate intensity.



R= Random assignment of subjects to
conditions
X=Experimental treatment (e.g., reading
curriculum)
O= Observation of the dependent variable
(e.g., test or observation measure)
One-shot case study
X O
Threats:
History, maturation, & mortality (drop out)

Other concerns using this design?
No control group & No pretest to know if it was
the intervention that affected outcome.
Very weak design

One-Group Pretest-Posttest Design
O X O
Threats:
 History, maturation,
 What would help control for these threats?
◦ Control group- both groups taking the tests at
same time, but one not receiving the intervention
◦ But sometimes it is difficult to find a control group



Involves measurement of the dependent variable
at periodic intervals.
O O O OO X O O O OO
If behavior is stable in baseline (before
intervention), then change can be attributed to
intervention.
Controls for several threats
◦ Maturation, testing, differential selection (same persons
involved)

https://www.youtube.com/watch?v=GUq_tO2Bja
U
Pre-test-Post-test Control Group Design
R O X O
R O
O
Controls for what threats?
history, differential selection, mortality (pre-test
can show differences in drop-outs)

Posttest-only Control Group Design
RXO
R O
 Controls for what threats?
◦ Same as above, except for mortality (no pretest)
Single-Factor Multiple-Treatment Designs
X1= intervention 1
X2= intervention 2
…….
R O X1 O
R O X2 O
R O
O
Controls for threats because participants
randomly assigned to comparison groups and
pre-post-tests conducted.


If worried about pretesting affecting validity
ROXO
RO O
R XO
R
O
https://www.youtube.com/watch?v=7RRVW4iO
7gA





Similar designs to experimental designs,
except for……
Lacking….
Random assignment
Can’t make a strong causal statement
https://www.youtube.com/watch?v=_WpBxER
GNVw
How do we know if a research study involves
rigorous, systematic and objective procedures?

CEC-Division for Research

Sponsored prominent researchers to author papers to
propose


Parameters for establishing that reported research has been
conducted with high quality (quality indicators)
Criteria for determining whether a practice has been
studied sufficiently (enough high-quality research studies
conducted on its effectiveness) and shown to improve
student outcomes (effects are strong enough)
Graham, S. (2005). Criteria for evidence-based practice in
special education [special issue]. Exceptional Children, 71.
Exceptional Children (2005) volume 71(2)

Group Experimental and Quasi-Experimental
Research (Gersten, Fuchs, Comptom, Coyne, Greenwood,
& Innocenti)

Single-Subject Research (Horner, Carr, Halle, McGee,
Odom, & Wolery)

Correlational Research (Thompson, Diamond,
McWilliam, Snyder, Snyder)

Qualitative Studies (Brantlinger, Jimenez, Klingner,
Pugach, & Richardson)
Quality Indicators (QIs) for Experimental
(and Quasi-Experimental) Research

Describing Participants


Implementation of Intervention and Description of
Comparison Conditions


Clear description of intervention (and comparison conditions)
with implementation fidelity assessed
Outcome Measures


Sufficient information about participants and interventionists,
selection procedures as well as comparability across conditions
Use of multiple measures at appropriate times
Data Analysis

Analysis techniques appropriate to questions and unit of
analysis with effect size calculated
Descriptive Statistics
Who is in your data?
sample
population
Inferential Statistics
What your sample says
about the population
sample
population
Mean, Median, Mode,
Standard Deviation, Variance
Prepared by M. Hara (mhara@pdx.edu)
Tests of significance
(t-, F-Tests)

Statistical analyses to determine whether a
difference is statistically significant (probability
for result to occur by chance).

Yes or No answer

Alpha level (p=)
◦ An established probability level which serves as the
criterion to determine whether to accept or reject the
null hypothesis
◦ Common levels in education
 .01
 .05
 .10
Objectives 4.1 & 6.1
Descriptive Statistics
Who is in your data?
sample
population
Inferential Statistics
What your sample says
about the population
sample
population
Mean, Median, Mode,
Standard Deviation, Variance
Prepared by M. Hara (mhara@pdx.edu)
Tests of significance
(t-, F-Tests)

T tests- used when have two groups to compare.
◦ Independent samples t- if groups are independent
 Different people in each group
◦ Dependent samples t-: if two sets of scores are available for the same
people (e.g., pre and post-tests of same group)
 Matched groups


ANOVA (analysis of variance)- when you have more than 2
groups to compare OR more than one independent variable
(reports an F-statistic, which is basically a t-value squared)
ANCOVA (analysis of covariance)- ANOVA that allows for control
of the influence of an IV (e.g., characteristics of people) that may
vary between your groups before treatment is introduced.
◦ Post-hoc method for matching groups on variables such as age, prior
education, SES, or a measure of performance
Effect Size
• Way of quantifying the difference between two
groups.
• Not just was there an effect, but the magnitude of
the effect.
• Many ways to calculate
•
ES= [Mean of experimental group] – [Mean of control group]/Standard Deviation
•
R-squared, Cohens-D
•
Standard deviation is how well the mean summarizes the data
Visible Learning
by John Hattie
• Over 800 Meta-analyses
• https://www.youtube.com/watch?v=sng4p3
Vsu7Y



Implications for your practice- why you selected
the article
Introduction- what was the aim of the article,
what were the specific research questions
Methods- Describe:
◦
◦
◦
◦
◦
◦
◦
Participants
Setting of study
Target behaviors- dependent variables
How they measured the dependent variables
Materials used
Intervention procedures used- specify
Were the procedures replicable?

Methods used for treatment integrity
◦ This asks how well does the author measure the
implementation of the intervention

What was the design?
◦ Experimental, quasi-experimental, single-subject
(specify what type of design…multiple baseline, etc.)




Results- give quantifiable information of how the
intervention worked
Discussion- issues discussed, limitations?
Conclusion- did they answer the research
question?
Reflection- commentary, questions



Get together with a partner.
Practice completing an article review for one
of the articles you read this week.
Note the design of the study.


A detailed explanation of the assignment is
posted on the wiki
What should you be doing in your groups?
◦ At this point you should have a topic and start
coming up with your framework for your research
project (based on literature).
◦ Start to draft your conceptual framework, research
questions & identify your dependent and
independent variables
◦ You should walk away from your group time with a
list of tasks to complete.
1.
Socially Important Issue:
2.
Conceptual Model/Hypothesis:
3.
Research Question(s):
4.
Dependent Variable:
5.
Dependent Variable Measure:
6.
Independent Variable:
7.
Independent Variable Measure:
8.
Research Design: