Direct Observations of ADHD Children in the Classroom: A Review
Michael J. Kofler, Thomas M. Timko, Jr., M. D. Rapport, Ph.D.,
Kimberly Weiss, Gezim Begolli, and Scott LaFond
)
University of Central Florida
Table 1. Demographic and Methodological Variables.
INTRODUCTION
Environmental demands impact the expression of ADHD-related symptoms (Rapport, Chung, Shore, Denney, &
Isaacs, 2001). Children with ADHD are indistinguishable from their peers in certain environments (e.g., during recess
and non-academically related work periods), but exhibit marked behavioral differences when encountering many
structured situational demands (Whalen et al., 1978). Children with ADHD pay less attention during academic
instruction (Abikoff, Gittelman-Klein, & Klein, 1977), complete academic work inconsistently (Barkley, DuPaul, &
McMurray, 1990), and shift from one activity to another at home more frequently compared with peers (DuPaul &
Stoner, 1994). Situations involving novel tasks (Power, 1992; Zentall & Meyer, 1987) and/or frequent reinforcement
(Douglas & Parry, 1983; Pelham, Milich, & Walker, 1986) are associated with improved behavior and performance in
children with ADHD.
Laboratory studies employing rigorous experimental paradigms such as the Continuous Performance Task (CPT) often,
yet not always, produce performance differences between ADHD and normal control children (for a review, see
Douglas, 1988; Rapport et al., 2000). The highly controlled nature of the laboratory setting allows hypotheses to be
tested regarding the underlying processes of ADHD. However, such situational demands may evoke attentional
processes that differ from those required in the natural environment (Barkley, 1991).
Study
Abikoff et al., (1977)
Abikoff et al., (1980)
Abikoff & Gittelman, (1984)
Abikoff & Gittelman, (1985)
Abikoff et al., (2002)
Cunningham & Siegel, (1987)
DuPaul et al., (1998)
Forness & Esveldt, (1975)
Jacob et al., (1978)
Klien & Young, (1979)
Nolan & Gadow, (1997)
Roberts (1990)
Skansgaard & Burns, (1998)
Werry & Quay, (1969)
Zentall (1980)
N
Age
M/F
Class Type
Obs
Duration
Obs Interval
Obs/Day
Beh Cat
Num of
Obsvrs
Def Offtask
120
8.17
112/8
Sim
16
15
3
14
5
Whole
Num
Dx Meth Child Obs
RS
Multiple
119
8.4
114/5
Reg
16
6
3
11
4
Whole
RS
Multiple
56
8.5
54/2
Reg
16
10
3
8
7
Whole
RS
Multiple
56
8.17
54/2
Reg
16
6
3
11
7
Whole
RS
Multiple
1004
8.4
806/198
Reg
16
10
3
12
NR
Whole
SSI
Multiple
60
8.71
60/0
Sim
15
5
1
16
2
Whole
RS
26
7.7
21/5
Reg
18
15
3
4
2
Whole
SSI
Single
48
7.2
48/0
Reg
10
6
6
5
6
Whole
BP
Multiple
16
Single
9.6
14/2
Sim
30
10
10
6
NR
Partial
RS
Multiple
34
NR
34/0
Reg
90
10
2
17
2
Whole
RS
Multiple
68
8.8
62/6
Reg
30
10
3
5
3
Whole
RS
Multiple
33
8.92
33/0
Sim
15
NR
1
6
NR
Partial
SSI
24
NR
Reg
10
5
4
96
2
Whole
RS
Single
21
8.92
21/0
Sim
15
20
3
17
6
Partial
PR
Single
62
7.3
62/0
Reg
20
NR
5
6
3
Partial
RS
Multiple
8/16
Single
Notes. N = Total number of subjects used in the study; M = Males; F = Females; Class Type = Classroom type (Sim = Simulated classroom setting; Reg = Regular
classroom setting); Obs Duration = Length of observation in minutes; Obs Interval = Observation length before recording; Obs/Day = Number of days of observation;
METHODS
Literature Search. The current study reviews empirical research of children’s attention in classroom settings to
examine the magnitude of effects of differing methodological techniques in studies distinguishing children with ADHD
from normal control children. A three-tier literature search was conducted using PSYCInfo, ERIC, and Social Science
Citation Index. After the initial search, studies cited by articles meeting inclusion criteria were examined (backward
search), and a forward search was conducted using the Social Science Citation Index to find studies citing those
meeting inclusion criteria. These procedures generated 140 studies published since 1969. Search terms included all
permutations of the ADHD diagnostic label (ADHD, ADD, attention deficit, hyperactivity, hyperkinesis, minimal
brain dysfunction/damage, MBD), classroom/direct observation, attention, and on/off-task. Search limits included age
range (child; 6-12 years) and peer reviewed journals. The following served as inclusion criteria for the review: (a) an
independent direct observation of children exhibiting inattentive, hyperactive, and/or impulsive behavior in an
elementary classroom setting, (b) between 6 and 12 years of age, (c) on- or off-task data reported, (d) low average or
higher estimated intelligence, and (e) a normal control comparison group. Exclusion criteria included: (a) direct
observations completed only by teachers or peers (b) borderline or below estimated intellectual functioning, (c)
comorbidity with other mental health disorders beyond LD, ODD, or CD, and (d) single case design. Eighteen studies
published from 1969 to 2002 met these search criteria. Three of these articles did not include necessary data to enable
effect size calculation and were removed from the analysis.
Meta-analytic Procedures. Baseline data was examined from studies employing treatment conditions. Hedges’ g
(1982) effect sizes were calculated to estimate the magnitude of differences in observed attention between children
with ADHD and control children (Note: Hedge’s g correction accounts for error due to small sample size). Means,
standard deviations, and sample sizes were used to compute Hedges g for 12 studies (80%), and an additional 3 studies
provided sample size and F or p values needed for estimation. Two effect sizes were calculated for Abikoff et al.
(2002) because the researchers reported separate descriptive statistics for males and females (N = 806 and 198,
respectively) in lieu of combined information. The Comprehensive Meta-Analysis Version 2 software package was
used to calculate effect sizes. Mean effect sizes were weighted by sample size for subsequent analyses. Ensuing
analyses were performed using SPSS for Windows 11.5. Standard difference scores were calculated to examine the
difference in frequency of off-task behavior between ADHD and Normal Controls by subtracting normal control offtask rates from ADHD off-task rates, dividing by the ADHD off-task rate, and multiplying by 100. These values
standardize the difference between groups as a percentage of ADHD off-task behavior (Losier, McGrath, & Klein,
1996).
Beh Cat = Number of behavioral categories; Num of Obsvrs = Number of observers; Def Off-task = Proportion of interval needed to be coded off task; Dx Meth = Diagnostic
method (RS = Rating scale(s); SSI = Semi-structured interview; BP = Referred for behavioral problems PR = Psychiatric referral); Num Child Obs = Number of children
observed per session
.5 0
A r e a b e t w e e n m e a n s = .3 7 7 0
N or m a l C on tro ls
.4 0
Methodology. Children were considered ADHD/ADDH (33% of studies), hyperactive/ hyperkinetic (53%), or behavioral
problem children (13%) in the reviewed studies by means of four general methods. These methods included formal semistructured diagnostic interviews, behavioral rating scales, referral for behavioral problems, and non-diagnostic psychiatric
interview. Rating scales were used in a majority of studies (N=10; 86.7%) to select sample participants. These studies
used at least one teacher rating scale, and 4 of the 10 studies (40%) also used at least one parent rating scale. Three
studies (20%) combined structured clinical interviews with rating scale data. The two studies not employing rating scales
selected subjects based upon referral for behavior problems by psychiatrist (i.e., Werry & Quay, 1969) or parental report
(i.e., Forness & Esveldt, 1975). Children were observed in their regular classroom environments (73% of studies) or
simulated classrooms (27%) by trained observers for periods ranging from 10 to 90 minutes, with a median of 16 minutes.
Within each session, observation intervals lasted between 5 to 20 seconds before recording, with a median of 10 seconds
across studies. Total sessions ranged from 1 to 10 days of observations, with a median of 3 days across studies.
Behavioral ratings taken by trained observers ranged from simple recording schemes involving as few as 4 categories to
as many as 96 (i.e., Skansgaard & Burns, 1998) separate categories of behavior. Many of the observers (56.3% of studies)
were simultaneously observing 10 or more behaviors. The mean number of behaviors observed was 15.4, with a median
of 11 observations per session across studies.
Off-task Comparisons. Children with ADHD were off-task 26.1% across studies on average (range = 13.1% to 47.0%),
compared with 12.0% (range = 2.1% to 26.6%) for control children. All but one study reported significant between-group
differences. The exception, which approached significance at p = .052, utilized a dyad design in which 30 ADHD
children were paired with a normal control child for the duration of the study (Cunningham & Siegel, 1987). The ADHDNormal child interaction may have contributed to the failure to find significant difference in attention between the groups.
F ig u r e 1 . S ta n d a r d iz e d N o r m a l a n d A D H D O ff-ta s k R a te s
A D H D C h ild r en
RESULTS
Frequency
Ecological validity concerns necessitate the investigation of more naturalistic procedures for characterizing behavior,
such as behavioral rating scales and direct observations by independent researchers. Ratings scales are cost- and timeefficient measures capable of encapsulating overall behavioral functioning. They provide a quantitative metric of the
frequency and severity of behavior, yet remain dependent upon subjective reports and are vulnerable to response bias,
misinterpretation of questions, and over/underestimation of behavior due to intensity/immediacy effects (Kazdin,
1997). In contrast, direct observations can provide more objective ratings by utilizing independent, well-trained
observers and specific, operationalized behavioral definitions (Kazdin, 1997). Direct observations feature limitations
surrounding their time- and labor-intensive nature, as well as a lack of agreement on behavioral definitions and
standardized observation techniques. The present study examines the relationship between differences in methodology
and resulting observed magnitude of differences between ADHD and control children when observed in the classroom.
Relationships between ten methodological variables and obtained effect sizes will be examined. Standard difference
scores and the standard error of the effect sizes will also be reported. None of the studies reviewed reported changes in
attention over time.
STUDY CHARACTERISTICS
Sample. Sample size in the fifteen studies ranged from 16 to 806 children, with a median of 56 subjects. Mean age (8.35
years, range of means = 7.2 to 9.6 years) was reported in 13 of the 15 studies. Gender was reported in all studies, with
males representing 86% (range = 33% to 100%) of all subjects.
.3 0
.2 0
.1 0
- 1 .2
-4
-3
Table 3. Stem and Leaf Display of 16 Hedges g Effect Sizes
Stem
Leaf
2.2 5
2.1 8
2
1.9
1.8 1
1.7
1.6 4
1.5 1
1.4 1, 9
1.3 4, 5, 6
1.2
1.1
1
0.9
0.8 0, 9
0.7
0.6 2
0.5 1, 7
0.4 6
-2
-1
1
2
0
Z -s c o r e s s ta n d a r d iz e d to A D H D r a te s
3
Table 2. Mean Off-task Rates; Standard Difference Scores; and Effect Sizes in ADHD and Normal Children
Study – N
Abikoff et al., (1977) – 120
Abikoff et al., (1980) – 119
Abikoff & Gittelman, (1984) – 56
Abikoff & Gittelman, (1985) – 56
Abikoff et al., (2002) Males – 806
Abikoff et al., (2002) Females – 198
Cunningham & Siegel, (1987) – 60
DuPaul et al., (1998) – 26
Forness & Esveldt, (1975) – 48
Jacob et al., (1978) – 16
Klien & Young, (1979) – 34
Nolan & Gadow, (1997) – 68
Roberts, (1990) – 33
Skansgaard & Burns, (1998) – 24
Werry & Quay, (1969) – 21
Zentall, (1980) – 62
Column (SD) =
Note:
1
Standard deviation of effect sizes
ADHD % Offtask (SD)
13.1(10.0)
15.1(23.4)
17.4(12.3)
15.7(10.4)
18.5(20.2)
14.7(19.7)
33.0(NR)
33.0(19.2)
47.0(16.5)
15.8(NR)
39.8(9.0)
30.5(15.9)
39.5(18.8)
23.8(10.3)
46.3(12.8)
15.0(NR)
26.12 (12.7)
Control % Offtask (SD)
2.1(2.6)
4.1(7.8)
3.5(6.6)
2.5(4.6)
5.5(11.2)
5.7(10.9)
26.4(NR)
9.5(11.9)
34(12.4)
10.5(NR)
26.6(5.0)
13.3(8.3)
12.9(20.9)
4.8(6.1)
23(15.4)
7.1(NR)
11.97 (10.06)
Std. Diff.
Scores (%)
84
73
80
84
70
61
20
71
28
33
33
56
67
80
50
52
59 (21)
Effect Sizes
(Std. Error)
1.51(0.21)
0.62(0.19)
1.41(0.30)
1.64(0.31)
0.80(0.07)
0.57(0.14)
0.51(0.46)
1.35(0.30)
0.89(0.41)
1.49(0.27)
1.81(0.40)
1.36(0.63)
1.34(0.55)
2.25(0.52)
2.18(0.57)
0.46(0.26)
1.26 (0.57)1
Effect sizes (ES), standard error of effect size (SEES), and standard difference scores (SDS) are shown in Table 2. Mean
weighted effect size for the 15 studies was 0.90 (95% CI: 0.80-1.00) with a range of 0.45 to 2.17. This result corresponds
to a large effect based on Cohen’s criterion (Cohen, 1977). A stem and leaf representation of the distribution of effect
sizes is shown in Table 3. Analysis of the 10 demographic and methodological variables revealed significant differences
in mean effect sizes when studies were grouped by sample size (t = 4.33, df = 14, p = .0007) and duration of observation
in minutes, F(2, 13)= 9.34, p = .0031. Post-hoc analyses of the duration of observation revealed that observations lasting
between 11 and 20 minutes (M = 0.844, SE=0.053) were associated with significantly lower effect sizes than observations
lasting both longer, (p = .0008; M =1.464, SE=0.204) and shorter (p = .0361; M =1.211, SE=0.256) time frames. No
differences were found between the longer and shorter studies. Sample size was inversely related to effect size, with
higher sample size associated with lower effect sizes. This result may be partially explained by a significant positive
correlation (r= .50, p= .049) between effect size and the standard error of effect size -- findings of very large off-task
differences between ADHD and normal control children may be partially due to error variance, suggesting that true
differences may be somewhat more modest than some studies suggest. Effect sizes did not differ systematically by
number of observation days, categories observed, discrete within-observation duration before recording (in seconds),
classroom type (normal vs. simulated), number of raters, number of children observed simultaneously, definition of offtask behavior, or diagnostic criteria. The mean standard difference score, representing the off-task rate of normal children
compared with their peers with ADHD, is 59 (95% CI = 49 to 69). This finding indicates that normal control children are
off-task 59% less often then children with ADHD in classroom settings on average. Converting the off-task rates of
control children to z-scores based upon the standardized ADHD-distribution allows further comparison to be made
between the two groups. Figure 1 shows the area (.3770) under the normal curve between the mean ADHD and normal
control off-task rates.
CONCLUSIONS
Direct observations of children with ADHD in classroom settings indicate significant deficiencies in their ability to pay
attention, and these differences are apparent regardless of most methodological differences across studies. Standard
difference scores across studies reveal that normal children are off-task an average of 59% less often than their ADHD
peers. Stated differently, the average ADHD child is approximately 40% more off-task than normal controls (see zdistribution shown in Figure 1). None of the reviewed studies, however, examined the effect of time or attention by time
interactions in ADHD and normal control children. The failure to examine time effects in classroom studies of attention
severely limits our understanding concerning possible underlying mechanisms and processes responsible for the
attentional difficulties associated with ADHD (e.g., whether they reflect a more general deficit in attentional mechanisms
or particular deficiencies in the ability to maintain attentional focus over time).