T r a f f i c
i n j u r y
r e s e a r c h
F o u n d a t i o n
white paper on adept drivers’
teensmart program:
Sa f e t y Performan c e
the traffic injury research foundation
The mission of the Traffic Injury Research Foundation (TIRF) is to reduce traffic-related deaths and injuries. TIRF
is an independent, charitable road safety institute. Since its inception in 1964, TIRF has become internationally
recognized for its accomplishments in identifying the causes of road crashes and developing program and policies
to address them effectively.
Traffic Injury Research Foundation
171 Nepean St. Suite 200
Ottawa, ON K2P 0B4
Ph: (613)238-5235
Fax: (613)238-5292
Email: tirf@tirf.ca
www.tirf.ca
August 2012
WHITE PAPER ON ADEPT DRIVERS’
teenSMART PROGRAM:
Safety Performance
Prepared by:
Dan Mayhew, Senior Vice President
Robyn Robertson, President and CEO
Ward Vanlaar, Vice-President of Research
Traffic Injury Research Foundation (TIRF)
August 2012
Traffic Injury Research Foundation
about the Lead author
Dan Mayhew, MA
Dan Mayhew is Senior Vice President of the Traffic Injury Research Foundation (TIRF), a
charitable independent road safety institute. TIRF is a world leader in research, safety
programs, and policy development.
Since joining TIRF in 1980, Dan has managed numerous research projects for the
Foundation. His extensive work in the areas of young and novice drivers, graduated driver
licensing, and driver education is very well known in the field of road safety in Canada and
elsewhere.
Dan’s research interests include: driver licensing improvement systems, motorcycle safety,
young driver accident prevention, senior driver safety, motor vehicle accident data systems,
driver competency and experience, commercial vehicle driver licensing, driver education
and training, enforcement, driving simulation, fuel efficient driving, and alcohol, drugs and
traffic safety.
Disclosure
Dan Mayhew participated on the Advisory Team during the development and initial
evaluation of teenSMART over 12 years ago. Since then, Mr Mayhew has not had any
involvement with teenSMART, including its’ evaluation, promotion or marketing. He is,
however, currently chair of an Advisory Council of experts providing guidance for the
development of Lifelong Driver, a computer-based training program for older drivers, and
a product of ADEPT Driver. Robyn Robertson and Dr. Ward Vanlaar were not employed at
TIRF when teenSMART was developed and have had no involvement in any way with this
program.
i
Traffic Injury Research Foundation
tabLe of contents
About the Lead Author
i
Executive Summary
v
Introduction
1
teenSMART Description
3
teenSMART Development and Early Research
5
teenSMART and Insurance Discounts
9
teenSMART and Collision Reduction
13
Discussion
17
References
19
Appendix A: Advisory Team Members
21
Appendix B: A New Approach to Reducing Collisions Among Young Drivers
25
Appendix C: Developmental Evaluation of teenSMART
97
Appendix D: Experiences of Three Insurance Companies with teenSMART
141
Appendix E: Safety Effectiveness of teenSMART
151
iii
Traffic Injury Research Foundation
executive summary
The Traffic Injury Research Foundation has conducted an objective and independent review
of teenSMART, a product of ADEPT Driver. The purpose of this review is to assess the
veracity and strength of the available evidence that this training program reduces teen
driver crashes, which is a stated objective and claim of the program. This critical review has
established the following.
>
teenSMART is a computer-based training program consisting of three core elements:
computer-based driving tutorials (CBTs); parent-teen materials and activities; and a
certification test to pass the program.
>
teenSMART has face validity and content (logical) validity as a collision reduction
program based on expert knowledge, judgment and consensus.
>
teenSMART has been shown to have criterion-related validity (meeting its’ safety
objectives) in terms of statistically significant improvements in: knowledge, drivingrelated skills, and on-road driving performance. These are encouraging results from
the developmental evaluation of teenSMART. The replication of these findings with
a larger sampe size is needed to further establish the criterion-related validity of
teenSMART.
>
teenSMART has been shown to have criterion-related validity based on the indirect,
antecdotal evidence and the practices of six insurance companies that offer
an insurance discount for teenSMART and Departments of Insurances in most
states that approve the rate filings by insurance companies for the discount. This
suggests that these insurance companies have solid evidence convincing them that
teenSMART reduces their claims experiences – i.e., teenSMART graduates have
statistically lower collision rates. And in this regard, three insurance companies –
Allstate, AAA Insurance, and Liberty Mutual – have recently gone on public record
to espouse the safety benefits of teenSMART.
>
teenSMART may also have criterion-related validity based on the results of several
studies showing that teen drivers exposed to teenSMART have statistically significant
lower collision rates than those not exposed to teenSMART. However, these results
are suggestive but not conclusive because other factors related to self-selection of
teen drivers who take teenSMART – e.g., greater safety consciousness -- may have
also contributed to, or account for, their lower collision rates.
This critical review supports the claim that teen drivers who take teenSMART have lower
collision rates than those who do not. Indeed, teenSMART has an impressive track record
of review and assessment by insurance companies over the past 12 years that attest to its
safety benefits. And, to our knowledge, it is unlikely that other driver training programs
v
Traffic Injury Research Foundation
have been so rigorously validated and have undergone such intense and ongoing scrutiny
by insurance companies.
The reasons teen drivers exposed to teenSMART have statistically significant lower
collision rates, however, are less clear and could be the result of teenSMART training or
self-selection, for example, by more safety conscious teen drivers taking teenSMART,
or a combination of these factors. Given that teenSMART has been shown in the
developmental evaluation to have some criterion-related validity – improved knowledge,
driving-related skills, and especially, on-road driving performance – it seems reasonable to
conclude that some combination of teenSMART training and other person-centered factors
likely contributed to the lower collision rates of teenSMART graduates. Improvements in
intermediate, criterion-related measures, such as on-road driving performance, through
teenSMART training should logically translate into fewer teen driver crashes.
Regardless of the relative contribution of teenSMART training to reducing teen driver
crashes, the fact that insurance companies offer discounts and make their own public
statements, as well as support those of ADEPT Driver, about teenSMART significantly
reducing collisions is remarkably telling about the safety performance of teenSMART within
an insurance environment.
vi
Traffic Injury Research Foundation
introduction
The Traffic Injury Research Foundation (TIRF) has conducted an objective and independent
review of teenSMART, a computer-based training program, developed by ADEPT Driver
(adeptdriver.com). To facilitate this review, Dr. Richard Harkness, President and CEO
of ADEPT Driver, provided TIRF information on the development, operation, and safety
effectiveness of the teenSMART program as well as information on the process by which
automobile insurance companies offer an insurance discount for teenSMART.
This paper critically reviews the available documentation on teenSMART to assess the
veracity and strength of the evidence that this training program reduces teen driver crashes,
which is a stated objective and claim of the program.
1
Traffic Injury Research Foundation
teensmart description
The teenSMART program consists of a Certification license, program software DVD,
Student Workbook, Parent Guide and Parent-Teen Activity Video DVD. There are three
core elements to this program. These are done at home and are self-paced so the teen can
progress at their own speed. The three elements are:
1. Computer-based driving tutorials (CBTs) are completed on the home computer.
These tutorials provide hands-on, in-depth practice. Teens are asked to demonstrate
specific driving skills and are given immediate, interactive feedback. The computer
software also provides a step-by-step roadmap to guide the teen through the
program.
2. Parent-Teen Activities include informational videos, workbook exercises in the
Student Workbook and Parent Guide, and in-car driving sessions conducted under
parental supervision. During the driving sessions, teens apply what they have
learned during the CBTs to real-world driving situations.
3. A Certification Test completes the program. It comprises multiple choice knowledge
questions and driving simulations similar to the training exercises, for example, on
hazard detection. Teens receive an ADEPT Driver certification once they pass the
test and upload test results to the online ADEPT Driver software tool.
It takes an estimated ten hours to complete the teenSMART program. Approximately
half of the time is allotted to Parent-Teen Activities, which include in-car driving exercises.
Computer activities and the at-home Certification Test take up the remaining time.
.
3
Traffic Injury Research Foundation
teensmart deveLopment and earLy
research
ADEPT Driver designed the teenSMART program in the mid-to late-1990’s over a four-year
period with the assistance of an Advisory Team of behavioral scientists, including experts
in traffic safety, adolescent development, risk perception/management, and instructional
design and technology. A list of experts on the Advisory Team is provided in Appendix A.
teenSMART was designed as a “breakthrough” product, distinct from traditional driver
education programs, with the objectives to change teen drivers’ knowledge, skills and
behaviours, and thereby, to reduce teen driver crashes.
A review of reports on the development and early evaluation of teenSMART as well as the
program itself establishes that it has both content (logical) validity, face validity, and some
aspects of criterion-related validity – e.g., achieving its objectives of improving knowledge
and driving-related skills. These reports are provided in Appendix B: A New Approach
to Reducing Collisions among Young Drivers and Appendix C: Developmental Evaluation
of teenSMART. The teenSMART program can be obtained from ADEPT Driver’s website:
adeptdriver.com.
teenSMART has content validity because it addresses the key factors that cause most teen
driver crashes – see report in Appendix B. These include skill deficiencies in: visual search,
speed adjustment, hazard detection, space management, risk perception, and lifestyle
issues (e.g., susceptibility to peer pressure). These key factors were identified through
a comprehensive literature and program review by TIRF (Mayhew and Simpson 1990;
Mayhew and Simpson, 1994; TIRF 1995) and a consensus amongst experts on the Advisory
Team that these factors are critical in teen driver crashes and needed to be influenced to
effectively address the problem. Importantly, these key crash factors are as valid today as
they were more than a decade ago when they were first identified by the panel of experts
during the development of the teenSMART program (Compton and Ellison-Potter, 2008;
Curry et. el., 2011; Mayhew et. al., 2006; NRC et. al., 2007; Shope and Bingham, 2008).
teenSMART also has face validity because it uses realistic driving simulation and computerbased driving skill exercises that on “face value” are good representations of the key risk
factors, driving skills, and driving situations relevant to teen drivers – e.g., the “Where is
it” exercises are a good representation of visual search; the “Hazard clicking” exercises
are a good representation of hazard detection; the “Risk-O-Meter” exercises are a good
5
Traffic Injury Research Foundation
representation of risk perception. More specifically, teenSMART training, for example, in
hazard detection, appears to be a good representation or measure of hazard detection
on “the face of it” and more generally, on “its face”, teenSMART is a good teen driver
crash reduction program. This was the conclusion of experts on the Advisory Team who
unanimously approved of the overall approach, psychometrics and instructional technology
to be used in teenSMART as well as the prototype that was developed for beta testing.
Given that teenSMART has both content and face validity based on expert knowledge,
judgement and consensus, it could also be concluded that teenSMART has some elements
of construct validity – i.e., translating a construct or concept into a functioning and
operating reality (operationalization). However, establishing construct validity also requires
assessing the performance of teenSMART against some external criterion related to the
stated objectives of this program – i.e., to change teen drivers’ knowledge, skills and
behaviours, and thereby, to reduce teen driver crashes (Cozby, 2007; Trochim, 2006).
The influence of the teenSMART training program on student outcomes – knowledge,
skills, behaviors – was assessed in a developmental evaluation conducted by scientists
from Carnegie Mellon University and TIRF – see the report in Appendix C. Teens’
knowledge, skills and driving behaviors were measured before and after taking
teenSMART. Approximately 150 teens age 16 and 17 participated in various aspects of the
developmental evaluation. How these teens were recruited and more detail on the study
method are provided in the report in Appendix C.
The teens were assessed using multiple choice knowledge questions, computer-based skills
tests involving a variety of scenarios showing videos of the roadway from the driver’s point
of view (POV), and a real-world on-road driving test over a predetermined route during
which performance was continuously videotaped. Subjects were randomly assigned to one
of four groups to control for testing effects, and double-blind evaluations of video clips
were used to determine pre- and post-driving behavior.
teenSMART evaluation findings indicated statistically significant improvement in:
>
knowledge in all of the domains taught in the teenSMART program – hazard
detection, space management, speed adjustment, and visual search;
>
driving-related skills measured by three of the five point-of-view (POV) exercises –
hazard detection, risk assessment and visual search; and,
>
on-road driving performance related to visual search, speed management and speed
control.
6
Traffic Injury Research Foundation
This well-designed developmental evaluation provided initial encouraging findings that
teenSMART improves knowledge, skills and driving performance and suggests that the
program has some criterion-related validity, in at least meeting its objectives of improving
student outcomes. In this regard, the authors of the developmental evaluation concluded
that these encouraging results warranted replication with a larger sample.
To our knowledge, a follow-up evaluation to confirm these initial findings with a larger
sample to determine the effectiveness of teenSMART in improving knowledge and skills
related to safe driving has not been conducted. However, there have been several recent
developments and studies that relate to the criterion-related validity of teenSMART and
its effectiveness in reducing collisions – see the following two sections for a review and
discussion of this evidence.
7
Traffic Injury Research Foundation
teensmart and insurance
discounts
Insurance rates and discounts are generally based on how likely someone is to get into
a crash. Insurance actuaries and statisticians use established and sophisticated formulas
based on years of data to calculate how likely someone will cause or be involved in a
collision, and discounts are offered to those who belong to a defined group that pose a
lower risk of collision – i.e., statistically less likely to be in a collision. Lower risk groups
are defined, for example, in terms of good grades (e.g. maintain a B average), graduation
from approved driver education, and low-mileage car use. In the case of driver education,
this could include premium discounts to teen drivers who graduate from traditional driving
schools or obtain certificates from non-traditional driving programs. In some cases,
companies offer their own driver education programs, including on-line defensive driving
programs and “day-at-the track” programs (e.g., skid control, slippery road, training in a
parking lot or an off-road track). State Farm, for example, offers the Steer Clear Safe Driver
Discount Program (see: http://www.statefarm.com/insurance/auto_insurance/steerClear.
asp). It includes 20 practice drives that young adults take with their parents, and those aged
16 to 24 who complete the free program qualify for a teen driver discount on car insurance
of up to 15%. AAA offers in some states an insurance discount for its’ Teaching Your Teen
to Drive program which comes with an illustrated in-car handbook and a 56-minute video
that instructors may incorporate into any driver education course (see: http://www.aaa.com/
Automotive/checkpoints/TYTTDflyer08b.pdf).
Insurance discounts for teenSMART are offered by six independent insurance companies:
AAA Insurance Exchange, Allstate, Liberty Mutual, American Commerce, Commerce
Insurance and Mapfre Insurance. The discounts, which can be as high as 35%, are available
in 44 states from one or more of these insurance companies. In the case of AAA Insurance
Exchange, the program and auto insurance discounts are being offered in 24 states. The
most common discount offered by AAA Insurance Exchange for teenSMART is 20%, which
compares to 5% or less, or no discount, for other driver training programs – e.g., AAA’s
Teaching Your Teen to Drive program in California.
Appendix D includes information by state on the rate filings for discounts with Departments
of Insurance, nationwide rollouts, and evaluations of teenSMART for AAA Insurance
Exchange, Allstate and Liberty Mutual, three of the larger insurance companies offering
9
Traffic Injury Research Foundation
discounts for teenSMART. This Appendix also includes a brief report on the experiences of
these three insurance companies with the teenSMART program. This report describes:
>
the process of evaluation used by the Insurance Industry;
>
department of Insurance filings for a teenSMART discount;
>
pilot programs;
>
pilot Program Evaluation Time Frames;
>
nationwide Roll Out History.
The information provided in Appendix D was prepared by Dr Richard Harkness, President
and CEO of ADEPT Driver.
A review of these reports establishes that teenSMART has undergone rigorous initial
and ongoing annual evaluations by independent and competitive insurance companies
to determine whether it reduces teen driver crashes. The process is laborious and timeconsuming with the safety performance of teenSMART from an insurance discount
perspective having been under review and evaluation for over 12 years. The process each
insurance company offering an insurance discount for teenSMART undertakes includes:
>
initial internal program review from a variety of perspectives in different
departments;
>
pilot studies offering the program in a few states to examine outcome performance
measures (e.g., collision frequencies) in large samples (about 5,000 teen drivers);
>
roll-out of the program nationwide based on favorable program performance;
>
state-by-state rate filings with state Departments of Insurance for program insurance
discounts;
>
annual, sometimes semi-annual, evaluations of the performance of the program,
which could result in the discount being pulled and/or the program dropped.
This review and evaluation process of teenSMART appears to have produced compelling,
and impressive evidence that teen drivers trained and certified by teenSMART have lower
collision claims experiences than teen drivers not exposed to the teenSMART program.
Logically, and from a business and financial standpoint, these insurance companies would
not continue to offer a premium discount and the Department of Insurances would not
approve one, in the absence of solid evidence convincing them that teenSMART reduces
claims experiences – i.e., teenSMART graduates have statistically lower collision rates.
10
Traffic Injury Research Foundation
As further recent evidence of the insurance sectors support for the crash reduction benefits
of teenSMART, California State Insurance Commissioner Dave Jones joined Dr. Richard
Harkness, the CEO of ADEPT Driver and maker of teenSMART, representatives from
insurance companies and others at May and June 2012 news conferences in California to
encourage a “safe summer for teen drivers”. According to the San Diego press release,
Commissioner Jones said “…it is important for parents, teachers – all of us – to regularly
take the time and educate our kids about becoming safe drivers and learning crash
avoidance techniques’”. The press release also mentions that teenSMART was “highlighted
as a resource that teens can use to improve their driving skills and avoid danger.” And,
it included several quotes about teenSMART from Allstate, AAA Insurance, and Liberty
Mutual:
>
“Allstate supports teen driver safety through research. The teenSMART program was
developed over 12 years ago with funding and support from Allstate. teenSMART
is a science based program designed to reduce teen car crashes. We have observed
the performance of teenSMART for many years and see that teens that complete
the program have fewer collisions and injuries. That is why we are proud to offer
a significant discount for those that complete the teenSMART program here in
California. We want our teens to be safe on the road and know they are in good
hands.” (Jeffery L. Thomas, territorial product manager for Allstate Insurance)
>
“AAA Insurance has been an advocate for teen driver safety for decades. We offer
the teenSMART discount because we want teen drivers to have the best tools
available to remain safe on the road. We find that teens that complete the program
have fewer crashes with less severe outcomes. That is why we recommend and
provide up to a 20% discount for teenSMART in Northern California.” (Matthew
Hardin, regional vice president for AAA Northern California, Nevada and Utah
Insurance Exchange)
>
“Liberty Mutual is a leader in driver safety research. We have a dedicated team
that focuses on young drivers and how to keep them safe. teenSMART is a key
component in our mission to keep young drivers safe on the road and out of the
hospital. We pilot tested the performance of teenSMART several years ago in
California and observed significant crash reduction with less severe crashes among
the teens that completed the program. That is why we offer a 20% discount for
those that complete the teenSMART program in California. Liberty Mutual is rolling
out teenSMART nationwide. Teen drivers are very important to Liberty Mutual and
learning crash avoidance techniques now will give our teen drivers skills they can
use for a lifetime.” (Kathleen Bissell, assistant vice president and senior regional
director of public affairs for Liberty Mutual Insurance)
The convergence and weight of evidence based on the experiences of independent
insurance companies with teenSMART in different states and over many years is certainly
11
Traffic Injury Research Foundation
convincing. And, it is bolstered significantly by the fact that these insurance companies
have recently gone on public record to espouse the safety benefits of teenSMART. As
noted by DR Richard Harkness, however, the reports that he has prepared on insurance
companies experiences with, and insurance discounts for, teenSMART provide antecdotal,
and not statistical, evidence that teenSMART reduces teen driver collisions. We concur with
this observation and recognize that Insurance companies use these data to compete in the
marketplace and the data are considered proprietary. However, some statistical evidence on
the safety effectiveness of teenSMART, including from insurance companies, is available and
examined in the next section.
12
Traffic Injury Research Foundation
teensmart and coLLision
reduction
ADEPT Driver provided TIRF with four summary reports on studies that have evaluated the
safety effectiveness of teenSMART to determine if the program achieves its objectives in
reducing teen driver collisions and claims. One of these studies was conducted during the
development and beta testing of teenSMART using police-reported collision data and the
other three are more recent studies conducted by ADEPT Driver using insurance collision
claims data. The reports on these studies are brief and included in Appendix E.
The first impact evaluation was conducted when teenSMART was beta tested in Elk Grove,
California during the summer of 1998. The study compared 145 teen drivers aged 1619 who completed the teenSMART 12 hour training course and passed the teenSMART
certification test with 135 teen drivers aged 16-19 who did not take teenSMART training.
The two groups were matched for school district, age, gender, and driving experience.
Both groups had taken driver’s training and had passed the states’ driver license exam. An
examination of DMV driving records one year after teen drivers completed teenSMART
showed a 30% lower collision rate for teenSMART teens one year after training – rates of
10.3 and 14.8 per 100 drivers, respectively. Although not provided in the study report,
however, a Chi-square test by TIRF research staff showed that this difference was not
statistically significant – p=0.258.
In this report, the collision rates of the teenSMART group were also compared to the overall
collision rate for teen drivers aged 16-19 in the United States during 1998 taken from
statistics included in the National Safety Council (NSC) “Accident Facts”. In the case of
this comparison, the teenSMART group had a collision rate 71% lower than the national
average – rates of 10.3 and 36 – and this difference was statistically significant (p < .001).
The reasons why the collision rate for 16-19 year old drivers nationally is so high compared
to the collision rates of the teenSMART group as well as the matched control group in this
study are not known but likely related to factors other than exposure to teenSMART – e.g.,
different state driver license systems, minimum driver licensing ages, and procedures for
counting collisions and licensed drivers than in California; different road infrastructure,
urban/rural mix, seasonal and weather patterns (e.g., severe winter conditions) than in Elk
Grove, California. For these myriad of reasons, the per-driver collision rates of 16-19 year
13
Traffic Injury Research Foundation
olds will be below or above the national average in the various states on which the average
is calculated.
The other three studies conducted by ADEPT Driver were based on an analysis of insurance
claims data from a nationwide sample of 15 states. In all three studies, the frequency of
collision claims were standardized by annualized exposure – a single annualized exposure
consists of 12 months of driving exposure – for two groups of drivers: the first group
comprised teens who purchased teenSMART and recorded a passing certification test score;
the second group comprised teens who did not purchase or complete the teenSMART
training. Each study report mentions that the non-teenSMART group came from the same
states and that the gender and age compositions by state within the non-teenSMART group
“proved to be a very close match” to the teenSMART group. However, no data on gender
and age are provided by state or overall for the two comparison groups in the study reports.
All three studies used large samples of teen drivers, especially in the non-teenSMART group.
These three studies differ in that each of them report results for different age groups and
years. The results and sample sizes are provided below.
Number
Collision Rate
16-18 year olds in 2002
teenSMART
non-teenSMART
Difference
2,235
184,886
11.7
15.9
-26.4%
16-17 year olds in 2000-2003
teenSMART
non-teenSMART
7,172
255,589
10.1
14.4
-29.8%
18 year olds in 2000-2003
teenSMART
non-teenSMART
3,511
160,290
9.3
12.2
-23.8%
All three studies report statistically significant lower collision rates for the teenSMART group
– lower collision rates of 26.4% for 16-18 year old teen drivers (p > .000); 29.8% for those
aged 16-17 (p > .000); and 23.8% for 18 year old teen drivers (p > .000).
The above results provide strong direct, statistical evidence that teen drivers who take
teenSMART have lower collision rates than teen drivers who do not. However, these results
do not conclusively establish that teenSMART reduces teen driver crashes.
14
Traffic Injury Research Foundation
These analysis all adopt a non-randomized, matched control design which is an acceptable
design but falls short of an experimental randomized design, which is often considered
the “gold” standard in evaluations of safety programs, such as driver education (Campbell
and Stanley 1966; Clinton and Lonero, 2006; Pedhazur and Schmelkin, 1991; Lonero and
Mayhew, 2010). The primary issue with these studies, which is an issue that has plagued
evaluations of driver education programs in the past, is that they do not adequately
control for volunteer bias and self-selection (Mayhew and Simpson 1997; Mayhew et.,
al., 1998; Mayhew et. al., 2002; Williams et al., 2009). Teen drivers taking teenSMART
are self-selected and may differ from non-teenSMART teen drivers on person-centered
characteristics beyond age, gender and geographic location which are typically the only
variables being matched. These volunteers may be more safety conscious than those
choosing not to take teenSMART and have a lower crash rate for these reasons and not
because of the training in teenSMART. In fact, this is the actuarial basis for some insurance
companies offering discounts to traditional, or their own, driver education programs
(Williams et. al., 2009). In this regard, similar to being female or being middle-aged or
having good grades or being a low mileage driver, teenSMART may provide the insurance
company with an indicator or “flag” to identify teen drivers statistically less likely to be
involved in a collision.
It is also possible that teen drivers who purchase teenSMART are from a household with a
higher socio-economic status and income, which are factors shown to be related to lower
collision rates (Chenetal, 2010; Hasselberg and Laflamme, 2003; Males, 2007; Males 2009).
As a counter argument, however, teen drivers may be more likely to take teenSMART
because of the insurance discount (suggestive of a lower socio-economic status) and not
because they are more safety conscious. However, another confounding factor may be that
teenSMART teens may be motivated not to report claims for collisions because this could
result in the insurance discount being pulled. This could contribute to, or be the reason for,
teenSMART teens having lower collision rates.
Importantly, the above discussion is somewhat speculative and to our knowledge, research
has not been conducted that has either established a relationship between insurance
discounts and being more safety conscious or teen drivers with insurance discounts being
less likely to report a collision to their insurance company.
Comparing self-selected groups in a quasi-experimental design is admittedly a weaker
design than an experimental one, fraught with interpretive difficulties, but it is better than
no evaluation at all, if a more rigorous evaluation is not feasible (Campbell and Stanley
15
Traffic Injury Research Foundation
1966). An experimental design with random assignment is costly and as can be attested by
the well-known DeKalb study of driver education, even an experimental evaluation design
can produce ambiguous results because not all factors can be controlled in a naturalistic
setting – e.g., teens can be assigned to a program and not take it or only partially complete
it resulting in drop-out rates that result in significant self-selection bias; some may choose
to delay licensure whereas others become licensed as soon as they are eligible introducing
further confounding factors (Mayhew et. al., 1998; Stock et. al., 1993; Williams et al.,
2009).
The bottom line is that teen drivers who take teenSMART have lower collision rates than
those who do not. This is supported by the analysis of insurance collision claims data as
well as the practice of insurance companies offering a discount for teenSMART. Far less
clear, however, is whether teenSMART training actually reduces teen driver collisions, or
other factors (greater safety consciousness of teen drivers choosing to take teenSMART),
independently or in combination with training, explain the lower collision rates of
teenSMART graduates.
16
Traffic Injury Research Foundation
discussion
The critical review of available documentation has shown that teenSMART has face validity
and content (logical) validity as a collision reduction program based on expert knowledge,
judgment and consensus. It has also been shown to have criterion-related validity (meeting
its safety objectives) in terms of statistically significant improvements in: knowledge, drivingrelated skills, and on-road driving performance. These are encouraging results from the
developmental evaluation of teenSMART. The replication of these findings with a larger
sample is needed to further establish the criterion-related validity of teenSMART.
The evidence is mixed and less clear for teenSMART in terms of collision reductions.
teenSMART has been shown to have criterion-related validity based on the indirect,
antecdotal evidence: the practices of six insurance companies that offer an insurance
discount for teenSMART; and Departments of Insurances in most states that approve the
rate filings by insurance companies for the discount. This suggests that these insurance
companies have solid evidence convincing them that teenSMART reduces their claims
experiences – i.e., teenSMART graduates have statistically lower collision rates. The
veracity of this evidence is further supported by three insurance companies – Allstate, AAA
Insurance and Liberty Mutual – recently going on public record to espouse the collisionreduction benefits of teenSMART. To some extent this is supported by direct, statistical
evidence from studies showing that teen drivers exposed to teenSMART have statistically
significant lower collision rates than those not exposed to teenSMART. However, these
results are suggestive but not conclusive because these studies have not adequately
accounted for confounding factors, such as self-selection.
This critical review supports the claim that teen drivers who take teenSMART have lower
collision rates than those who do not. The reasons for this, however, are less clear and
could be the result of teenSMART training or self-selection, for example, by more safety
conscious teen drivers taking teenSMART, or a combination of these factors. Given that
teenSMART has been shown in the developmental evaluation to have some criterionrelated validity – improved knowledge, driving-related skills, and especially, on-road driving
performance – it seems reasonable to conclude that some combination of training and
other person-centered factors likely contributed to the lower collision rates of teenSMART
graduates. Improvements in intermediate, criterion-related measures, such as on-road
driving performance, through teenSMART training should logically translate into fewer teen
driver crashes. The extent to which this is the case, however, and the relative contribution
17
Traffic Injury Research Foundation
of teenSMART and other person-centered factors related to self-selection, cannot be
determined from the available information provided for this critical review and White Paper.
18
Traffic Injury Research Foundation
references
Campbell, D.T. and Stanley, J.C. (1966). Experimental and Quasi-Experimental Designs for
Research. Chicago, IL.: Rand McNally.
Chen, H.Y., Ivers, R. Q., Martiniuk, A.L.C., Boufous, S., Senserrick, T., Woodward, M.,
Stevenson, M., and Norton, R. (2010). Socioeconomic status and risk of car crash injury,
independent of place of residence and driving exposure: results from the DRIVE Study.
Journal of Epidemiology and Community Health 64, 998-1003.
Clinton, K. and Lonero, L. (2006). Evaluation of Driver Education: Comprehensive
Guidelines, Washington, D.C.: AAA Foundation for Traffic Safety.
Compton, R. and Ellison-Potter, P. (2008). Teen Driver Crashes: A Report to Congress.
Washington, D.C.: National Highway Traffic Safety Association.
Cozby, P.C. (2007). Methods of Behavioral Research, Ninth Edition. New York, NY: McGrawHill.
Curry A.E., Hafetz J, Kallan M.J., Winston F.K., Durbin D.R. (2011). Prevalence of teen driver
errors leading to serious motor vehicle crashes. Accident Analysis and Prevention.
Hasselberg, M and L. Laflamme, L. (2003). Socioeconomic background and road traffic
injuries: A study of young car drivers in Sweden. Traffic Injury Prevention, 4 (3), 249–254.
Lonero, L. and Mayhew, D.R. (2010). Large-Scale Evaluation of Driver Education Review
of the Literature on Driver Education Evaluation: 2010 Update. Washington, D.C.: AAA
Foundation for Traffic Safety.
Males, M. (2007). Poverty, not “risk taking,” may be the real problem in teenage accidents.
The Chronicle of the American Driver and Traffic Safety Education Association, 55(2), 4–5.
Males, M. (2009). The role of poverty in California teenagers’ fatal traffic crash risk.
Californian Journal of Health Promotion, 7(1), 1–13.
Mayhew, D.R. and Simpson, H.M. (1990). New to the Road. Young Drivers and Novice
Drivers: Similar Problems and Solutions? Ottawa. Ontario: Traffic Injury Research
Foundation.
Mayhew, D.R. and Simpson, H.M. (1995). The Role of Driving Experience: Implications for
the Training and Licensing of New Drivers. Toronto, Ontario: Insurance Bureau of Canada.
Mayhew, D.R., and Simpson, H.M. (1997). Effectiveness and role of driver education
and training in a graduated licensing system. Ottawa, Ontario: Traffic Injury Research
Foundation.
19
Traffic Injury Research Foundation
Mayhew, D. R., H. M. Simpson, and Robinson, A. (2002). The safety value of driver
education and training. Injury Prevention 8 (Supplement II):3ii-8.
Mayhew, D.R., Simpson H.M., Singhal D; and Desmond K. (2006). Reducing the Crash Risk
for Young Drivers. Washington, D.C.: AAA Foundation for Traffic Safety.
Mayhew, D.R., Simpson, H.M., Williams, A.F., and Ferguson, S.A. (1998). Effectiveness and
role of driver education and training in a graduated licensing system. Journal of Public
Health Policy 19(1): 51-67.
NHTSA (2009) Traffic Safety Facts 2009: Teen Drivers. Washington D.C.: National Highway
Traffic Safety Administration, Dept. of Transportation (US).
National Research Council, Institute of Medicine, and Transportation Research Board.
(2007). Preventing Teen Motor Crashes: Contributions from the Behavioral and Social
Sciences, Workshop Report. Program Committee for a Workshop on Contributions from
the Behavioral and Social Sciences in Reducing and Preventing Teen Motor Crashes.
Washington, DC: The National Academies Press.
Pedhazur, E.J. and Schmelkin, L.P. (1991). Measurement, Design and Analysis: An
Integrated Approach. Hillsdale, New Jersey: Laurence Erlbaum Associates Inc.
Shope, J., T.C. and Bingham, C.,R. (2008). Teen Driving Motor-Vehicle Crashes and Factors
That Contribute. American Journal of Preventive Medicine: 35(3S):S261–S271.
Stock, J. R., Weaver, J. K., Ray, H. W., Brink, J. R., & Sadoff, M. G. (1983). Evaluation of Safe
Performance Secondary School Driver Education Curriculum Demonstration Project (DOT HS
806 568). Washington, DC: U.S. Department of Transportation.
TIRF (1995). A New Approach to Reducing Collisions Among Young Drivers. Ottawa,
Ontario: Traffic Injury Research Foundation.
Trochim, W. (2006). Research Methods Knowledge Base. (http://www.
socialresearchmethods.net/kb/measval.php)
Williams A., Preusser, D., and Ledingham, K. (2009). Feasibility Study on Evaluating Driver
Education Curriculum. Washington D.C.: National Highway Safety Administration.
20
Traffic Injury Research Foundation
appendix a:
advisory team members
21
Traffic Injury Research Foundation
teensmart advisory team
Affiliations at the Time of teenSMART Development
Dr. Baruch Fishhoff, Ph.D
Director and Howard Heinz University Professor
Department of Social and Decision Making Sciences
Carnegie Melon University
Dr. Richard Harkness, Ed.D, CEO
CEO
ADEPT Driver
Dr. Richard Jessor, Ph.D
Director and Professor of Psychology
Institute of Behavioral Sciences
University of Colorado at Boulder
Mr. Dan Mayhew, M.A.
Senior Vice President
Traffic Injury Research Foundation
Dr. Jim McKnight, Ph.D
Principal Researcher
Pacific Institute for Research and Evaluation
Dr. Allison Rossett, Ed.D
Professor of Instructional Technology
School of Education San Diego State University
Dr. Herb Simpson, Ph.D
President and CEO
Traffic Injury Research Foundation
23
Traffic Injury Research Foundation
appendix b:
a new approach to reducing collisions
among young drivers
25
Traffic Injury Research Foundation
27
Traffic Injury Research Foundation
28
Traffic Injury Research Foundation
29
Traffic Injury Research Foundation
30
Traffic Injury Research Foundation
31
Traffic Injury Research Foundation
32
Traffic Injury Research Foundation
33
Traffic Injury Research Foundation
34
Traffic Injury Research Foundation
35
Traffic Injury Research Foundation
36
Traffic Injury Research Foundation
37
Traffic Injury Research Foundation
38
Traffic Injury Research Foundation
39
Traffic Injury Research Foundation
40
Traffic Injury Research Foundation
41
Traffic Injury Research Foundation
42
Traffic Injury Research Foundation
43
Traffic Injury Research Foundation
44
Traffic Injury Research Foundation
45
Traffic Injury Research Foundation
46
Traffic Injury Research Foundation
47
Traffic Injury Research Foundation
48
Traffic Injury Research Foundation
49
Traffic Injury Research Foundation
50
Traffic Injury Research Foundation
51
Traffic Injury Research Foundation
52
Traffic Injury Research Foundation
53
Traffic Injury Research Foundation
54
Traffic Injury Research Foundation
55
Traffic Injury Research Foundation
56
Traffic Injury Research Foundation
57
Traffic Injury Research Foundation
58
Traffic Injury Research Foundation
59
Traffic Injury Research Foundation
60
Traffic Injury Research Foundation
61
Traffic Injury Research Foundation
62
Traffic Injury Research Foundation
63
Traffic Injury Research Foundation
64
Traffic Injury Research Foundation
65
Traffic Injury Research Foundation
66
Traffic Injury Research Foundation
67
Traffic Injury Research Foundation
68
Traffic Injury Research Foundation
69
Traffic Injury Research Foundation
70
Traffic Injury Research Foundation
71
Traffic Injury Research Foundation
72
Traffic Injury Research Foundation
73
Traffic Injury Research Foundation
74
Traffic Injury Research Foundation
75
Traffic Injury Research Foundation
76
Traffic Injury Research Foundation
77
Traffic Injury Research Foundation
78
Traffic Injury Research Foundation
79
Traffic Injury Research Foundation
80
Traffic Injury Research Foundation
81
Traffic Injury Research Foundation
82
Traffic Injury Research Foundation
83
Traffic Injury Research Foundation
84
Traffic Injury Research Foundation
85
Traffic Injury Research Foundation
86
Traffic Injury Research Foundation
87
Traffic Injury Research Foundation
88
Traffic Injury Research Foundation
89
Traffic Injury Research Foundation
90
Traffic Injury Research Foundation
91
Traffic Injury Research Foundation
92
Traffic Injury Research Foundation
93
Traffic Injury Research Foundation
94
Traffic Injury Research Foundation
95
Traffic Injury Research Foundation
96
Traffic Injury Research Foundation
appendix c:
developmental evaluation of teensmart
97
Traffic Injury Research Foundation
99
Traffic Injury Research Foundation
100
Traffic Injury Research Foundation
101
Traffic Injury Research Foundation
102
Traffic Injury Research Foundation
103
Traffic Injury Research Foundation
104
Traffic Injury Research Foundation
105
Traffic Injury Research Foundation
106
Traffic Injury Research Foundation
107
Traffic Injury Research Foundation
108
Traffic Injury Research Foundation
109
Traffic Injury Research Foundation
110
Traffic Injury Research Foundation
111
Traffic Injury Research Foundation
112
Traffic Injury Research Foundation
113
Traffic Injury Research Foundation
114
Traffic Injury Research Foundation
115
Traffic Injury Research Foundation
116
Traffic Injury Research Foundation
117
Traffic Injury Research Foundation
118
Traffic Injury Research Foundation
119
Traffic Injury Research Foundation
120
Traffic Injury Research Foundation
121
Traffic Injury Research Foundation
122
Traffic Injury Research Foundation
123
Traffic Injury Research Foundation
124
Traffic Injury Research Foundation
125
Traffic Injury Research Foundation
126
Traffic Injury Research Foundation
127
Traffic Injury Research Foundation
128
Traffic Injury Research Foundation
129
Traffic Injury Research Foundation
130
Traffic Injury Research Foundation
131
Traffic Injury Research Foundation
132
Traffic Injury Research Foundation
133
Traffic Injury Research Foundation
134
Traffic Injury Research Foundation
135
Traffic Injury Research Foundation
136
Traffic Injury Research Foundation
137
Traffic Injury Research Foundation
138
Traffic Injury Research Foundation
139
Traffic Injury Research Foundation
140
Traffic Injury Research Foundation
appendix d:
experiences of three insurance companies
with teensmart
141
Traffic Injury Research Foundation
143
Traffic Injury Research Foundation
144
Traffic Injury Research Foundation
145
Traffic Injury Research Foundation
146
Traffic Injury Research Foundation
147
Traffic Injury Research Foundation
148
Traffic Injury Research Foundation
149
Traffic Injury Research Foundation
150
Traffic Injury Research Foundation
appendix e:
safety effectiveness of teensmart
151
Traffic Injury Research Foundation
153
Traffic Injury Research Foundation
154
Traffic Injury Research Foundation
155
Traffic Injury Research Foundation
156
Traffic Injury Research Foundation
157
Traffic Injury Research Foundation
158
Traffic Injury Research Foundation
159
Traffic Injury Research Foundation
160
Traffic Injury Research Foundation
161