CLEAR 2003 Annual Conference
Pushing the Boundaries of Regulation
Practice Makes Perfect: Or Does It?
Presenters
Terry Kowalski, Business Development
Promissor
Moderator
I. Leon Smith, President & CEO
Professional Examination Service
Steven S. Nettles, V. P., Research & Development
Applied Measurement Professionals, Inc.
Paul H. King, Vice President for Programs
Professional Examination Service
William A. Hatherill, CEO
Federation of State Boards of Physical Therapy
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Opinions about Practice
Tests
Please complete the four
pre-presentation ratings NOW.
THANK YOU!
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
No Amount of Practice
Will Help!
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Presentations will
Address the Following
Questions
What are the different ways practice
tests are offered?
What are candidate reactions to
practice tests?
Is there a relationship between practice
test results and the “real thing”?
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Presentations will
Address the Following
Questions (continued)
Who takes practice tests and when?
How are practice tests marketed and
positioned?
Are there ethical concerns when a
sponsor of a credentialing program
offers practice tests?
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
PRACTICE TESTS
IN STATE REGULATORY
PROGRAMS
Terry Kowalski
Business Development - Promissor
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
On-Line Practice Tests
Markets
State Regulatory
Potential licensure candidates
Pre-licensing schools
Professional Associations
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Why offer practice
tests?
For test-takers….
An experience that
emulates the actual
exam.
For test sponsors…
An opportunity to
provide a valued
customer service
An opportunity for
additional revenue
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
State Regulatory Market
Background
Insurance and real estate
Offered P&P practice tests for 10 years
Migrated tests to new CBT platform
January 2003
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice Tests
Same format, content, structure
Same length as actual exam
Same rigorous test development
process
Same delivery platform
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice Tests
The candidate gets
Item level feedback on correct response
Score reports with total correct/incorrect by
content area
A guide to content areas that need further
study
A chance to familiarize themselves with
test format
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice Tests
Candidates are advised repeatedly not
to use the practice test as a predictive
indicator of performance on the actual
licensing exam, but as a guide to
content areas needing further study and
a way to familiarize themselves with the
format.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Survey Results - Dec ‘02
Percentage of candidates agreeing
Instructions clear - 96%
Easy to navigate – 95%
Item formatting easy to read – 92%
Easy to mark for review – 93%
Review screen helpful – 87%
Feels better prepared for licensing exam after P.T. – 89%
Easy to check for correct answer – 92%
Would pay more to take P.T. in actual test center –24%
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Presentation Follow-up
Please give me your business card to
receive an e-mail of the presentation
materials -ORPresentation materials will be posted on
CLEAR’s website
THANK YOU!
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Candidate Perceptions of
Web-delivered Self
Assessment
Examinations
Steven S. Nettles, EdD
VP, Research and Development
Applied Measurement Professionals, Inc.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Survey Plan
A 27 item survey was placed on the
Web for approximately 2 months this
summer.
Candidates from the last 12 months
were encouraged to visit the site and
complete the survey.
188 respondents completed the survey.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Gender
Male
23.4%
Missing
20.7%
Female
55.9%
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Racial Group
missing
25.0%
Asian / PI
White
7.4%
52.1%
Black
8.5%
Hispanic
5.3%
Multiracial
1.6%
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
60
53
50
40
30
26
Percent
20
10
11
5
0
Missing
4
Disagree
Strongly Disagree
Agree
Neutral
Strongly Agree
The SAE helped me learn the content for the real test.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Percent
50
45
40
40
30
20
10
9
3
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
Strongly Agree
The SAE helped me become more comfortable with CBT
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Percent
60
50
51
40
30
25
20
13
10
6
3
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
Strongly Agree
The SAE helped me identify areas of weakness in test content.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Percent
50
47
40
41
30
20
10
9
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
The SAE allowed me to practice with the test format.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Strongly Agree
Percent
40
35
30
31
20
20
10
11
3
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
Strongly Agree
The SAE allowed me to practice taking the test multiple times
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
60
50
50
40
30
20
20
Percent
15
10
9
5
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
Strongly Agree
The SAE gave me immediate feedback about specific skills in this job.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
50
45
40
37
30
20
Percent
10
9
8
0
Missing
Strongly Disagree
Neutral
Agree
Strongly Agree
Being able to purchase the SAE any time is convenient.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
60
53
50
40
34
30
Percent
20
10
9
4
0
Missing
Strongly Disagree
Neutral
Agree
Strongly Agree
Being able to take the SAE any day, any time is convenient.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
50
42
40
40
30
20
Percent
10
10
5
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
It was easy to complete the SAE on my computer.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Strongly Agree
Percent
40
37
30
20
21
18
10
10
9
5
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
Strongly Agree
The SAE was a good value for what I paid.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
50
40
39
30
23
20
21
Percent
10
9
5
3
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
The SAE has items similar to those on the real test.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Strongly Agree
40
35
30
20
20
18
Percent
10
10
10
8
0
Missing
Disagree
Strongly Disagree
Agree
Neutral
Taking the SAE guarantees success on the real test.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Strongly Agree
40
32
30
29
20
17
12
10
Percent
10
0
Missing
Between 1 and 3 mo
Less than 1
More than 6
Between 3 and 6 mo
# months between taking SAE and real test
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Percent
50
40
38
36
30
20
10
13
10
3
0
Missing
Minimal Value
No Value
Extreme Value
Moderate Value
Correct/Incorrect response explanations
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
50
45
40
30
26
20
14
Percent
10
10
5
0
Missing
Minimal Value
No Value
Extreme Value
Moderate Value
Diagnostic Score Reports showing sub-scores
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
50
42
40
30
30
20
14
Percent
10
11
3
0
Missing
Minimal Value
No Value
Extreme Value
Moderate Value
Total Score (number of items answered correctly)
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
100
90
80
70
60
50
40
60
70
80
90
Exam A Test Score (%)
r=0.59, p<.05, n=12
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
100
100
90
80
70
60
50
40
60
70
80
90
Exam B Test Score (%)
r=0.114, n.s., n=31
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
100
Summary
Respondents appear to have positive
perceptions about the Web-based SAEs
addressed in this survey.
The failure of many respondents to
provide test score information allowed
no conclusive results about the
relationship between real and SAE
exam scores.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Presentation Follow-up
Please give me your business card to
receive an e-mail of the presentation
materials -ORPresentation materials will be posted on
CLEAR’s website
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice Makes
Perfect – Or Does It?
Paul H. King
Vice President for Programs
Professional Examination Service
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Stated Purposes of
Practice Test
Provide exposure to format and content of
certification test
Identify possible areas of weakness so that
further study efforts can be more focused
Assess readiness to take certification test
Emphasizes that specific score on the
practice test does not guarantee the same or
similar score on the certification test
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice vs. Actual Test
Similarities
Constructed to identical test specifications
Actual pre-calibrated test questions from
certification test
Timing simulates that of certification test
Passing score equated to same standard as
certification test
Feedback on test performance – overall and
by domain or major area
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice vs. Actual Test
Differences
No eligibility requirements
Shorter test – potential for more
measurement error, especially for domains
Uncontrolled test administration conditions
Raw vs. scaled total scores
Links to relevant study material provided
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Potential Advantages
Accessibility – 24/7; anywhere with internet
access; ease of registration; allowed
interruptions
Immediate feedback – performance and what
to do next
Narrows the disconnect between training
courses and test expectations
Significantly less cost than certification test to
assess readiness ($45 vs. $215 /$355)
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice vs. Certification
Candidates
December 2002
Level
May 2003
Practice
Minus
Repeaters
Cert.
Cands.
Practice
Minus
Repeaters
Cert.
Cands.
Level 1 (874 - 89)
7031
(754 - 57)
697(12%)
5696
4229
(681 - 43) 3320
638 (19%)
785 (11%)
Level 2 (776 - 99)
677 (16%)
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Timing and Number of
Practice Tests
1200
1000
800
600
400
200
0
Oct- Nov- Dec- Jan- Feb- Mar- Apr- May- Jun- Jul02
02
02
03
03
03
03
03
03
03
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Percentage Passing
Practice Test by Month &
Exam Level
80
70
60
50
40
30
20
10
0
Level 1
03
ay
3
pr
*M
_0
3
_0
A
M
ar
03
b_
Fe
n_
03
2
Ja
_0
ec
2
*D
_0
ov
N
O
ct
_0
2
Level 2
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Consistency of Performance
Practice (P) vs. Certification (C)
LEVEL
Taking
Both
Pass or
Fail Both
Fail P
Pass C
Pass P
Fail C
Level 1
1217
993
(81%)
166
(14%)
58
(5%)
Level 2
1139
887
(78%)
166
(15%)
86
(7%)
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
“The Good Book”
Standard 8.1: “Any information about test content and
purposes that is available to any test taker prior to
testing should be available to all test takers.
Important information should be available free of
charge and in accessible formats.”
Comment: “…More detailed information, such as
practice materials, is sometimes offered for a fee.
Such offerings should be made to all test takers.”
- Standards for Educational and Psychological
Testing, AERA, APA, NCME (1999).
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Ethical Considerations
Fee to take practice test – charging for what
should be free? Necessary information vs.
added value? Enabling early decision on
readiness hence saving money? Recouping
cost of developing actual items?
Perceptions of performance – luring good
performers into complacency? Misleading
candidates as to readiness? Realistic guide
to readiness and focus of future study?
Education vs. certification – process or
product? Test preparation or test readiness?
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Presentation Follow-up
Please pick up a handout from this
presentation.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Practice makes
Perfect: Or does it?
William A. Hatherill, CEO
Federation of State Boards of
Physical Therapy
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Decision Making
Process
Context of the organization
Ethical Issues?
Mission oriented?
Moving forward
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Context of the
Organization
Charged to develop alternate sources of
revenue





90% of Federation revenue is from NPTE
NPTE volume has declined by 30+% in last 5
years (19,500 to 14,000)
Compromised old form NPTE
Test Phobia = Level the Playing Field
Transition from didactic learning to applied clinical
application skills
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Ethics
A branch of philosophy, is considered a
normative science, because it is
concerned with norms of human
conduct, as distinguished from the
formal sciences, such as mathematics
and logic, and the empirical sciences,
such as chemistry and physics
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Ethics
Benchmark Study of Other Similar
Organizations
National Association of Boards of
Pharmacy
 Association of Social Work Boards
 PSAT

Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Mission
The purpose of the Federation is to
protect the public by providing service
and leadership to promote safe and
competent physical therapy practice.
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Mission
Does the Mission support the
development of a review guide?
Does it protect the public?
 Does it harm the public?
 Does it promote safe and competent
physical therapy practice?

Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Ethics & Mission
Should a review guide or practice exam be
offered by the sponsor of a licensure
examination?

Appearance of a Conflict of Interest




Concerns from stakeholders
Design the Test = Design Study Guide
Unfair Marketing Advantage: Assess to Applicants
Internal Conflicts


Marketing
Resource Allocation
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Moving Forward:
Research
Student Needs and Perspective
Faculty Perspective
Financial Considerations
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Student Needs and
Perspectives
Student Need

87% plan to purchase some prep material
for NPTE
Pricing Strategy

65% will spend between $50 - $150
Marketing Strategy

67% plan to acquire one specific vendor’s
product
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Student Needs and
Perspectives
Desirable Product Features
Overall Quality
 Sample tests for review
 Rationales for correct answers
 Multiple, different exams with one purchase
 Immediate performance feedback

Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Student Needs and
Perspectives
Positive features of Federation product
Ability to stop and restart
 Reputation of the provider
 Similarity to look and feel of NPTE
 Inclusion of rationales

Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Faculty Perspectives
Mixed responses as to faculty’s role to
assist students with NPTE prep
95% felt unauthorized copying and
distribution would be a problem if site
licenses were offered
Faculty Training in Item Writing skills
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Financial Considerations
Initial investment for product development

$100,000
Estimated annual marketing expenses

$5 per prospect/year or $50,000
Estimated market penetration for new product
per year

10% year 1; 15% year 2; 20% year 3
Necessary break-even point

Approximately 16%
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Product Performance:
First Year?
Approximately 5% Market Share
$25,000 in revenue
$35,000 in marketing expense
$33,000 in start-up cost ($100K
allocated over 3 years)
$43,000 Loss
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Needs and Perspectives:
Next Round
Students who purchased
Students who did not purchase
Faculty
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Lessons Learned
Refocus Product:

Provide one time frame for one set price
Online Review Guide becomes Practice
Exam and Assessment Tool (PEAT)
Reposition Product: Supplement for other
review guides
Instant Gratification – Allow purchase by
Credit Card online
Item Writing
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Study Considerations not
followed
Second Practice Exam
Marketing
• Consider offering students promotional discounts for
ordering at the same time as registering for the
NPTE. (90% favor this option)
• Consider developing a “student distributor” program
whereby selected students might receive a small
cash payment ($25)
• Consider initiating discussions with existing Test Prep
companies to package the FSBPT Exam with the
Test Prep companies review guide
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Unintended Consequences
Copyright Violation
Exam Prep Company, requested
eliminating PEAT in turn for shutting down
the “Discussion Forum”
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
PEAT Today
Year 3
2003
Approximate 14% Market Share
Projected $120,000 In Annual Revenue
$ 0 In Marketing Expense
Preparing to launch PTA PEAT
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003
Opinions about Practice
Tests
Please complete the four
post-presentation ratings NOW.
THANK YOU FOR YOUR
PARTICIPATION!
Presented at CLEAR’s 23rd Annual Conference
Toronto, Ontario September, 2003