The Case of the Repeating Licensure
Examination Candidate:
Psychometric, Policy, and Legal Issues
Moderator
I. Leon Smith, President & CEO
Professional Examination Service
Presenters:
Mary Browne, Program Director
Professional Examination Service
Julia C. Works
Atkinson & Atkinson
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
The Case of the Repeating Licensure
Examination Candidate:
Psychometric and Policy Issues
Mary Browne
Program Director, PES
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Why Retake?
•
•
•
•
Failure
Exam Malfunction
Suspicion of cheating
Illness or other incident
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Second, Third…..Chances
•
•
•
•
Legal rulings support programs that allow
retakes
The spirit behind most US education and
law is to allow second chances
How does this impact on public protection
Should there be a cutoff on number of
retakes allowed
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Instead of a Retake
•
•
Alternative form of assessment
Partial retake - Retaking only domains or
sections failed
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Remediation
•
•
Should it be required
Is it effective
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Retake Concerns
•
•
•
•
•
•
Overlap with previous forms
Timing
What results patterns are expected?
Suspicious?
Practice effect
False Positives
Adverse Impact
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Overlap With Previous Forms
• Repeated Items
– Equating
– Item Bank limitations.
• How do candidates perform on a
repeated item?
– Repeat previous response
– Intervening learning based on recollection of
the item?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Retake Timing Considerations
• How frequently is testing offered?
• How many forms of the test are available?
• How different are test forms?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Reasons for Concern
• False Positives
• False Negatives
• Errors of Measurement
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
True Score and Standard Error
• Obtained score = True Score + Random Error
83 = 83 ± 3 or 80-86
• Errors are extraneous variables such as the
fatigue, the particular sample of questions
chosen for the test, environmental conditions or
luck in guessing.
• For every candidate there is a “best” and
“worst” score possible.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Some Examples of Score Changes in a
Licensing Program.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
From One Exam to the Next
Times Tested
% Passing
Average Score
Change
Standard
Deviation
2 (N=532)
75%
+25
23
3 (N=169)
4 (N=52)
53%
48%
+21
+16
38
21
5(N=49)
6 (N=36)
7(N=13)
37%
47%
38%
+11
+11
+16
25
21
20
8 (N=13)
9 (N=12)
23%
25%
+8
+3
23
18
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Two Common Failing Score Patterns
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Score Patterns - Failing
NOT LIKELY TO PASS
450
400
350
300
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Score Patterns - Failing
SOME CHANCE TO PASS - SHOULD THEY?
450
400
350
300
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Passing Score Patterns
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Score Patterns - Passing
SOME TYPICAL PATTERNS
500
450
400
350
300
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Some Score History Illustrations
SUCCESS!
383
410
407
383
441
404
382
413
378
401
405
414
421
430
419
443
433
436
447
455
INTERVENTION
338
407
413
404
376
383
424
412
431
430
477
415.25
23.10359963
408.6363636
35.54229235
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Passpoints
• Most passpoints are set to pass the
minimally qualified candidate.
• Candidates whose qualifications are at a
level just below the minimum may pass
due to score error.
– With repeated testing, this becomes more
likely.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Score Patterns - Passing
PASSED! JUST BARELY
450
400
350
300
1
2
3
4
5
6
7
8
9
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
10
11
Dilemmas
• Balance between fairness in allowing
candidate retests and protecting the
public from unqualified candidates.
• Eliminating candidate advantage gained
from repeated retakes
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Remedies
• Adaptive Testing
• Set a Confidence Interval or Indifference
Zone
• Change the passpoint depending on
number of retakes
– Raise the passpoint to reflect retake
advantage
– Average last two or three test scores
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Who Should Benefit?
• Should decisions be weighted on the side
of public protection or fairness to
candidates? Where is the balance here?
• Will more stringent retake requirements
result in adverse impact?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Contact Information
Mary Browne
PES
475 Riverside Drive
212 367-4250 (ph)
212 367-4266 (fax)
browne@proexam.org
http://www.proexam.org
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Case of the Repeating Licensure
Examination Candidate:
A Legal Perspective
Julia C. Works
Atkinson & Atkinson
1603 Orrington Avenue
Suite 2080
Evanston, IL 60201
Tel. 847.864.0070
Fax.847.864.0588
E-mail: aanda@ameritech.net
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Legal Perspective
• Credential:
– Voluntary
– Private Sector
vs.
• High Stakes Licensure Examination:
– Mandatory
• Statutes and/or Rules
– Public Protection
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
High Stakes Licensure Examination
• Who relies on examination results?
Regulatory Boards
• For what purpose?
– Measure of Minimum Competence
– Protection of the public from unqualified
practitioners
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Legal Perspective
• Limiting the number of attempts:
How do regulatory boards look at this
issue?
 Security – exposure of exam items
 Validity of Results
– Does the fact that it took an applicant X number of
times to pass the licensing exam mean they are less
than minimally competent?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Currently…
• Majority of regulatory boards do not limit
the number of examination attempts.
• Those boards that do limit the number of
attempts may require candidates to seek
some type of remedial education before
being permitted to sit for the examination
again.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Currently…
• Many high stakes licensure examinations
may not be retaken within a 90-day
window
• Examination attempts may be limited to a
specified number within a specified time
period
– For example:
• Limited to 3 attempts/retakes
• Limited to 4 times within open window of
administration of examination
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
How do you limit the number of
attempts?
•
•
•
•
Statute
Rule/Regulation
Contractually
Policies and Procedures
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Legality of Limitations
• Constitutional Claims challenging
limitation on number of attempts:
– Equal Protection
– Due Process
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Equal Protection
• Argument may be asserted that limiting
the number of attempts violates Equal
Protection because such limitations,
whether established through statute or
regulation, are not rationally related to any
legitimate state interest.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Linton v. Missouri Veterinary
Medical Board
• Legislature changed law to limit number of attempts to
pass licensing exam to three(3). (Effective Aug. 1992)
Facts:
– Following revision to law, applicant failed licensing exam 3 times
in MO but passed on the 4th attempt in Illinois
– Applicant then applied to MO Board for license
– Application was denied
– Applicant appealed board decision asserting statute and
regulation violated her right to Equal Protection
Linton v. Missouri Veterinary Medical Board, 988 S.W.2d 513 (Mo.banc 1999).
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Linton v. Missouri Veterinary
Medical Board
• Court held applicant’s right to Equal Protection
not violated where three-exam limitation was
rationally related to a legitimate state interest
(i.e. the establishment of a high level of
competence for veterinarians in Missouri).
• Court found the interest of the legislature here
was the same interest as establishing the test in
the first place.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Linton v. Missouri Veterinary
Medical Board
• Next question under Equal Protection
Analysis:
Is the means chosen rationally related to
achieving the legitimate state interest?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Linton v. Missouri Veterinary
Medical Board
In other words…
 Are those who must take the exam four or
more times before passing of equal or greater
competence than those who pass the test in
three or fewer attempts?
 Is there a reasonable connection between
competence and those who pass the exam in
three or fewer attempts?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Linton v. Missouri Veterinary
Medical Board
• Following this decision…
Statute amended to read, in part, as follows:
…If that passing score was not received within three
attempts, the board may require the applicant to
appear before the board or submit evidence that the
applicant has completed at least thirty hours of
board-approved continuing education.
V.A.M.S. 340.234 Examination– application and fees.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Currently…
• Similar restriction in Alabama with regard to the
state examination where board rule reads, in
part, as follows:
(4) A maximum of three retakes of the state
written exam will be permitted.
Alabama Admin. Code 930-X-1-.09.Examinations
Alabama State Board of Veterinary Medicine.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Due Process
• Allegations that limitation on number of
attempts violates Due Process clause of
U.S. Constitution because it deprives
individuals of a vested right to take the
licensing examination.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
State Board of Registration for the
Healing Arts v. Boston
Statute limited the number of times physical therapist assistants
could take licensing examination
Facts:
- Applicant failed physical therapy assistant exam three times prior
to August 1999, when statute limiting number of attempts became
effective.
- Applicant applied to take physical therapy assistant exam for 4th
time after the statute effective date.
- Board denied application. Administrative Hearing Commission
reversed Board decision and ordered Board to admit her to the
examination.
- Board sought review of Administrative Hearing Commission
decision that found statute could not be applied retroactively.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
State Board of Registration for the
Healing Arts v. Boston
• Court found that the legislative intent of the
statute clearly manifested that it be applied
retroactively.
– Statute:
• The board shall not issue a license to practice as a physical
therapy assistant or allow any person to sit for the Missouri
state board examination for physical therapist assistants
who has failed three or more times any physical
therapist assistant licensing examination administered
in one or more states or territories of the United States or
the District of Columbia
V.M.S.A. Section 334.655.3, amended Aug. 28, 1999
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
State Board of Registration for the
Healing Arts v. Boston
• Court found that the Missouri
Constitution bars retrospective
application of a statute except where:
1) legislative intent is clearly manifested
that the statute is to be applied
retrospectively; and 2) the statute is
procedural only and does not affect any
substantive or vested right.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
State Board of Registration for the
Healing Arts v. Boston
• Court found that statutory language
unmistakably refers to past conduct
• Court further held that application of the
statute did not deprive applicant of any vested
right.
– As a general matter, professional licensing in a
particular profession is a privilege granted by the
state.
– Restrictions surrounding the practice of a profession
are for the benefit of society, not the practitioner
– Licensing statutes confer no substantive rights upon
applicants seeking licensure
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Other situations to be considered
• Licensure by Endorsement
– See Yap v. Zollar, 691 N.E.2d 18 (Ill. App. 1 Dist.
1997) Statute requiring candidates for licensure as
registered nurses to have passed nationwide exam
within 3 years of first attempt rather than passing
within 6 attempts within an unlimited period of time
could only be applied prospectively because this was
a substantive change and retroactive application
would deprive applicant of a vested right where
applicant had reasonably relied on prior statute.
– See also John v. Dept. of Professional Regulation,
713 N.E.2d 673 (Ill. App. 1 Dist. 1999).
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Other situations to be considered
• Foreign Graduate Equivalency
Programs
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Contact Information
Julia C. Works
Atkinson & Atkinson
1603 Orrington Avenue
Suite 2080
Evanston, IL 60201
Tel.: 847.864.0070
Fax: 847.864.0588
E-mail: aanda@ameritech.net
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Reference
Millman, J. (1989). If at First You Don’t Succeed:
Setting Passing Scores When More Than One
Attempt is Permitted. Educational Researcher,
18(6), 5 – 9.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri