COMLEX-USA UPDATE FOR NEW PROGRAM
DIRECTORS
ACGME Annual Meeting
February 26, 2015
“To protect the public by providing the means
to assess competencies for osteopathic
medicine and related health care professions”
www.nbome.org
© 2014 NBOME
JOHN R. GIMPEL, DO, MEd
President & CEO
www.nbome.org
© 2014 NBOME
National Board of Osteopathic Medical Examiners
Mission
To protect the public by providing the means to
assess competencies for osteopathic medicine
and related healthcare professions
www.nbome.org
© 2014 NBOME
NBOME’s Participating in Assessment across the Continuum
Practice
CLIENT
Licensure
Education
CLIENT
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
Comprehensive Osteopathic Medical Licensing
Examination – USA:
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
COMLEX-USA:

Since 1994, a distinctive, comprehensive examination blueprint and unique test
materials that address the competencies required for the safe and effective practice
of osteopathic medicine, including osteopathic principles, practices, and osteopathic
manipulative treatment (OMT)
•
A maturation-linked examination; each Level must be passed in order to take the
next Level, and Levels 1 and 2 (CE and PE) must be passed to earn the U.S. DO
degree
•
The Federation of State Medical Boards of the United States (www.fsmb.org) with an
independent review has found evidence for COMLEX-USA validity to be “exemplary”
•
COMLEX-USA is accepted in all 50 states for licensure, and in increasingly recognized
internationally (e.g., Canada, UK, New Zealand, Australia)
•
Widely used by residency program directors in resident screening and selection
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
COMLEX-USA Series
• COMLEX-USA Level 1
• COMLEX-USA Level 2 – Cognitive Evaluation (CE)
• COMLEX-USA Level 2 – Performance Evaluation (PE)
• COMLEX-USA Level 3
www.nbome.org
© 2014 NBOME
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
COMLEX-USA Current Blueprint for Testing
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
COMLEX-USA Level 1
• A computer-based, cognitive evaluation that emphasizes
application of foundational biomedical science concepts and
principles necessary for an understanding of health and
disease for the practice of osteopathic medicine. Level 1
integrates the foundational biomedical sciences of anatomy,
behavioral science, biochemistry, microbiology, osteopathic
principles, pathology, pharmacology, physiology and other
areas necessary to solving clinical problems and promoting
and maintaining health, as well as to serve as a foundation
for lifelong learning.
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
COMLEX-USA Scoring and Reporting
• COMLEX-USA uses “criterion-referenced”
methodology in standard-setting
• COMLEX-USA exams are “Pass/Fail”
– “Pass” if meets or exceeds the established
passing score
– Passing means having met or exceeded the
standard for competency (minimum competency
required)
– Candidates cannot retake the test for a higher
score upon passing
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
COMLEX-USA Scoring and Reporting, continued
• Scores for COMLEX-USA Levels 1 and 2
– Mean three-digit standard score ~ 500-540
– COMLEX-USA Level 1 and Level 2-CE passing
three-digit standard score – 400
– COMLEX-USA Level 2-PE- provides “pass/fail”
only
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
COMLEX-USA Scoring and Reporting, continued
• Score Reporting (2-digit standard scores*, 3-digit
standard scores, percentile scores and pass/fail
determinations) *phasing out 2-digit standard scores in
2015
• Percentile conversion tool available on website,
linked to ERAS score reports (to residency
programs), and smart phone APP available!
www.nbome.org
© 2014 NBOME
NBOME Update For ACGME PDs
Sample Percentile Score Conversions*
(Calculator on NBOME website, App)
COMLEX-USA Level 1 (April 2008)
2 digit
• 75
• 78
• 82
• 85
• 89
• 92
• 96
3 digit
400
450
500
550
600
650
700
Percentile
10
28
51
73
89
96
99
*vary by testing cycle
www.nbome.org
© 2014 NBOME
COMLEX-USA Level 1 Pass Rates
Test Cycle
(1st
N
Timer)
Passing Rate
(1st Timer)
COMLEX-USA Level 1
2011-2012
4,853
91.9%
2012-2013
5,056
90.6%
2013-2014
5,448
92.0%
www.nbome.org
© 2014 NBOME
COMLEX-USA Level 2-CE Pass Rates
Test Cycle
(1st
N
Timer)
Passing Rate
(1st Timer)
COMLEX-USA Level 2-CE
2011-2012
4,569
86.8%
2012-2013
4,866
89.4%
2013-2014
4,973
92.3%
www.nbome.org
© 2014 NBOME
COMLEX-USA Level 2-PE Pass Rates
Examination Outcome
2011-2012
N = 4749
2012-2013
N = 5127
2013-2014
N=5541
Pass
95.0%
94.1%
92.6%
Fail
5.0%
5.9%
7.4%
Fail Both Domains
0.4%
0.3%
0.7%
Fail Humanistic Domain
3.3
3.9
3.3%
Fail Biomedical/
Biomechanical Domain
1.3
1.7
3.5%
www.nbome.org
© 2014 NBOME
COMLEX-USA Level 3 Pass Rates
Test Cycle
(1st
N
Timer)
Passing Rate
(1st Timer)
COMLEX-USA Level 3
2011-2012
4,357
95.0%
2012-2013
4,118
95.1%
4,807
96.0%
2013-2014
www.nbome.org
© 2014 NBOME
News About Residency Program Directors and COMLEX-USA
NRMP Program Director Survey 2014 Shows 77
percent of ACGME Residency Program Directors
Use COMLEX-USA for DOs (same as 2012 PD
Survey)…
www.nrmp.org
www.nbome.org
© 2014 NBOME
NRMP ACGME Program Director Survey 2014
Shed Light on Factors PDs Use…
• Board Examination Scores (USMLE Step 1, COMLEXUSA Level 1) are used heavily (#1) in selecting for
interviews 94% (4.1)
• # 6: US Allopathic School graduates- 75% (4.1- V.
important)- don’t feel bad, 100% of AOA-accredited
PDs currently only accept DO applicants
• Majority-71% report typically interviewing and
ranking DO applicants- Congratulations!
• Varies by specialty, other factors
http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdf
www.nbome.org
© 2014 NBOME
COMLEX-USA and the Evidence
Score Concordance Studies
•
•
•
•
•
From the Residency Program Director Perspective, acknowledging that
these are different tests, it may be favorable to know how COMLEXUSA and USMLE scores relate to each other for use with DO applicants.
This is relevant only to the secondary use with residency program
requirements and the new ACGME Common Program Requirements for
Fellowships.
Somewhat analogous situation to the ACT and SAT tests for college
admissions. The College Board and ACT acknowledged that these are
different tests with distinct constructs and test specifications, and
that performance on one cannot predict precisely the performance on
the other, but can get an estimation.
Now that about 50% of DO students elect to take one “step” of
USMLE, in addition to the COMLEX-USA, it may be possible to better
provide an estimated comparison.
High correlations (0.85) have been demonstrated, published, and
replicated. New, soon to be published, more robust studies (multischool, variables adjusted for) also show higher correlations with
COMLEX-USA and undergraduate science and overall GPA.
www.nbome.org
© 2014 NBOME
COMLEX-USA and the Evidence
Score Concordance Studies, continued
• Lee AS, Chang L, Feng E, Helf S. Reliability and
Validity of Conversion Formulas Between
Comprehensive Osteopathic Medical Licensing
Examination of the United States Level 1 and United
States Medical Licensing Examination Step 1. Journal
of Graduate Medical Education: June 2014, Vol 6., No.
2, pp. 280-283.
• Schenarts PJ, Termuhlen PM, Pasley J, Rose JS,
Friedell ML. A primer on how to select osteopathic
applicants to an allopathic general surgery residency.
Journal of Surgical Education. 2011;68(3):239-245.
www.nbome.org
© 2014 NBOME
COMLEX-USA and the Evidence
Score Concordance Studies, continued…
• Chick DA, Friedman HP, Young VB, Solomon D.
Relationship between COMLEX and USMLE scores
among osteopathic medical students who take
both examinations. Teach Learn Med. 2010;
22(1):3-7. doi: 10.1080/10401330903445422.
• Slocum PC, Louder, JS. How to predict USMLE
scores from COMLEX-USA scores: a guide for
directors of ACGME-accredited residency
programs. J Am Osteopath Assoc. 2006;
106(9):568-9.
www.nbome.org
© 2014 NBOME
COMLEX-USA and the Evidence
Predictive Validity Studies
• Pierce DL, Mirre-Gonzalez MA, Carter MA, Nug D,
Salamanca Y. Performance on COMLEX-USA Exams
Predicts Performance on EM Residency In-training
Exams. Academic Emergency Medicine. 2013; 20 (5)
Suppl1.
• Li F, Gimpel JR, Arenson E, Song H, Bates BP, Ludwin
F. The relationship between COMLEX-USA scores and
performance on the American Osteopathic Board of
Emergency Medicine Part I Certifying Examination. J
Am Osteopath Assoc., 2014.
www.nbome.org
© 2014 NBOME
COMLEX-USA and the Evidence
Predictive Validity Studies, continued…
• Cavalieri TA, Shen L, Slick GL. Predictive validity of
Osteopathic Medical Licensing Examinations for
osteopathic medical knowledge measured by graduate
written examinations. J Am Osteopath Assoc. 2003;
103(7): 337-342.
• Langenau EE, Pugliano G, Roberts W, Hostoffer R.
Summary of ACOP program directors’ annual reports
for first-year residents and relationships between
resident competency performance ratings and
COMLEX-USA test scores. Electronic Journal of the
American College of Osteopathic Pediatricians. 2010;
2(7).
www.nbome.org
© 2014 NBOME
COMLEX-USA and the Evidence
Predictive Validity Studies, continued…
• Sevensma SC, Navarre G, Richards R. COMLEXUSA and in-service examination scores: Tools for
evaluating medical knowledge among residents. J
Am Osteopath Assoc. 2008: 108.713-716.
• Langenau EE, Pugliano G, Roberts W. Relationship
between high-stakes clinical skills exam scores
and program director global competency ratings
of first-year pediatric residents. Medical
Education Online. 2011, 16: 7362-DOI:
10.3402/mer.v16i0.7362.
www.nbome.org
© 2014 NBOME
Tools for Residency Program Directors
• Website, “What PDs Should Know”; Annual Report; FACEBOOK and
Twitter
• ACGME Annual Meeting Exhibit, Presentation at New PD Orientation
2015
• Research: presentations and publications, e.g., Program Directors
Meetings such as CORD (EM), ACP (IM), etc.
• ERAS links regarding interpretation of COMLEX-USA scores
• COMLEX-USA Percentile Conversion Tool (PCT)
• Smart Phone App includes PCT
• Grass-roots efforts- You can help!
• JGME Full-page Ads and publications
www.nbome.org
© 2014 NBOME
Journal of Graduate Medical Education June 2014
“The Allopathic Program Director’s Dilemma”
“By developing a method to correlate COMLEX Level 1 to
USMLE Step 1, program directors of both osteopathic and
allopathic residencies would have the ability to evaluate the
full applicant pool on at least 1 metric.”
“In contrast to USMLE scores, which are difficult to interpret
and for which there is no published mean, COMLEX scores are
easy to interpret.”
Paul J. Schenarts, MD, FACS
JGME, June 2014
www.nbome.org
© 2014 NBOME
NBOME Update
Comprehensive Osteopathic Medical Self-Assessment Examination
Phase 1, Phase 2 and Phase 3 have
distinct test forms
(Faculty Edition available)
Visit: www.nbome.org and click on
COMSAE in the left navigation
www.nbome.org
© 2014 NBOME
Resident In-Service Examinations
Osteopathically Distinctive Assessment of DO
Residents- NBOME Partners include:
•
•
•
•
•
ACOFP- Family Medicine
ACOP- Pediatrics
ACOEP- Emergency Medicine
ACOS- Surgery
ACOI- Internal Medicine
www.nbome.org
© 2014 NBOME
The National Board of Osteopathic Medical Examiners
Visit or Contact Us:
jgimpel@nbome.org
www.nbome.org
Phone: 610-825-6551 (X304)
THANK YOU!
www.nbome.org
© 2014 NBOME
www.nbome.org
© 2014 NBOME