A Program Director’s Guide to the
Medical Student Performance Evaluation
(Former Dean’s Letter) With a Database
James B. Naidich, MDa, Gregory M. Grimaldi, MDa, Pamela Lombardi, MDa,
Lawrence P. Davis, MDa, Jason J. Naidich, MDa
Purpose: The value of the Medical Student Performance Evaluation (MSPE) for a program director is in the
information it contains comparing how a student performed in medical school relative to his or her classmates.
The Association of American Medical Colleges has recommended that a student’s class ranking be included in
the summary paragraph of the MSPE and that this information be repeated in a supplementary appendix.
Methods: The authors reviewed the MSPEs from 1,479 applications for residency training positions. The
aim was to determine to what extent and in what manner individual schools reveal how their students perform
relative to their peers. The authors then set out to create a database containing this information.
Results: Working from a list of 141 US members of the Association of American Medical Colleges,
complete information for 107 schools (76%) and partial information for the remaining 34 schools (24%) was
gathered. Only 12 schools (9%) included complete comparative information in the summary section in
accordance with the guidelines of the Association of American Medical Colleges. Other schools were in partial
compliance or did not comply at all. The database the authors constructed will inform users if comparative
information is available, guide users to its location in the MSPE, and explain the meaning of the language
different schools use to rank or classify their students.
Conclusions: The authors recognize that this database is incomplete and that the individual institutions
will alter their ranking system from time to time. But this database is offered in an open format so that it can
be continuously updated by users.
Key Words: Dean’s letter, MSPE, medical student performance, resident applications, student evaluations
J Am Coll Radiol 2014;11:611-615. Copyright © 2014 American College of Radiology
INTRODUCTION
The Medical Student Performance Evaluation (MSPE),
formerly the dean’s letter, has the potential to provide
valuable information, not available elsewhere, concerning how a medical student has performed relative to his
or her classmates [1]. This information is of particular
importance to program directors trying to decide which
applicants to invite for interviews. But despite the
importance of this information, as the Association of
American Medical Colleges (AAMC) wrote in its 2002
a
North Shore-LIJ Health System, Hofstra North Shore-LIJ School of Medicine at North Shore University, Department of Radiology, Manhasset, New
York.
Corresponding author and reprints: James B. Naidich, MD, North Shore-LIJ
Health System, Hofstra North Shore-LIJ School of Medicine at North Shore
University, Department of Radiology, 300 Community Drive, Manhasset,
NY 11030; e-mail: jnaidich@yahoo.com.
ª 2014 American College of Radiology
1546-1440/14/$36.00 http://dx.doi.org/10.1016/j.jacr.2013.11.012
Guide to the Preparation of the Medical Student Performance Evaluation [2], “A common recurrent complaint
of those who interpret deans’ letters of evaluation is that
too often it is impossible to estimate how a candidate
performed in comparison to his or her peers.” This is a
problem with a long history.
In 1989, Wagoner and Suriano [3] wrote a short
paper titled “A New Approach to Standardizing the
Dean’s Letter.” Their paper pointed out that the inconsistencies among different medical schools reduced
the value of the dean’s letter and that residency program
directors would benefit from a concise and comprehensive form that was consistent from school to school.
That same year, the AAMC first published guidelines
recommending that the dean’s letter contain information
that would allow readers to understand each medical
student’s performance compared with his or her peers
[4]. In 1993, Hunt et al [4] published a paper titled
611
612 Journal of the American College of Radiology/Vol. 11 No. 6 June 2014
“Characteristics of Dean’s Letters in 1981 and 1992.”
They reported that 45% of medical schools were not
compliant with the 1989 AAMC guidelines. The authors
warned that failure to provide comparative information
would diminish the value of the dean’s letter, with the
result that program directors would place too much
emphasis on “simple-minded numerical scores from
licensure examinations.” In a follow-up paper written in
2001, Hunt et al [5] reported some improvement, but
35% of schools were still noncompliant.
In 2002, the AAMC [2] again addressed the issue.
First, they renamed the dean’s letter the MSPE. Then the
AAMC specifically recommended that the MSPE
contain a summary section and that the summary section
include a student’s comparative performance in medical
school relative to his or her peers. The AAMC further
recommended that the summary section define the
school-specific categories used in differentiating among
the levels of student performance. The AAMC also recommended that the MSPE contain a supplementary
appendix D containing a graphic representation of the
student’s overall performance relative to his or her
classmates, with numerically defined boundaries for the
individual medical school’s specific categories.
In 2007, in our first paper [6], we documented deans’
continued indifference to compliance with the AAMC
recommendations. We echoed the concern of Hunt et al
[4,5] that the lack of evaluation accuracy of the MSPE
resulted in de facto ceding of this task by the deans of
American medical schools to the United States Medical
Licensing Examination. The problem with this, we
thought, is that the licensing examination scores, however objective, are a thin statistic that offers no insight
into a student’s interpersonal or communication skills,
medical professionalism, or other domains beyond
medical knowledge. This kind of information can be
included in a well-written dean’s letter and is information not found elsewhere in the application.
Shea et al [1], in a paper published in 2008, shortly
after ours, reached many similar conclusions. Importantly, they found that the summary paragraph provided
comparative information in only 17% of cases, contrary
to the 2002 AAMC guidelines.
Most recently, in December 2012, Green et al [7]
wrote an op-ed piece in Virtual Mentor: American
Medical Association Journal of Ethics titled “Standardizing
and Improving the Content of the Dean’s Letter.” Note
how similar this title is to that used by Wagoner and
Suriano [3] 23 years earlier. Not only the title but the
issues discussed have remained the same.
Clearly little has changed in the intervening years.
Too often it remains difficult to judge a student’s performance from reading the MSPE. To try a new
approach to remedy the situation, we set out to develop
a database. Our purpose was to create a tool to enable
program directors to quickly extract the useful information from the MSPE.
METHODS
Institutional review board approval was obtained for
this project.
We reviewed the Electronic Residency Application
Service submissions of students from the US AAMCaccredited medical schools [8]. The submissions were
for students applying for graduate medical education
positions in diagnostic radiology at 3 institutions: North
Shore University Hospital (Manhasset, New York),
Northwestern University (Chicago, Illinois), and Long
Island Jewish Medical Center (New Hyde Park, New
York) for the 2012e2013 academic year. There were
239 applicants to the North Shore program, 544 to
Northwestern, and 224 to Long Island Jewish. We then
reviewed an additional 472 applications to the Northwestern program for the 2013e2014 academic year.
The MSPE accompanying each of these 1,479 applications formed the basis for our investigation.
We set out to create a comprehensive database. Our
intent was that a program director reading an application from an unfamiliar school might access our database and quickly learn what comparative information
the individual medical school offered and where in the
MSPE, whether in the summary paragraph, an appendix, or another location, this information was located.
Furthermore, when comparative information was not
included in the MSPE, the database would so inform the
user, to prevent fruitless searches.
The first column of the database is an alphabetized list
of the 141 US AAMC member schools [8], including
their full names and their city and state addresses, for
easy identification.
We next examined the summary paragraphs of the
MSPEs to determine what comparative data they
included. A rare summary paragraph would contain the
exact class ranking of a student. Somewhat more often,
the summary paragraph would contain a numerically
defined category, perhaps the student’s performance
quartile. Most often, the student’s performance was
categorized using a descriptor. The most frequently used
descriptor was excellent. Often, but not always, the
descriptor was used in a defined hierarchy. The most
commonly used hierarchy, in descending order, was
some variant of outstanding, excellent, very good, and
good. The schools would often assign numeric boundaries for each of the descriptor categories. Although the
descriptor was frequently included in the summary
paragraph, its definition was not. Sometimes this information was located in Appendix D in accordance
with the AAMC guidelines. Sometimes the information
was included in another location in the MSPE. Sometimes the schools used undefined descriptors. Sometimes
there were no comparative data whatsoever. Sometimes
the schools would inform readers that it was their
intent to offer no comparative data, and other times they
would not. Regardless, we searched the MSPEs for
whatever school-specific comparative data they might
Naidich et al/Medical Student Performance Evaluation 613
contain, and this information was included in a cell
next to the school’s name. In this manner we created
our database.
The design of our database is thus two columns wide.
The first column contains the alphabetized medical
school names and other identifying information. The
second column contains the corresponding data we
collected concerning the content of the MSPE (Table 1,
online only).
Once the database was constructed, it provided the
means to investigate a number of interesting questions:
1. We identified the number of schools from which we
had no data.
2. We identified the schools from which we had
incomplete data. These were schools for which we
thought that more examples (larger sample sizes)
might be helpful in better understanding the descriptors used in the summary paragraph.
3. We counted the number of schools for which we had
complete data. These were schools for which we were
confident we had a complete understanding of the
MSPE content.
4. We counted how many medical schools would inform
readers in the summary paragraph of their students’
comparative performance in order to compare this
number to the 17% statistic reported by Shea et al [1].
5. We looked at use of the descriptor excellent. We
counted how many schools used excellent in their
summary paragraphs. We looked for schools that
used excellent in hierarchies with defined numeric
boundaries. For schools meeting this criterion, we
then graphed the range of excellent for 2013 (Fig. 1)
and compared it with a similar graph reproduced
from our 2007 data (Fig. 2).
6. Finally, we looked at the descriptor outstanding. We
counted how often schools would use outstanding as a
category descriptor for something less than their best
performers.
Note that we knew from the outset that there would
be duplicate submissions for academic year 2012e2013
because the same student might easily apply to 2 or all 3
Fig 1. 2013 Medical Student Performance Evaluation range
of excellence data, 41 medical schools.
Fig 2. 2007 Medical Student Performance Evaluation range
of excellence data, 35 medical schools. Three schools
defined the upper but not the lower boundary (blurred) for
their descriptors of excellent.
programs. We made no effort to correct for this because
we thought that repetition would improve accuracy.
Subsequently, we did cross-check our lists to determine
the number of unique applicants.
RESULTS
The chief outcome of this project is the database
(Table 1). As planned, the database is only two columns
wide. It is simple to use. With the database open, a
program director or coordinator using the keyboard
search function can rapidly locate a medical school of
interest. The adjacent cell will detail all the comparative
information we were able to find and where these
comparative data are located in that school’s MSPE.
There were 141 medical schools listed as US members
of the AAMC [8]. We had data from all 141 schools. We
had incomplete data from 34 medical schools (24%).
These were schools that did not define their descriptors.
The amount of information we had from these schools
varied. At one end of the spectrum were schools from
which we had available enough examples to know some
but not all of the descriptors used in the summary
paragraph. For example, if one school described a student as very good and another student from the same
school as outstanding, most probably the second student
had performed better than the first. For other schools,
we had access to enough student applications to form
reasonable estimates of the school-specific category
hierarchies, but because the schools did not formally
define their hierarchies, we were not completely confident of our estimates. We think that with more examples of the MSPEs from these 34 schools, our database
could better describe how these schools stratify their
students’ levels of performance. Regardless of our
opinion, all we did was label the information cell in the
second column of the database “Incomplete Data.” We
then included in the cell whatever information we had
available concerning each school’s ranking system to
allow users to form their own opinions.
614 Journal of the American College of Radiology/Vol. 11 No. 6 June 2014
There were 107 medical schools (76%) for which our
data were complete. Of these 107, 13 medical schools
were explicit about not ranking their students’ performance. These schools offered no comparative information in their MSPEs. The database will readily identify
these schools. There were 88 medical schools that fully
described their ranking systems and used well-defined
category descriptors with numeric margins. Finally,
there were 8 new medical schools that have not yet
published MSPEs.
Because Shea et al [1] found that 17% of medical
schools provided comparative data in the summary
paragraph we looked at our own numbers. We found in
our survey that only 10 schools (7%) would routinely
include comparative information in the summary paragraph. Add the 2 other schools that placed this information next to but not in the summary paragraph, and
there were 12 schools (9%) in compliance with this
AAMC recommendation. Three additional schools
offered this information in the summary section, but
only for their better students.
Concerning the choice of summary paragraph descriptors, 77 medical schools (55%) used excellent as a
descriptor in the summary paragraph, of which 41
(29%) used excellent in a defined hierarchy with defined
numeric boundaries. Figure 1 graphically represents the
range of excellent used by 41 medical schools in 2013.
Figure 2 graphs the range of excellent for 35 medical
schools in 2007. Qualitatively, they are similar.
Twelve schools used outstanding as their second best
category, supplanted 3 times by exceptional, 3 times by
distinguished, twice by superior, and once each by most
outstanding, top student, superb, and recommend with
distinction.
Of the 1,479 applications we surveyed, 329 were
duplicated or triplicated because the same student applied
to more than one program. There were 1,150 unique
applications.
DISCUSSION
There is an inherent conflict of interest between medical
school deans trying to make their graduating medical
students look as attractive as possible to place them in
the best graduate medical education programs available
and residency program directors trying to recruit the
best performing medical school graduates for their
postgraduate programs. Probably this is the reason deans
are reluctant to reveal their students’ comparative performance in the summary paragraph or sometimes not
reveal it at all. Shea et al’s [1] finding of 17% compliance was based on a survey of students graduating in
2005. Eight years later, we find that compliance has
dropped to 9%. Deans may become even more hesitant
to provide easily accessible comparative data in the
future. The growth of new medical schools is exceeding
the expansion of residency training programs [9]. This
will make it increasingly difficult for medical schools to
place their graduates in the specialty training programs
of their choice.
Medical schools have adopted a number of approaches
to circumvent the AAMC recommendation that the
MSPE provide readily accessible data to program directors. One such strategy, used more frequently by the
more prestigious schools, is to not rank their students.
We found 13 schools (9%) adopting this strategy. Examples include Stanford University and Yale University.
Our database can rapidly identify these schools to save
readers the effort of searching for nonexistent data. The
database cannot, of course, offer insight into applicants’
performance at such medical schools.
The drawback of not ranking students is that
although it may veil a poorer student’s performance, at
the same time, it does not reveal who the better performing students are. To get around this difficulty, some
schools will include students’ performance only when it
is favorable. Examples include the University at Albany
and Northeast Ohio Medical University.
Albany uses 3 descriptors in the summary paragraph;
recommend highly for the top third of their students,
recommend without reservation for the middle third, and
recommend as a positive presence for the bottom third.
When a student is in the top third of the class, the dean
will indicate this in the summary paragraph. However,
the dean will not inform the reader in the summary
paragraph that recommend without reservation designates
the middle third or recommend as a positive presence the
bottom third of the class.
For most students from Northeast Ohio Medical
University, comparative information was not included
in the summary paragraph, an explanatory appendix, or
elsewhere in the MSPE. However, an exception was
made for one student. His standing in the top quartile of
the class was proclaimed in boldface type on the first
page of the MSPE.
A variant of this strategy is to omit from the summary
paragraph the student’s ranking when it is at the bottom
of the class. Examples are Stony Brook University and
Indiana University. The database identifies this tactic.
A puzzling strategy used by some schools is to deny
ranking their students even when they do use defined
categories with numeric margins. Two examples are the
University of Chicago and Georgetown University. The
database provides the descriptor categories used by these
schools and their numeric boundaries.
Many schools will use attractive descriptors in the
summary paragraph and either not define the descriptors
or, more commonly, define their meanings in some
location remote from the summary paragraph. The most
frequently used descriptor is excellent, a word with a
pleasant connotation and ordinary English-language
definitions including “exceptionally good” and “of superior merit” [6]. However, MSPE writers assign very
different meanings to this word. Figures 1 and 2 show
that no school used excellent to describe its very best
Naidich et al/Medical Student Performance Evaluation 615
students and also show how frequently excellent
was used to describe students in the bottom half of their
classes. Whereas the University of Chicago used
excellent to describe students ranking between the 5th
and 43rd percentiles, The Ohio State University used
the same descriptor for students in the 71st to 90th
percentiles, and Vanderbilt University used outstanding
to describe the top quarter of its class and excellent
to describe everyone else. Although an MSPE reader
will usually be correct in assuming that outstanding is a
category reserved for a school’s best performing students,
we found 13 examples (9%) for which this was not true.
A program director encountering a descriptor that is
undefined in the summary paragraph can reference the
database with the expectation that it usually will make
clear the school’s idiosyncratic use of the word.
One limitation of our database is that it is not complete. We have only partial data concerning 34 medical
schools. However, even for these 34 schools, usually
there is enough information for the database to
be helpful. On the other hand, for the remaining
107 schools, although our data are complete, there are
13 schools for which all the database will reveal is that
these schools elect not to offer comparison information
concerning their students’ performance. And there
are 8 new schools that have not yet begun to write letters. Despite these limitations, we think this is a good
start.
Another concern is avoidable errors. There is just too
much information included in this database for it not to
contain some errors. To some extent, our methodology
worked in our favor to reduce errors. We reviewed 1,497
MSPEs from 3 different programs. This ensured that
there would be multiple different student applications
from the same medical school. Furthermore, we were
unconcerned that for academic year 2012e2013, the
same student might easily apply to 2 or all 3 programs.
We welcomed the repetition. We thought the more
applications we reviewed, the fewer would be our errors.
In all, we had 1,150 unique applications.
A third limitation is that what had been true for
2012e2013 may no longer be true for the future. Deans
may change their policies from year to year. New schools
will begin submitting Electronic Residency Application
Service applications. This database might best be looked
upon as a work in progress requiring continuous
updating. However, if it proves as useful a tool as we
hope, as better data become available, users can fill in
the gaps and correct the mistakes.
If the AAMC, medical school deans, and program
directors were ever able to reach a consensus concerning
the use of well-defined category descriptors with numeric
boundaries, this would enhance the value of the MSPE.
Ideally, a student’s class ranking should be based
not only on grades but also on nonclinical categories
of achievement, including professionalism, research, and
community service. The AAMC’s mission [10] does
include “supporting the entire spectrum of education,
research and patient care activities.” Activities that
contribute to this goal, we think, deserve more credit in
evaluating a student’s performance.
However, for the interim, this database is our solution
to medical school deans’ reluctance to provide easily
understandable comparative information concerning the
performance of their graduating students. The database
should speed users’ ability to locate relevant information
within an MSPE and to understand an applicant’s performance relative to his or her peers.
TAKE-HOME POINTS
The summary paragraph of an MSPE should describe
a student’s performance in medical school relative to
that of his or her peers.
Most often, this comparative information is not
included in the summary paragraph.
Too often, this information is not included anywhere
in the MSPE.
Accessing the attached database will describe what
information is made available from different medical
schools.
SUPPLEMENTARY DATA
Supplementary data can be found online at: http://dx.
doi.org/10.1016/j.jacr.2013.11.012.
REFERENCES
1. Shea JA, O’Grady E, Morrison G, Wagner BR, Morris JB. Medical
Student Performance Evaluation in 2005: an improvement over the
former dean’s letter. Acad Med 2008;83:284-91.
2. Association of American Medical Colleges. A guide to the preparation of
the medical student performance evaluation. Report of the Dean’s Letter
Advisory Committee. Washington: District of Columbia: Association of
American Medical Colleges; 2002.
3. Wagoner NE, Suriano JR. A new approach to standardizing the dean’s
letter. Acad Med 1989;64:688-9.
4. Hunt DD, MacLarean CF, Scott CS, et al. Characteristics of dean’s
letters in 1981 and 1992. Acad Med 1993;68:905-11.
5. Hunt DD, MacLarean CF, Scott CS, et al. A follow-up study of the
characteristics of dean’s letters. Acad Med 2001;76:727-33.
6. Naidich JB, Lee JY, Hansen EC, Smith LG. The meaning of excellence.
Acad Radiol 2007;14:1121-6.
7. Green MM, Sanguino SM, Thomas JX. Standardizing and improving
the content of the dean’s letter. Virtual Mentor 2012;14:1021-6.
8. Association of American Medical Colleges. Member directory search
result—medical school. Available at: https://members.aamc.org/eweb/
DynamicPage.aspx?site¼AAMC&webcode¼AAMCOrgSearchResult&
orgtype¼Medical%20School. Accessed December 9, 2013.
9. Iglehart JK. The residency mismatch. N Engl J Med 2013;369:297-9.
10. Association of American Medical Colleges. About the AAMC. Available
at: https://www.aamc.org/about/. Accessed December 9, 2013.