Rethinking the Educator Portfolio: A criteria-based model
Running title: A criteria-based educator portfolio
Authors:
Kanade Shinkai
Brian Schwartz
Cynthia Ashe
Helen Loeser
David Irby
University of California, San Francisco School of Medicine
Office of Medical Education
533 Parnassus Avenue, U-80
San Francisco, CA 94143-0710
Corresponding author: [decide]
Address
Phone
Fax
E-mail:
Article type: How we (implemented EP2.0 here at UCSF)
1
ABSTRACT
The educator portfolio makes visible the educational contributions of faculty
members for the purpose of peer review. The existing model of the educator
portfolio was cumbersome, time consuming to create and update, lacked a
structured framework and specific criteria for evaluation, and was infrequently
integrated into the academic promotion process. We describe the development of a
succinct, criteria-based model and its implementation in several settings. The
model consists of two parts: a one-page executive summary followed by detailed
descriptions of selected educator roles. The executive summary describes up to five
of the educator’s most significant, recent contributions to teaching and education,
categorized by educator roles (teacher, mentor, curriculum developer, educational
leader and learner assessor). The one- to two-page detailed role descriptions of key
contributions highlighted in the executive summary comprise the second
component. Both parts of the portfolio utilize criteria-based templates emphasizing
the evaluation of educational scholarship. This new portfolio model was initially
implemented at UCSF as the basis for application to the Academy of Medical
Educators and subsequently integrated into the campus-wide advancement process
for academic promotion. This model offers a practical format and mechanism for
faculty members to display their important scholarly contributions to the education
mission.
2
INTRODUCTION
Teachers in the health professions often have difficulty making a clear case for their
academic promotions as educators. Not only are the roles of teachers, educational
leaders and innovators challenging to describe but the criteria for judging those
contributions are unclear. Universities also struggle with how to define, document
and evaluate teaching . (Beasley, 1997) Scholarship is highly valued in universities
and therefore teachers/educators must make visible their contributions for peer
review. Providing documentation of teaching excellence and educational scholarship
is essential yet difficult to achieve. A succinct, criteria-based mechanism to
document faculty members’ contributions to the educational mission and to the
scholarship of teaching and learning is needed.
Over the past several decades, educational scholars have progressively refined
definitions of scholarship, educator roles, and criteria for judging those roles.
Expanded definitions of scholarship have been advocated along with guidelines for
evaluating teaching. (Boyer, 1990; Glassick 1997 and 2000) This has included
broader definitions of excellence in a variety of educator roles (Fincher, 2000) as
well as rigorous evaluation of these roles using well-established rubrics for
evaluating discovery research. (Beattie, 2000) Each educator role (teacher, mentor,
curriculum developer, educational leader and learner assessor) can now be
described and evaluated using clearly defined criteria. (Kirkpatrick, 2006; Simpson
2007) Recently published criteria for evaluating educational scholarship (Gusic,
3
2013; Gusic, 2014) provide a framework for the documentation of teaching
excellence.
Analogous to an artist portfolio, the educator portfolio should serve as a
representation of an educator’s best work. (Niebuhr V, 2013) The portfolio makes
visible these contributions for peer review. Typical components included teaching
philosophy, evidence of faculty development, teaching evaluations and other
measures of competence, description of educational leadership roles, enduring
teaching materials, and educational scholarship. Guidelines for writing and
evaluating portfolios have varied greatly.
Academies of medical educators have emerged as a way to recognize distinguished
teachers and improve the quality of teaching (Irby, 2004). Academies frequently use
educator portfolios to select new members and have advocated for institutional
adoption of educator portfolios as required components of educators’ promotion
documentation.
Unfortunately, the current model of the educator portfolio was highly variable, often
lengthy, time-consuming to create and update, did not always highlight the
educator’s most recent work, and lacked a structured framework and specific
criteria for evaluation. Distinctions were not clearly drawn between teaching
(quantity and quality), scholarly teaching (building on the work of others) and
scholarship of teaching and learning (advancing and disseminating knowledge that
4
others can build on). (Simpson 2007) Since these portfolios were prepared as
supplements to the entire CV, they were often included as optional attachments and
rarely integrated into academic promotion materials. The separation of the CV and
educator portfolio resulted in the perception that educators’ accomplishments were
of lesser value than discovery research and grant funding. Thus, we recognized the
need for a redesigned educator portfolio that would facilitate rigorous peer review
and academic promotion. In this article, we describe the development of a new
format for the educator portfolio and how we are using it for application to the
Academy of Medical Educators and for academic advancement at the University of
California, San Francisco (UCSF).
NEW PORTFOLIO DEVELOPMENT
Since the formation of the UCSF Academy of Medical Educators in 2001, candidates
have prepared and submitted an educator portfolio as part of the application
process. The portfolio consisted of a personal statement (describing educational
philosophy, professional development in education, and aspirations for contributing
to the academy, school and campus) and a summary of teaching activities. The
summary of teaching activities described evidence of the educator’s excellence in
direct teaching and at least one of the following activities: advising and mentoring,
curriculum development and instructional design, educational administration and
leadership, and educational research.
5
For over a decade, Academy efforts to advocate for inclusion of the educator
portfolio in the academic promotion process at UCSF were unsuccessful, due in part
to the length of the portfolios and the lack of clear criteria for evaluating them. In
2013, the Academy appointed a working group to reassess the portfolio for
Academy membership and for academic promotions, and make recommendations
for its redesign. Drawing on the work of the AAMC Toolbox for Evaluating
Educators, (Gusic 2013, MedEdPORTAL #9313) the working group examined
examples of members’ application portfolios to determine if the existing portfolio
provided the information necessary to document excellence using the Toolbox
criteria. The clear answer was that the existing portfolio did not provide the right
information in a concise format. The working group also reviewed the literature on
educator portfolios and concluded that a new model was needed.
In the process of designing a new structure to the portfolio, working group
members reflected on the UCSF CV requirement for documenting research
contributions. The CV has two sections entitled, “Research Program” and
“Significant Publications” that are narrative fields describing one’s research
activities and most significant publications. The former encourages the faculty
member to describe their current area(s) of research and its significance, and the
latter instructs the individual to select the most important recent publications,
detail their contributions to those publications and describe the publication’s
importance. The working group envisioned a similar structure for
teaching/educational contributions: an executive summary for the most important
6
educational accomplishments, and detailed descriptions of specific contributions
mentioned in the executive summary. The working group next adapted the specific
criteria for evaluating educational scholarship (Gusic 2013) for each of the five
specific educator roles (teaching, mentoring, curriculum development, educational
leadership and learner assessment).
Together, this vision resulted in the redesign of a portfolio documenting an
individual’s accomplishment as an educator comprised of two components: an
executive summary listing up to five key, recent contributions to education, and
detailed role descriptions of one to three of those contributions. The one-page
executive summary (Figure 1) opens with a statement of time allocation to educator
roles, key changes in educator roles in the past three years, and a list of up to five
important contributions to education. Each contribution describes evidence for
excellence in a few sentences. The detailed role descriptions (Figure 2) provide
more granular information on the specific educational activity, emphasizing
scholarly approach, clearly defined goals, appropriate methods and assessment as
well as evidence of impact and dissemination. Specific templates for detailed role
descriptions for each educator role and instructions for template completion were
created (insert Hyperlink here to AME website or cite supplemental figures).
Importantly, the new model emphasizes recent accomplishments, aligning with the
UCSF advancement process that requires review of faculty achievements every two
years at assistant and associate professor levels and every three years at the full
7
professor level. The new model also provides templates for all the criteria salient for
its evaluation. The critical elements of the new model include definition of a
streamlined, contemporary format and provision of clear standards for
documentation and evaluation, derived from the AAMC Toolbox for Evaluating
Educators (Table 1).
To engage members of the Academy of Medical Educators in the revision process
and to provide proof of concept, the working group evaluated a group of portfolios
from a previous round of Academy applicants using the new criteria and found the
prior applications insufficient. We then asked members of the working group to
complete the new portfolio templates in order to have real examples to share with
the Academy. Academy members enthusiastically received the overall concept and
suggestions for improving clarity were incorporated. Academy applicants were
invited to submit their applications in the new model in 2014. Applicants found the
new template easier to complete and also to incorporate into the academic
advancement process. They lauded the clarity of directions and appreciated the
transparency of portfolio evaluation. Applicants also benefited from multiple
examples of completed portfolios and were encouraged to meet with “portfolio
coaches”, who helped applicants select which roles to emphasize and what
documentation to include. Finally, the new format provided superior information
for the membership committee to facilitate peer review and render decisions on
membership.
8
PORTFOLIO INCLUSION IN THE UNIVERSITY ADVANCEMENT PROCESS
In 2015, the campus convened a faculty committee to review the existing on-line CV
template and to recommend changes. Members of the Academy educator portfolio
working group served on that committee and advocated for inclusion of the new
educator portfolio model into the university CV template. Central to the inclusion of
the new templates was the understanding that assessment of educator excellence is
highly relevant to particular types of promotions (where educator roles are a major
academic activity) and where improved, rigorous evaluation of those roles was
needed. The committee agreed that faculty members whose major contribution is to
teaching faced bias in the existing CV structure due to an emphasis on discovery
oriented research publications and the absence of any structured format to
document excellence in education. In the existing promotion system, the educator
portfolio was accepted as “supplemental information” and appended as an
electronic attachment; this material was not necessarily made available to those
writing letters of support. This meant that promotion committee review of the
portfolio was variable, and further limited by a lack of specific criteria for judgment.
The proposal to include the new portfolio model within the CV template itself,
making it available to all CV reviewers, would enable uniform, criteria-based
advancement, and also provide evidence of impact through dissemination of work
analogous to publications.
9
During the UCSF committee deliberations, a key barrier was identified to the
inclusion of the educator portfolio: the lack of specific criteria to rigorously evaluate
educator contributions and tools for assessing the impact of such contributions.
Committee members agreed that peer-review is analogous to the standard
publication process of discovery research. The length of the CV was also identified
as an additional barrier. These concerns were mitigated by the succinct format of
the new executive summary, by the limited number of detailed descriptions and
criteria for their evaluation, and by offering the option to replace narrative sections
of the existing CV (e.g., Teaching narrative, mentoring narrative) with the new
templates. Acknowledging that the new template would elevate the scholarship of
educators on campus through transparent criteria for promotion, the committee
recommended inclusion of the new EP templates directly in the university CV, with
instructions embedded into the on-line platform encouraging only faculty with
educator roles to utilize these optional formats.
REFLECTIONS
Clear and concise documentation of excellence in educator activities is vital to
academic promotion of educators; the creation and adaptation of a new educator
portfolio format facilitates criteria-based evaluation of educational scholarship
towards this end. We designed a streamlined two-part model for the educator
portfolio that successfully addresses the major shortcomings of the traditional
portfolio, which were high variability; length; work and time to create and update;
10
lack of a structured framework with specific criteria for evaluation; and its exclusion
from the academic promotion process. Specifically, the new portfolio has a clear
structure and transparent set of criteria, constrains narrative to essential and
concise descriptions of accomplishments, and is less time-consuming to complete –
although any portfolio, like a good CV, takes time to create and maintain. Finally, the
new templates have been incorporated into the academic advancement process at
UCSF, which offers educators the opportunity to display their important scholarly
contributions to the educational mission of the university, provides faculty
members and promotion committees with rigorous criteria for evaluating those
contributions and makes this information available to all reviewers.
Importantly, the new format offers a framework for educators to learn how to best
achieve and demonstrate educational scholarship and serves as a tool for
mentorship and career planning. Faculty development will be required for those
reviewing the new EP format as mentors, reviewers of Academy membership
applications, members of academic promotion committees and referee letter
writers.
The new format for the educator portfolio builds on important recent work
expanding the definition of scholarship in education and improving understanding
of specific criteria used for evaluating the scholarly contributions of educators. The
structured templates also help to make visible and elevate the importance of the five
educator roles in the minds of the academic community.
Capsule summary
11
• We describe a succinct 2-part educator portfolio format that enables criteria-based
evaluation of educational scholarship based on a previously published framework.
• The new format was recently implemented as basis for selection into the Academy
of Medical Educators with positive evaluations.
• The new educator portfolio was also recently incorporated into our universitywide CV template for academic promotion.
12
Glossary term: Educator Portfolio
• Niebuhr V et al. 2013. Educator Portfolios. MedEdPORTAL Publications.
https://www.mededportal.org/publication/9355
13
Figures
• Figure 1: Executive Summary of Most Significant Contributions to Teaching and
Education (completed example)
• Figure 2: Detailed Descriptions [completed example – direct teaching]
• Figure 3 (supplemental): blank Executive Summary
• Figure 4 (supplemental): blank Direct Teaching Detailed Description
• Figure 5 (supplemental): blank Mentoring Detailed Description
• Figure 6 (supplemental): blank Educational Leadership Detailed Description
• Figure 7 (supplemental): blank Curriculum Development Detailed Description
• Figure 8 (supplemental): blank Learner Assessment Detailed Description
• Figure 9 (supplemental): instructions for new educator portfolio completion
Table
• Table 1: General Criteria for Evaluating the Scholarly Contributions of Educators,
adapted from the AAMC Toolbox for Evaluating Educators
14
Notes on contributors (50 words max/ person)
Kanade Shinkai, MD is associate professor of Dermatology, at the University of
California, San Francisco.
David M. Irby, PhD is professor of Medicine and member of Research and
Development in Medical Education at University of California, San Francisco
15
Acknowledgments:
• Maxine Papadakis
• Karen Hauer
• Sharad Jain, Rebecca Jackson and members of the UCSF Haile T. Debas Academy of
Medical Educators working group on advocacy
• Brian Alldredge and the UCSF MyAdvance Task Force
Declaration of Interests: The authors report no declaration of interests.
16
REFERENCES
Beasley BW et al. 1997. Promotion criteria for clinician-educators in the United
States and Canada: a survey of promotion committee chairpersons. JAMA 278:723728.
Beattie DS. 2000. Expanding the view of scholarship: introduction. Acad Med
75:871-876.
Boyer EL. 1990. Scholarship Reconsidered: Priorities of the Professoriate. Princeton,
NJ: Carnegie Fondation for the Advancement of Teaching.
Fincher RM et al. 2000. Scholarship in Teaching: An Imperative for the 21st Century.
Acad Med 75:887-894.
Glassic CE, Huber MR, Maeroff GI. 1997. Scholarship Assessed – Evaluation of the
Professoriate. San Francisco, CA: Jossey-Bass.
Glassick CE. 2000. Boyer’s expanded definitions of scholarship, the standards for
assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med
75:877-880.
Gusic M, Amiel J, Baldwin C, Chandran L, Fincher R, Mavis B, O'Sullivan P, Padmore J,
Rose S, Simpson D, Strobel H, Timm C, Viggiano T. 2013. Using the AAMC Toolbox for
Evaluating Educators: You be the Judge!. MedEdPORTAL Publications.
https://www.mededportal.org/publication/9313
Gusic ME, Baldwin CD, Chandran L, Rose, S, Simpson D, Strobel HW, Timm C, Finger
RM. 2014. Evaluating educators using a novel toolbox: applying rigorous criteria
flexibly across institutions. Acad Med 89:1006-1011.
Irby DM, Cooke M, Lowenstein D, Richards B. 2004. The academy movement: a
structural approach to reinvigorating the educational mission. Acad Med 79:729736.
Kirkpatrick, DL, & Kirkpatrick, JD. 2006. Evaluating Training Programs: The Four
Levels (3rd Ed). San Francisco, CA: Berrett-Koehler Publishers.
Niebuhr V et al. 2013. Educator Portfolios. MedEdPORTAL Publications.
https://www.mededportal.org/publication/9355
Simpson D et al. 2007. Advancing educators and education by defining the
components and evidence associated with educational scholarship. Med Ed
41:1002-1009.
17