1
Evolution of an RSS case conference reporting system for documenting
competency that leads to performance change.
Tina Kehoe, M. A. Ed., Education Research Coordinator, Office of Continuing Medical
Education, Medical University of South Carolina
Elizabeth Gossen, B.S., RSS Program Coordinator, Office of Continuing Medical
Education, Medical University of South Carolina
Odessa Ussery, M. Ed., Director, Office of Continuing Medical Education,
Medical University of South Carolina
Correspondence:
Tina Kehoe, M. A. Ed.
MUSC Office of CME
19 Hagood Ave, Ste 1004 MSC 754
Charleston, SC 29425
Tel: 843-876-1926
Fax: 843-876-1931
E-mail: kehoet@musc.edu
2
Abstract
Limited data is available to support the assumption that traditional medical school
case conferences and lecture series result in real performance changes. Using
guidance from published literature, we developed a qualitative data collection process
for identifying competency within the case conference, which also led to changes in
practice performance that could be objectively documented through the electronic
medical record.
Collaboration among RSS activity directors and relevant stakeholders was
necessary to adapt our methodology for implementation into the clinical teaching
environment. Group training and one-on-one training, which included real-time
demonstrations during the case conference, were necessary to achieve competence
and compliance among stakeholders.
We used four data capture methods: 1) A new Documentation and Verification of
Change in Practice form, 2) a web-based survey to subjectively assess applied learning
among participants, 3) one-to-one follow-up with the RSS activity director and
coordinator to track progress of related systems-based changes and/or, 4) EMR chart
audits to track case-specific practice changes.
After several modifications, our new reporting system has resulted in the capture
of previously undocumented data. We found it to be most effective for treatment
planning conferences such as tumor boards where changes are common and readily
implemented into practice. This process benefits the RSS’ clinical department(s) by
showcasing the importance and impact of the RSS case conference on clinical practice
and provides a snapshot of the successful translation of academia to the bedside.
3
Evolution of an RSS case conference reporting system for documenting
competency that leads to performance change.
Limited data is available to support the assumption that traditional medical school
case conferences and lecture series result in real performance change, and those that
do report changes in performance struggle to indicate a resulting benefit to patients that
is directly related to the CME activity1-10. There are different reasons for this, one being
the varying types of learning environments within the academic medical center; a grand
rounds format is different from a case conference format and the learning goals are
typically different. Also, patient outcomes may be the result of a complex care process
making attribution to one specific intervention difficult10.
In the reviewed literature, the methodology used to document performance
changes were either complicated or non-existent. Some studies focused on process
and function within the activity versus following up on the learning outcome5,10, while
others acknowledged a lack of accountability of the activity itself in verifying
implementation of recommended changes4,7. Recognizing a need to establish a
documentation system for our own institution, we chose to focus on developing a
method that could be uniformly used for all variations of the hospital case conference:
tumor board, M&M, treatment planning, and interesting cases. Using guidance from
published literature11-13, we developed a qualitative data collection process for
identifying competency within the case conference, which also led to changes in
practice performance that could be objectively documented through the electronic
medical record.
4
METHODS
Collaboration among RSS activity directors, RSS coordinators, and other
relevant stakeholders was necessary to adapt our methodology for implementation into
the clinical teaching environment.
We used four data capture methods in an effort to triangulate results:
1. A new Documentation and Verification of Change in Practice form
(Appendix A, B, C, D), which is required of all CME-certified RSS case
conferences,
2. A web-based survey to subjectively assess applied learning among
participants (Appendix D, Fig. 1),
3. One-to-one follow-up with the RSS activity director and coordinator to
track progress of related systems-based changes (Appendix B) and/or,
4. EMR chart audits to track case-specific practice changes (Appendix A).
Training was necessary to achieve competence and compliance among RSS
coordinators and activity directors with completing the new Documentation and
Verification of Change in Practice form. Group training in a traditional classroom
environment was conducted twice to allow activity directors and coordinators to select a
day and time that better accommodated their busy schedules. Subsequent one-on-one
training, which included real-time demonstrations during the case conference, was
found to be critical toward helping trainees successfully apply what they learned.
Instructions and case examples were added to the back of the reporting form to further
assist in this effort.
5
Completed reporting forms were signed† by the activity director before being sent
to the Office of CME at the designated time. Forms were reviewed by the education
research coordinator and, if necessary, inquiries were made to clarify discrepancies
about case information or how a form was completed. Forms that reported cases
demonstrating a change to practice were followed up with a chart audit or further inquiry
to the activity coordinator or director by the education research coordinator.
Bi-annual web-based CME evaluation surveys were established for RSS case
conferences. At that time, case-specific questions were added to the evaluation survey
to subjectively assess changes in practice that resulted from the RSS activity based on
findings from relevant quarterly reports. Surveys were unique to each different RSS
case conference and were deployed only to those who had attended the conference
during a specific time period. For example, a survey to evaluate the weekly Thoracic
Tumor Board was sent to anyone who had attended Thoracic Tumor Board during a
five-month period. Responders were redirected to a case-specific question based on
their positive response to an initial question about attendance on the day the case was
presented.
RESULTS
The new reporting system successfully documented achieved competency in the
RSS case conference and performance change in the practice setting. Competency
was reflected in the interactive case discussion which led to a recommended practice
change and which was documented on the reporting form (Appendix A, B, C).
Competency was also reflected in the self-report of the web-based survey (Fig. 1).
† Appendix A, B, & C show actual reports however the signatures and contact information of the activity director/moderator were removed for this publication.
6
Figure 1. Results from a web-based survey to attendees of the MUSC Thoracic Tumor Board.
Chart audits were used to confirm performance changes in practice which
resulted from the educational discussion and subsequent recommendations of the case
conference (Appendix A). Follow up with the activity director or coordinator regarding
the report of changes not directly related to a specific patient case proved effective in
verifying systems-based outcomes (Appendix B, C).
Regular compliance audits and reminders by CME staff were necessary. Most
activities reported at the frequency requested and with the requested number of cases,
although there were some deviations. A few activities requested to substitute their own
established report for our reporting form, which was approved upon confirmation that
7
the substituted form contained the information required by the CME Office. This type of
cooperative action strengthened the spirit of collaboration.
Common barriers were lack of time to complete the form and difficulty in
discerning which changes were reportable. To address the time barrier, CME staff
changed the reporting frequency from two cases per month to four cases per quarter.
For some case conferences, based on the frequency with which they met (i.e. monthly
vs. weekly) and the number of cases typically discussed, the reporting requirement was
further amended.
After the initial group training it became apparent that more in-depth training
would be necessary due to the quality of the information being reported and the direct
feedback from the activity reporters expressing frustration with not being able to
correctly identify reportable changes for completion of the reporting form. Therefore,
extensive one-on-one training was offered by the CME staff to insure that the
designated reporter for each activity had a solid understanding of how to identify
reportable changes that occurred as a result of their CME activity. This training required
CME staff to accompany the reporter to audit the case conference and demonstrate
how to recognize and document when a reportable change occurred during the case
analysis and discussion. After the activity, the trainer and trainee would then review the
process together, sharing and comparing how they each interpreted what happened
during the case discussions, and then together identifying which changes were
reportable. Examples of reportable changes were added to the back of the reporting
form (Appendix D) to aid in the identification process. Examples of common mishaps in
reporting included:
8
•
Reporting that no changes occurred during a three-month period of
weekly case conferences of 10+ new patient cases per week.
•
Reporting a change in treatment planning that occurred prior to the case
conference and was therefore unrelated to the CME activity.
•
Reporting on a case that was never presented during the case
conference.
WHAT DID WE LEARN?
Implementing a reporting system to document applied learning from RSS case
conferences is labor intensive for all involved. We’ve found our process to be most
effective for treatment planning conferences such as tumor boards where changes are
common after a case is assessed by the multidisciplinary oncology team and are then
readily implemented into practice. Case-based changes related to treatment planning
conferences are also much easier to validate using the electronic medical record
compared to changes that may evolve from other types of case conferences, such as
morbidity and mortality (M&M) conferences. M&M conferences typically address safety
issues after a patient has discharged from care or has deceased. Recommendations
are prospective and often systems-based7 making targeted follow-up a challenge. In
some cases, the M&M activity did not have an accountable designee responsible for
assuring the recommendations were implemented and therefore our attempts to followup on applied changes were unsuccessful. This is not uncommon, based on existing
literature9.
9
While the burden of initial documentation during the case conference primarily
falls on each activity’s director and/or coordinator, the CME staff are responsible for
training, general management (including compliance audits, reminders, etc), and followup of outcomes. For a few conferences, compliance was very challenging; on-site audits
and one-on-one training by CME staff produced the only documentation. Regardless of
the desire to comply and good intentions, these few were unsuccessful at
documentation without direct support of the CME Office.
Recent and future improvements include:
¾ Reduction of the number of reported cases from four to two per quarter to ease
the burden on the reporter and improve compliance.
¾ Modification of the reporting process for M&M and other non-treatment planning
conferences, referencing more recently published literature for guidance.
¾ Development of a supplemental online training module for RSS activity directors
and coordinators.
¾ Identify a means to quantify the impact of our case conferences on patient
outcomes, in particular the M&M conferences.
SUMMARY
Interactive case-based educational strategies are more effective than noninteractive ones11-14. Documenting these effective strategies in an academic medical
center’s RSS program are challenging but rewarding. Aside from the obvious benefits to
the CME Office, our reporting system also benefits the RSS’ clinical department(s) by
showcasing the importance and impact of the RSS case conference on clinical practice
and provides a snapshot of the successful translation of academia to the bedside.
10
References
1. Greene FL. The need for assessment and reassessment of the hospital cancer
conference. Ann Surg Oncol. 2009. 16:2673-2674. DOi 10.1245/S10434-009-0640-0.
2. Gatliff TA, Coleman RL. Tumor board: more than treatment planning – a 1 year
prospective survey. J Ca Educ. 2008;23:235-7.
3. Petty J, Vetto J. Beyond doughnuts: tumor board recommendations influence patient
care. J Ca Educ. 2001;17:97-100.
4. Berenholtz SM, Hartsell TL, Pronovost PJ. Learning from defects to enhance morbidity
and mortality conferences. Am J Med Qual. 2009 May-Jun;24(3):192-5.
5. Wright FC, Lookhong N, Urback D, et al. Multidisciplinary cancer conferences –
identifying opportunities to promote implementation. Ann Surg Oncol. 2009.
DOI:10.1002/ejoc.200700567.
6. Newman EA, Guest AB, Helvie MA, et al. Changes in surgical management resulting
from case review at a breast cancer multidisciplinary tumor board. Cancer
2006;107:2346-51.
7. Aboumatar HJ, Blackledge CG Jr, Dickson C, et al. A descriptive study of morbidity and
mortality conferences and their conformity to medical incident analysis models: results of
the morbidity and mortality conference improvement study, phase 1. Am J Med Qual.
2007 Jul-Aug;22(4):232-8.
8. Szotek JH, Wieland ML, Loertscher LL, et al. A systems approach to morbidity and
mortality conference. Am J Med. 2010 July;123(7):663-668.
9. Kauffmann RM, Landman MP, Shelton J, et al. The use of multidisciplinary morbidity and
mortality conference to incorporate ACGME general competencies. J Surg Educ. 2011
July-Aug;68(4):303-308.
10. Lamb BW, Sevdalis N, Arora S, et al. Teamwork and team decision-making at
multidisciplinary cancer conferences: Barriers, facilitators, and opportunities for
improvement. World J Surg. 2011;35:1970-1976.
11. Price DW, Felix KG. Journal clubs and case conferences: from academic tradition to
communities of practice. J Cont Educ Hlth Prof. 2008;28(3):123-130.
12. Moore DE, Green JS, Gallis HA. Achieving desired results and improved outcomes:
Integrating planning and assessment throughout learning activities. J Cont Educ Hlth
Prof. 2009;29(1):1-15.
13. Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of continuing
medical education. Evid Rep Technol Assess. 2007 Jan;(149):1-69.
14. Kiessling A, Lewitt M, Henriksson P. Case-based training of evidence-based clinical
practice in primary care with decreased mortality in patients with coronary heart disease.
Ann Fam Med 2011;9(3):211-218.
Appendix A
11
Appendix B
12
Appendix C
13
Appendix D
14