The Institute for Evaluation and Outcome Studies
A modest proposal
Jack Kues
April, 2010
There continues to be a dearth of organized research on the evaluation and
outcomes of CME effort. All accredited CME activities are required to assess the
impact of educational efforts and commercial supporters of CME typically require
detailed outcome reports with specific metrics related to number of attendees and
practice changes. We have not, however, seen an increase in our knowledge and
understanding of CME and, for the most part, the massive amount of data that has
been generated has been untouched.
I believe that there are three critical factors that have retarded our ability to benefit
from the ongoing evaluation and outcome efforts of the CME Enterprise: Expertise,
Organization of data, and Time/Resources. Most CME providers have sufficient
knowledge and experience to develop evaluation and outcome strategies that meet
the ACCME requirements and the review of commercial supporters. However, they
lack the statistical expertise to organize and analyze their outcomes data beyond
simple descriptions. Most providers do not have the expertise to developing
databases that can examine pooled data across many activities. The result is a
cursory analysis of individual activities. Large databases of activity outcomes are
gathered by some commercial supporters but they have not typically been able to
organize these data to the point where they can answer questions about the efficacy
and impact of their CME grant programs at an aggregate level. Finally, there is
typically little time or resources to take on the challenge of an outcomes research
program. At the provider level, this would require a sophisticated database or
registry as well as the expertise to analyze the data on an ongoing basis. Some highlevel analysis is being conducted by commercial supporters but they lack the
research and informatics expertise. Additionally, they do not regulate the quality
and organization of the outcomes data that is reported to them.
An Institute
A potential solution to the issues described above is to create an institute based on
informatics resources and experts in theory, data management, biostatistical
methods, outcomes measurement, and qualitative analysis. This institute could be
developed virtually but might also reside in an existing organizational or physical
entity. The human resources could be a loose confederation of experts that could be
put together is small teams around individual projects. This informatics and data
management resources could be newly constructed or purchased from an existing
organization (such as a university) as needed. The Institute would be available to
contract with individual CME providers, Commercial Supporters, or Regulatory
Bodies or the purposes of organizing and analyzing evaluation and outcomes
datasets. Additionally, the Institute might develop a central repository for
evaluation and outcomes data that could be made available for analysis and
reporting. The Institute could provide training and mentor opportunities for
research fellows and graduate students from university-based degree programs and
could develop training and educational materials that reflect the work of the scholar
teams.
The Institute would add value to several organizations and programs that may want
an affiliation in order to facilitate training or research programs. The AAMC,
ACCME, SACME, ACME, and other organizations could have portions of their existing
programs integrated into the Institute for education, research, and data resources.
A viable business model would include income from contracts, data warehousing,
tool development, training, and technical reports. Individuals providing services to
the Institute would be paid on a contractual basis. The infrastructure could be
“rented” from existing resources and overall management would be included in the
overhead of individual contracts.
Potential Partners:
Commercial Supporters:
This is a group that is responsible for reviewing the
quality of educational activities as proposals requesting funding. They are also the
recipients of activity reports, including outcomes data. In order to establish a value
proposition for their grant programs they need to be able to demonstrate that the
activities that they fund have an impact on physician practice and patient care. They
are building large databases of proposals, funded projects, and outcomes reports.
There is very little standardization of these data and the result has been a large, but
very disorganized, database. They can benefit from standardization of data,
organization of their databases, and analysis of the outcomes data from their funded
projects.
CME Providers:
CME providers have seen their missions change as a result of
changes in the environment and ACCME requirements. Their primary focus prior to
the new ACCME accreditation criteria was to design and implement CME events.
The primary data they collected under this model was attendance, perception of
how well learning objectives were met, and their intent to make a change to their
practice. Under the new ACCME criteria, the mission has added a new emphasis on
the impact of CME activities and higher level outcomes that get closer practice
change and patient outcomes. For many CME providers, this is beyond their level of
expertise and they are struggling to make this transition. As commercial supporters
are requiring more explicit outcomes, more CME providers are finding commercial
support beyond their reach or they are not able to meet the expectations of
commercial supporters in their post-activity reports. CME providers have no
templates or guides for measuring outcomes and often have no standards for these
data within their organizations. They typically are unable to examine programmatic
outcomes in any systematic way.
Accreditation Council for Continuing Medical Education (ACCME): The ACCME has
an obligation to gather data from individual accredited providers regarding the
outcomes of their programs. Presently, accredited providers report on the
mechanisms for program outcomes as well as an overview of the data used to
evaluate their program mission. It appears to be increasingly important that
additional data, perhaps at the activity level, is reviewed by the ACCME. There are
no current metrics for these outcome data and it is difficult to compare providers or
to aggregate program evaluation data across providers.
Alliance for Continuing Medical Education (ACME): The Alliance plays a number of
roles in supporting members of the CME community. This includes the
dissemination of information, tools, and other resources. Part of their current plans
for developing member services is to function as a clearinghouse for best practices.
There is currently no place for CME providers and other CME partners to go to find
information about what kinds of programs are being developed and implemented
by their colleagues. There is value in developing a simple database of basic
descriptions of CME activities developed by colleagues. This could be supplemented
with outcomes data or gap analyses. Additionally, tools and standards related to
outcomes may also be listed in this database.
MedEd Portal: The MedEd portal has been developing a variety of strategies to
expand the content and services available to users. Educational materials, as well as
tools, can be very valuable to CME providers, faculty, and physician learners. There
are many CME activities that produce these byproducts and the MedEd Portal is a
good vehicle for dissemination. It is also possible to track who accesses and uses
these materials. This could be a very good measure of the “reach” of best practices.
Association of American Medical Centers (AAMC): The AAMC has been partnering
with several groups to develop a research fellowship for individuals in CME. Such a
program could benefit from working with others who are developing outcome
metrics, analyzing large datasets, and exploring the impact of CME at an enterprise
level. Fellows will need access to ongoing research projects and datasets to provide
practical experience. Additionally, fellows who work on questions closely linked to
CME outcomes will be better positioned to apply for, and obtain funding for CME
research and effective CME activities.
Society for Academic Continuing Medical Education (SACME): The SACME
Research Endowment Council has, as it’s mission, the promotion of research in CME.
They are leveraging a relatively small amount of money in an effort to have a
relatively large impact on critical CME questions. They lack access to significant
datasets although they are connected to many experienced academic researchers.
These resources can be utilized to take on methodological, theoretical, and
statistical issues that would likely exist in a large outcomes database. SACME is a
partner with AAMC in the development of a research fellowship described above.
National Institute for Quality Improvement and Education (NIQIE): NIQIE has
positioned itself as a bridge between CME and Quality Improvement. It’s current
projects include the development communities of practice, a PI registry, and
research initiatives. NIQIE also has a significant IT infrastructure and informatics
expertise. These are resources that are critically important to development of a
CME data warehouse and outcomes registry but are not readily available in other
institutions. Additionally, NIQIE is in a good position to develop regular reports
from such databases and registries.
Medbiquitous: Medbiquitous has taken on the task of developing data standards
related to CME. These standards have been largely developed and tested but have
not resulted in tools or other data collection mechanisms that have been widely
developed across the CME Enterprise. They can contribute these standards and
support implementation throughout the larger data organizing effort. They can also
help analyze and improve standards over time. They would value feedback from the
implementation of their standards.
The Model:
The institutions and organizations described as potential partners have a variety of
existing relationships with each other. The nature of the relationships seems to fall
into three constellations: CME programs/accreditation; academic/professional
CME; and CME informatics. Within these three areas, the organizations have
frequent contact and relationships that focus on their missions and operations.
However, most of these organizations have relationships across constellations to
almost all of the other entities. Overall, there is a need to share information,
resources, and planned initiatives to support themselves and the overall CME
Enterprise. The model depicted below is a loose description of these relationships.
The proposed Institute functions as an independent broker of data, resources, and
collaboration among all of the potential players. The Institute’s independence is
rooted in a mission that is focused on research, validation of data, and the
development of collaborative resources.
The Collaborative Institute Model