Resident Evaluation of a Web-based Integrative
Medicine Curriculum: The Function of
Formative Evaluation
Ben Kligler, M.D., Patricia Lebensohn, M.D.,
Sally Dodds, Ph.D., Raymond Teets, M.D., &
Victoria Maizes, M.D.

The objectives of this presentation are to:
▫ Describe the function of formative evaluation for curriculum development.
▫ Review the results from learners’ evaluation of the Integrative
Medicine in Residency curriculum.
▫ Describe the procedures used in applying feedback from learners’ evaluations to guide curriculum changes and revisions.

• Competency-based, online, 200-hour, curriculum.
• In-depth training in Integrative Medicine.
• Incorporated through all 3 years of Family Medicine residency.
• Piloted at 8 residencies nationwide.
• Seamless, online evaluation of the curriculum and the residents.
• Responds to ACGME competency requirements.
• Evaluation developed simultaneously with the curriculum.

Hennepin County
Maine-Dartmouth
Maine Medical Center
University of
Connecticut
Beth Israel
Carolinas Medical Center
University of Arizona
University of Texas
Medical Branch

Prevention and
Wellness
Tools in
Integrative
Medicine
Pediatrics
Acute
Care
Women’s
Health
Chronic
Illness

Introduction to Integrative Medicine
Prevention and Wellness:
• U.S. Preventive Health Services
• Nutrition and Diet
• Supplements for Prevention
• Physical Activity
• Sleep
• Stress and Mind-Body Medicine
• Spirituality
• Clinical Integration
Tools in Integrative Medicine:
• Integrative Medicine Intake and Care Plan
• Botanicals
• Mind-Body Medicine
• Manual Medicine
• Introduction to Energy Medicine and Whole Systems
• Practice Management
• Motivational Interviewing for Behavioral Change
Acute Care:
• Acute Back Pain,
• Urinary Tract Infection,
• Gastroenteritis,
• Otitis Media,
• Vaginitis,
• Atypical Chest Pain,
• Upper Respiratory Infection
Pediatric Topics:
• ADD/ADHD
• Chronic Pain Syndrome
• Asthma and Allergies
Women’s Health Topics:
• PMS/PMDD
• Dysmenorrhea
• Menopause
• Fibromyalgia
• Osteoporosis
• Depression
• Eating Disorders
• Pregnancy and Lactation
Chronic Illness:
• Cardiovascular Disease
• Type II Diabetes
• Osteoarthritis
• Rheumatoid Arthritis
• Obesity
• Irritable Bowel Syndrome
• Chronic Back Pain
Special Topics:
• HIV/AIDS
• Cancer Survivorship
• Environmental Medicine

 Needs assessment informed curriculum design.
 Web-based curriculum written and edited by Integrative
Medicine educators.
 Competencies aligned with the ACGME Outcomes Project.
 Flexible modular format to meet the needs of residency schedules.
 Case-based, interactive learning and streaming video.
 Experiential exercises and process-oriented group activities at the residency sites.
 A community of learners through online dialogues with faculty and peers.
 Emphasis on teaching and promoting physician well-
being and self-care.

Flexible modular format
Direct Observation Checklists
Reflections
Resources and Links
Online portfolios

“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.”
- Robert Stakes

• Helps form and strengthen programs by:
▫ Examining the need for them,
▫ Their delivery or technology
▫ The quality of their implementation
▫ Assessing their organizational contexts, procedures, and resources
(Scriven, 1991).
• Methods emphasize data collection and analysis prior to
completion.

• Medical knowledge in Integrative Medicine is dynamic, evolving rapidly; revisions are largely driven by new information.
• Distributed, web-based curricula requires feedback from learners to be successful.
• In the IMR, formative methods include:
1. Preliminary needs assessment
2. Feedback from resident evaluation of courses

(Benn, Maizes, Guerrera, Sierpina, Cook, & Lebensohn, 2009)
• Methods
▫ 222 faculty and residents from 8 family medicine programs
(60.2% response rate).
▫ Online survey w/ structured and open-ended questions.
• Results
▫ Preferred IM be woven throughout all curriculum areas (67%).
▫ Top topics: Nutrition, Supplements, and Physician Wellness.
▫ Top curricular areas for IM enhancement: Chronic illness, behavioral health, and outpatient medicine
▫ Viewed IM central to family medicine training, patient care, and the field of family medicine (84%).
▫ Top challenges: Limitations in time, resources, and acceptance.

• Measures assess learners’ ratings of the course in:
▫ Meeting course objectives
▫ Clinical utility of the course
▫ Time needed to complete the course
▫ Functionality of the online technology
 Analyzed when 50% of pilot residents complete a course.
▫ Ratings < 8o% and open-ended comments targeted for review.
 Review of 01 pilot group suggested Likert-type response categories be changed from 4 to 5 points to increase precision and variability.

2011 (n = 67) 2012 (n = 64) Controls (n = 31)
Sex
Female
Male
Mean Age range
Marital Status
Married
Live w/ partner
Single
Unk.
Graduate Status
US MD
US DO
FMG
Unk.
64%
36%
32
(27 – 43)
45%
13%
27%
15%
51%
18%
27%
4%
64%
36%
32
(24 – 56)
27%
6%
34%
33%
41%
26%
28%
5%
45%
55%
30
(26 – 38)
58%
10%
32%
0%
55%
16%
16%
13%

Year 01 Content
Very definitely/Definitely
2011 2012
92% 92%
94% 94% 94%
96%
82%
91% 91%
95% 94%
89%
95%
*Currently, data are available for 4 courses for the 2012 class.

Very Useful/Useful
2011 2012
78%
74%
95% 96%
93%
97%
99%
96%
86%
89%
91%
47%
93%
*Currently, indicators are available for 4 courses for the 2012 class.

Year 01 Content
Very Smoothly/Smoothly
2011 2012
78%
82%
91%
83%
95% 94%
97%
94% 94%
90%
93%
81%
85%
*Currently, indicators are available for 4 courses for the 2012 class.

Courses currently at 50% threshold
Meet Learning Objectives?
97%
Very Definitely/Definitely
98%
100% 100%
Clinical Utility?
100%
Very Useful/Useful
98%
100%
94%

Courses currently at 50% threshold
Smooth Online technology?
92%
Very smooth/Smooth
100%
82%
77%

Year 01 &Year 02 Content
Year
1
Year
2
Class Course
2011 Introduction
2012
2011 Supplements
2012
2011 Nutrition & Diet
2012
2011 Physical Activity
2012
2011 Stress & Mind Body
2011 Spirituality
2011 Sleep & Health
2011 Preventive Services
2011 Clinical Integration
2011 Allergy/Asthma
2011 ADHD
2011 Pediatric Pain
2011 PMD/PMDD
34%
72%
94%
59%
17%
11%
48%
55%
53%
< 1 hr
34%
34%
32%
50%
13%
21%
61%
66%
1-2 hr 2-3 hr 3-4 hr 4-5 hr
44% 13% 5% 4%
36%
35%
26%
12%
4%
12% 5%
36%
25%
42%
38%
28%
43%
26%
6%
32%
38%
9%
31%
29%
2%
7%
17%
2%
7%
21%
27%
16%
3%
8%
11%
6%
4%
2%
43%
48%
27%
40%
14%
7%
9%
3%
4%
4%
5-6 hr
3%
5%
7%
4%
8%
5%

I enjoyed the interactive nature of the modules.
The tools are amazing!
Did an excellent job of relating spirituality to health care.
I think sleep is one thing most neglected in health care topics. This course has rightly identified it.
The ability to self reflect as well as think about our patients.
I loved the video lecture – very engaging and interesting.
This is a very comprehensive topic.
Great links and resources.
The seven cases. Great way to pull all of these courses together!
The buttons aren’t working.
There was a lot of data in this module
[Nutrition and Diet], but not enough interactive learning.
Too lengthy and too in-depth.
Some links were not available.
Too many resources were given and I did not have enough time to thoroughly review them.
Too much reading, too many links, and too many videos.

Acknowledgements
University of Arizona Pilot Sites
Emily Sherbrooke
Paula Cook
Rhonda Hallquist
Tieraona Low Dog, M.D.
Victor Sierpina, M.D.
Selma Sroka, M.D.
Mary Guerrera, M.D.
Dael Waxman, M.D.
Craig Schneider, M.D.
John Woytowicz, M.D.