TESTING IN THE WORLD OF HEALTH EDUCATION

With a classmate:

IDENTIFY at least TWO ways to assess your students’ learning in new/modified ways

CLARIFY how these ways of assessing learning could help (a) your students, (b) you as the instructor, or (c) both!

CREATE one or two assessment items of a type that could contribute to your project (or to your own teaching)

Evidence-Based Teaching, Spring, 2012
Drs. Anne Belcher & Linda Adamson, Instructors
Session 9


At the end of this session, the learners will be able to demonstrate the ability to use the following strategies for the evaluation of teaching and learning:

Formative feedback

Summative/course evaluation

Peer review


Purposes of formative feedback

Guides future learning and teaching (evidence suggests that appropriate, well-delivered formative evaluation can significantly improve learning processes and outcomes)*

Promotes reflection in the learner and in the teacher
Shute (2007); Epstein (2007)


More effective when it provides details about how to improve (elaboration) rather than just indicate the student’s/teacher’s work is correct
(verification)


Is intended to help the learner to modify their thinking or behavior

Should permit comparison of actual performance to an established standard

Think: non-evaluative; supportive; timely; credible; specific; learner-driven

FORMATIVE FEEDBACK AND THE MURDER
MYSTERY*

Motive—the learner needs the motivation

Opportunity—the learner receives the feedback in time to use it

Means—the learner is ready and willing to use the feedback
*Shute; Elliott





Archer, J.C. (2010). State of the science in health professional education: Effective feedback. Medical
Education, 44 , 101-108.
Black, P. & William, D. (2007). Inside the black box:
Raising standards through classroom assessment. Phi
Delta Kappan, 80 (2). http://www.pdkintl.org/kappan/kblas9810.htm
Epstein, R.M. (2007). Assessment in medical education.
New England Journal of Medicine, 365 (4), 387-396.
Shute, V.J. (2007). Focus on formative feedback.
Princeton: Educational Testing Service.


Review examples from Schools of Nursing,
Medicine, and Public Health that have been made available in class tonight

ISSUES TO CONSIDER WITH REGARD TO
STUDENT RATINGS

“just a popularity contest”

“just plain unreliable and invalid”

Students “too immature, inexperienced, and capricious”

Concern that teacher can “buy” good ratings

Easier to get good ratings in higher level courses

Rating of required course harsher than of elective course


Why are the response rates low and what can be done about that?

What should be done with the data/reports?


Arreola, R.A. (2007). Developing a comprehensive faculty evaluation system. A guide to designing, building, and operating large-scale faculty evaluation systems . Bolton,
MA: Anker Publishing Company, Inc.

JHU SCHOOL OF NURSING PEER REVIEW OF
TEACHING

The two components of the JHUSON process are peer observation in the classroom setting and peer review of course materials (including
Blackboard course sites)

According to Arreola (2007), peers can provide the following categories of information:
Content-dependent performance, including currency of the content, appropriateness of the sequencing of the content to help learners achieve the objectives of the course;

Content-independent performance, for example, service to the institution


Is formative evaluation of classroom interactions clinical instruction, and/or teaching materials

Is intended to be developmental rather than judgmental

Provides opportunity for teacher to receive collegial feedback that complements that received from students


Non-participant observation

At least once during the course

Date, time and length of observation negotiated with teacher

Ideally a content reviewer and a pedagogy reviewer


Number of students in the class

Description of the classroom environment

Copy of the course syllabus and lesson plan

Audiovisual materials, including power point

List of issues/strategies/concerns


Creation of a written report

Participant debriefing

ISSUE TO CONSIDER

What is the relative advantage of a review:

“live
 recorded


Arreola, R.A. (2007). Developing a comprehensive faculty evaluation system .
Bolton, MA: Anker Publishing Company, Inc.

Chism, N. V. N. (2007). Peer review of teaching.
A sourcebook.
Bolton, MA: Anker Publishing
Company, Inc.

For the purposes of this JHUSON peer review process, the focus will be on content-dependent performance and pedagogy.
The general guidelines proposed for the peer observation are:
Each observation will, whenever possible, be conducted by a content expert and an expert in pedagogy; these experts can be internal
(within the SON) or external
(within JHU)

The length of the observation will also be negotiated
(a sample in time versus the entire class)
The revised Faculty Evaluation
Forms (based on form created by
Berk et.al, as well as a seminar form to be created) will be used by the reviewer for recording observations in the classroom setting.

The expert/reviewers will have been trained in the use of the
Faculty Evaluation Forms and the procedures to be followed.
The non-participant observations should occur at least once in the course, the date and time to be negotiated by the faculty reviewee in collaboration with the reviewer(s). If possible the reviewee should arrange to have the class media-sited for followup anaylsis with the reviewer.

The faculty member to be reviewed will share a list of any concerns/questions about their teaching in preparation for the observations as well as the course syllabus, “lesson plan” and/or power point presentation one week prior to the class. If the review is of teaching materials, access to the
Blackboard course site will also be provided.

The reviewer(s) will conduct a debriefing session with the reviewee within a reasonable period of time after the observation(s). The reviewer(s) will provide the reviewee with a written report which the reviewee may choose to submit, in combination with the reviewee’s response to the report and plans for faculty development, with his/her portfolio to the A&P Committee.
Evaluative data to be submitted by the reviewer(s) and the reviewee to the OTE is the completed Peer
Review Survey regarding satisfaction with the peer review process and usefulness of the process.

References
Arreola, R.A. (2007). Developing a comprehensive faculty evaluation system . Bolton, MA: Anker
Publishing Company, Inc.
Chism, N. V. N. (2007). Peer review of teaching. A sourcebook.
Bolton, MA: Anker Publishing
Company, Inc.

Evaluation Items/Rubric:
 How will you gather evidence of your students’ learning?

How will you use it to identify different levels of learner success?

How will results help inform your subsequent teaching?

once in the course, the date and time to be negotiated by the faculty reviewee in collaboration with the reviewer(s). If possible the reviewee should arrange to have the class media-sited for followup anaylsis with the reviewer.
The length of the observation will also be negotiated
(a sample in time versus the entire class)
The revised Faculty Evaluation
Forms (based on form created by
Berk et.al, as well as a seminar form to be created) will be used by the reviewer for recording observations in the classroom setting.
Rubrics are to be developed for review of course materials using

Revisit tonight’s learning objectives.
Evaluate the extent to which each objective was successfully addressed.
Describe any new understandings, greater clarity, or additional insights that you developed during this session.
Provide a suggestion for how this session could have been more effective.

Elzubieir, M. & Rizk, D. (2002). Evaluating the quality of teaching in medical education: Are we using the evidence for both formative and summative purposes? Medical Teacher, 24 (3), 313-319. Retrieved from http://informahealthcare.com/doi/abs/10.1080/01421590220134169
Hatzipanagos, S. & Lygo-Baker, S. (2006). Teaching observations: A meeting of minds? International Journal of Teaching and Learning, 17 (2), 97-105.
Retrieved from http://www.isetl.org/ijtlhe/pdf/IJTLHE52.pdf
Nilson, L. B. (2010). Teaching at its best: A research-based resource for college instructors (3 rd ed.).
San Francisco: Jossey-Bass, 315-328.