Making Our Way from
Anecdote to Evidence
Presentation for
Georgia College & State University
Caroline R. Noyes, Ph.D.
Assistant Director, Office of Assessment
BACKGROUND AND
INTRODUCTION
2
Four Purposes of Evaluation
(Mark, Henry, & Julnes ,2000)
1. Assessment of merit and worth: the development of warranted
judgments, at the individual and societal level, of the value of a
policy or program.
2. Program and organizational development: the effort to use
information to directly modify and enhance program operations.
3. Oversight and compliance: the assessment of the extent to
which a program follows the directives of statutes, regulations,
rules, mandated standards or any other formal expectations.
4. Knowledge development: the discovery or testing of general
theories, propositions, and hypotheses in the contexts of policies
and programs.
3
Program
Mission
Program
Goals
Student
Learning
Outcomes
4
Program Mission
History:
The department has as its mission through the BA in History to provide
students with a basic knowledge of selected areas of history; to teach students
to think critically in analyzing historical issues, to write clearly and cogently on
historical topics, and to conduct historical research; and to provide students
with a basic understanding of history as a discipline.
Theatre:
We seek to provide the best education possible in theatre and dance for our
students, preparing them for careers in the professional and educational
arenas both in and out of the performing arts. We cultivate imagination,
passion, discipline, and collaboration in the creation of arts and artists. We
emphasize interdisciplinary connections and a global perspective as essential
components of a liberal arts education.
5
Sample Program Goals
a.
Graduates will be able to communicate their knowledge effectively,
both orally and in writing.
b. Graduates will demonstrate knowledge of the major authors, periods,
movements, and issues in drama and theater.
c. Graduates will understand and apply basic research methods in
psychology, including research design, data analysis, and
interpretation.
6
Properties of Effective
Student Learning Outcomes
7

Have a clear purpose

Use action words

Describe meaningful learning

Are easily understandable

Represent high level learning

Result in observable behaviors/products
Formulating Student Learning Outcomes

Method
 Inventory of program content

Rank most important items

Categorize by outcome type

Draft outcome statement
Revise using criteria for quality outcomes
Use action verbs



8
Should be observable (either directly or indirectly
inferred)
Bloom’s Taxonomy of the Cognitive Domain
Evaluation
Synthesis
Analysis
Application
Comprehension
Knowledge
9
Action Verbs Accompanying Bloom’s Taxonomy
10
Knowledge
Comprehension
Application
Analysis
Synthesis Evaluation
Describe
Compare
Apply
Analyze
Arrange
Argue
Find
Describe
Classify
Appraise
Assemble
Assess
List
Distinguish
Complete
Categorize
Collect
Choose
Locate
Discuss
Construct
Compare
Combine
Conclude
Name
Explain
Demonstrate
Contrast
Comply
Decide
Recall
Express
Dramatize
Debate
Compose
Evaluate
Record
Identify
Employ
Diagram
Construct
Interpret
Relate
Outline
Illustrate
Differentiate
Create
Judge
Repeat
Predict
Interpret
Distinguish
Design
Justify
State
Recognize
Operate
Examine
Devise
Measure
Tell
Restate
Practice
Experiment
Formulate
Rate
Underline
Tell
Schedule
Inspect
Imagine
Recommend
Write
Translate
Sketch
Inventory
Manage
Revise
Solve
Investigate
Organize
Support
Use
Question
Plan
Value
Separate
Predict
Verify
Bloom’s Taxonomy of the Affective Domain
Characterization
Organization
Valuing
Responding
Receiving
11
Action Verbs Accompanying Bloom’s Taxonomy
http://www.abet.org/_TrainingCD/data/references/Assessment%20Links/Goals_revised_Blooms.pdf
Receiving
Responding
Valuing
Organization
Characterization
Asks
Answers
Completes
Adheres
Acts
Chooses
Assists
Describes
Alters
Discriminates
Describes
Complies
Differentiates
Arranges
Displays
Follows
Conforms
Explains
Defends
Influences
Gives
Discusses
Follows
Explains
Listens
Identifies
Helps
Initiates
Generalizes
Modifies
Names
Performs
Joins
Identifies
Performs
Selects
Presents
Justifies
Integrates
Practices
Replies
Selects
Reads
Modifies
Proposes
Uses
Tells
Reports
Organizes
Qualifies
Selects
Prepares
Questions
Shares
Relates
Serves
Studies
Synthesizes
Solves
Works
Uses
Verifies
12
Student Learning Outcomes:
Oral Communication
Graduates will be able to communicate their knowledge effectively,
both orally and in writing.
Students will:
13
Knowledge
support arguments with relevant and adequate
evidence
Skills
answer questions directly and accurately
Dispositions
convey enthusiasm for their topic
Student Learning Outcomes:
Written Communication
Graduates will be able to communicate their knowledge
effectively, both orally and in writing.
Students will:
14
Knowledge
respond to essay questions on exams with clear, wellorganized presentations focused on the topic and
relatively free of errors
Skills
recognize well-written texts and advise peers on how to
improve their written work
Dispositions
develop a personal voice and sense of style in writing
Student Learning Outcomes:
Research Methods (Psychology)
Graduates will understand and apply basic research methods in
psychology, including research design, data analysis, and
interpretation
Students will:
15
Knowledge
articulate strengths and limitations of various research
designs
Skills
formulate testable hypotheses, based on operational
definitions of variables
Dispositions
recognize that theoretical and socio-cultural contexts
as well as personal biases may shape research
questions, design, data collection, analysis, and
interpretation
WORKING ON LEARNING
OUTCOMES IN GROUPS
16
GC&SU Core Outcomes
• Communicate clearly in written English, demonstrating
comprehension, analysis, and critical interrogation of a variety of
texts. (A: Essential Skills; English)
• Use description, analysis, and synthesis of data, ideas, or
information appropriate to the purpose. (A: Essential Skills:
English)
• Express and interpret mathematical information, concepts, and
thoughts in verbal, numeric, graphical, and symbolic form, and to
shift among these different modes when solving problems. (A:
Essential Skills; Math)
• Use appropriate technology in the location, evaluation, analysis,
synthesis, and application of information in problem-solving
situations (A: Essential Skills; hybrid of Math and English)
17
FINDING THE EVIDENCE
18
Things to Consider:

What’s already in place?
Exams, assignments, or projects?
 Common exams, assignments, or projects?
 Writing samples (common question)
 Capstone experience products


Look for logical assessment points in the curriculum
At program entrance
 Program courses required for the major
 At program conclusion


19
The evidence you collect depends on the questions you
want to answer
Kinds of Evidence, Part 1
Direct Evidence
Indirect Evidence
Measure student learning outcomes
directly
Measure opinions or thoughts
about students’ (or alumni)
own knowledge, skills, attitudes,
learning experiences, etc. Also
includes external evaluations.






20
Embedded questions/tasks
Pre-post testing
Evaluation of projects/work
Standardized tests (e.g., MFT)
Portfolios
Presentations




Alumni surveys
Exit surveys
Employer surveys
Focus Groups
Kinds of Evidence, Part 2
Quantitative Evidence




21
Ratings of student
skills/performance
Scores and pass rates on tests
(licensure/certification)
Scores on locally-designed
tests (final exams, qualifiers,
comprehensives) when
accompanied by test
“blueprints”
Classroom response systems
(clickers)
Qualitative Evidence






Observations of student
behavior w/ systematic notes
Summaries of discussion
threads
Student reflections
Portfolios evaluation
Written work, performances,
or presentations scored with a
rubric
Analysis of comments on
surveys
Student Learning Outcome:
answer questions directly and accurately
22
Course
Point of
Assessment
Assessment
Method
Introduction to Psychology
Oral
presentation on
40 STCP
chapter
Presentation
rubric
(professor)
Research Methods
Oral
Presentation
presentation of rubric
research project (professor and
peers)
Psychology Seminar
Classroom
Presentation
guest lecture on rubric
historical figure (professor)
in psychology
Results
Oral Presentation Rubric
Subject
Knowledge
23
(http://www1.uprh.edu/cruzmigu/OralRubric.pdf)
4
3
2
1
Student
demonstrates
full knowledge
by answering
all class
questions with
explanations
and
elaboration
Student is at
ease with
expected
answers to all
questions,
without
elaboration
Student is
uncomfortable
with
information
and is able to
answer only
rudimentary
questions
Student does
not have grasp
of information;
student cannot
answer
questions
about subject.
Student Learning Outcome:
articulate strengths and limitations of various research designs
Course
Point of
Assessment
Assessment
Method
Introduction to Psychology
1st Test
Multiple Choice
or Matching
Question
Developmental Psychology
1st Test
Application
short answer
Final Exam
Essay question
a) 1st Test
Application
question
b) Research
Proposal
Rationale for
research
method
selection
Research Methods
24
Results
IDENTIFYING SOURCES
OF EVIDENCE IN GROUPS
25
SHARING YOUR FINDINGS
26
At Last: Reporting your Findings
27

Analyze the data…identify the results

Integrate results from various assessment methods

What conclusions can you draw from that evidence?

What recommendations arise from that data?

What actions should/will you take based on the
recommendations?
Student Learning Outcome:
answer questions directly and accurately
Course
Point of
Assessment
Assessment
Method
Results
Introduction to Psychology
Oral presentation
on 40 STCP
chapter
Presentation
rubric (professor)
Spring 2010:
n=55, avg. score
was 1.9
Research Methods
Oral presentation
of research
project
Presentation
rubric (professor
and peers)
Fall 2009: n=25,
avg. score was
2.6 (avg. peer
rating was 3.1)
Psychology Seminar
Classroom guest
lecture on
historical figure in
psychology
Presentation
rubric (professor)
Spring 2010:
n=25, avg. score
was 2.9
In general, as students progress though the major, there is improvement in students’ ability to
answer questions at the end of their oral presentation, but students may benefit from more
practice in elective classes. Increase use of peer ratings to increase student critical reflection
on this task.
28
Closing the Loop, Option 1
Outcome
Practice
29
Closing the Loop, Option 2
Practice
Outcome
30
Conclusions
The “systematic collection, review, and use of information
about educational programs undertaken for the purpose of
improving learning” allows us to address three
fundamental questions:
1. What have our students learned?
2. How well have our students learned that information?
3. How successful have we been at what we are trying to
accomplish?
31
Contact Information
Caroline R. Noyes, Ph.D
Assistant Director, Office of Assessment
Georgia Institute of Technology
(404) 894-0510
cnoyes@gatech.edu
32