Interpreting Data &
“Closing the Loop”
Margaret Kasimatis, PhD
VP for Academic Planning & Effectiveness
Location in Assessment Cycle
IDENTIFY
SPECIFIC
OUTCOMES
DETERMINE
PRACTICES USED
TO ACHIEVE
OUTCOMES
GATHER
EVIDENCE
ARTICULATE
MISSION/
GOALS
RECOMMEND
ACTIONS
REVIEW &
INTERPRET
RESULTS
Step #1: Organize Your Data
• Assemble the following in one place:
– All data or possible sources of data
– List of learning outcomes/research questions
– Curriculum, experiences, practices
• Map data sources to outcomes
– Outcomes x Measures Map
• Map results to outcomes
– Present in “user-friendly” way
Mapping Data Sources to Outcomes:
Engineering Example
ASSESSMENT MEASURES
LEARNING
OUTCOMES
Alumni
Survey
Ability to apply knowledge in math,
science, and engineering
National Survey of
Student Engagement
(NSSE)
Evaluation of
Clinic
Presentations
Evaluation of
First-Year
Presentations
X
Rubric Evaluation
of Upper-level
Coursework
X
Ability to design a system, component,
or process to meet desired needs
X
Ability to identify, formulate, and
solve engineering problems
X
X
Ability to communicate effectively
X
X
Ability to use techniques, skills, and
modern engineering tools necessary
for engineering practice
X
X
X
X
X
X
X
Ability to function on
multidisciplinary teams
X
X
Recognition of need for and ability
to engage in lifelong learning
X
X
Understanding of professional and
ethical responsibility
X
X
Broad education necessary to
understand the impact of engineering
solutions in a global & societal context
X
X
Presenting Results
• It’s best to present results one outcome
at a time
– It’s not recommended to present results primarily
by measure
• Make sure to present the data in a userfriendly way (i.e., in a way that faculty in your
department are comfortable with)
HMC Senior Engineers
AITU Seniors
Liberal Arts Seniors
Mean Response (1-4 scale)
4
3
2
1
Experience contributed to ability
to write clearly and effectively
Rubric Evaluation of Student Work: Writing Style
Mean Rubric Score on Writing Style (1-5 scale)
NSSE: Data on Communication Skills
Percent Indicating "Very Much"
60
Biology
Chemistry
Comp Sci
Engineering
Math
Physics
40
4
3
2
1
Experience contributed to ability
to speak clearly and effectively
Course #1
Alumni Survey: Data on Communication Skills
50
5
Course #2
Rubric Evaluation of Student Presentations
4
Delivery
Language
Organization
Content
3
30
20
2
10
0
To what extent was an emphasis on
writing skills present while you were at
HMC?
To what extent was an emphasis on oral
communiction skills present while you
were at HMC?
1
First-Year Presentation
Clinic Presentation
NSSE
LMU Jesuit Master’s Total
ACADEMIC CHALLENGE:
How often did you work harder than
you thought you could to meet an
instructor's standards or expectations?
FY
62%
55%
54%
52%
(Percent of students who said ‘Often’ or
‘Very Often)
SR
54%
59%
60%
56%
How much time did you spend
preparing for class (e.g. studying,
FY
65%
68%
49%
56%
SR
40%
58%
52%
54%
FY
21%
17%
17%
19%
(Percent of students who said ‘Often’ or
‘Very Often’)
SR
27%
23%
20%
23%
Describe the extent in the current
year to which exams challenged
you to do your best work.
FY
5.47
5.56
5.33
5.42
(mean response on a 7-point scale)
SR
5.33
5.46
5.43
5.40
reading, writing, doing homework, rehearsing,
other academic activities)?
(Percent of students who reported
spending 11 or more hours per week)
How often did you come to class
without completing readings or
assignments?
Step #2: Interpret the Data
• What type of criterion?
• What is “significant?”
• Are the findings reliable/valid?
What type of criterion?
• Norm-referenced
– Need appropriate comparison group
• Avoid percentile rankings
– Good for more nebulous findings
• Absolute standard
– Usually more appropriate for
performance-based measures
What is significant?
• Important to test when making
comparisons
– Numbers that look different may not really
be different
• However, just because a difference is
significant doesn’t mean it’s important
Are findings reliable/valid?
• Can we believe student self-reports?
• Are standardized measures more
valid?
• Single measure vs. multiple measures
Step #3: Make Recommendations
• Start by considering
– Where in the curriculum are outcomes
addressed? And at what level?
• Refer to (or create) curriculum map, or
• Inventory for particular outcome
– How? What practices/techniques are used?
– Where are the gaps?
– What can be changed?
• In the short-term? In the long-term?
Step #3: Make Recommendations
• Possible types of recommendations:
– Changes to pedagogy
– Changes to curriculum/programming
– Allocation of resources
• Keep in mind that you can’t fix everything
at once – so start small
• It’s usually better to modify than add
• Be as specific/concrete as possible
Step #4: Implement & Document
Recommend Actions
Example
PROPOSED ACTION
IMPLEMENTATION
Document (with grades) the
importance of presentation
clarity
Essay questions added to exams
Engineering Curriculum Committee developed
writing rubric to be used to evaluate project report
Department selected Hacker, A Writer’s
Make common writing style
guidelines available to faculty Reference
and students
Hacker, A Writer’s Reference required in E4
Hold Clinic team members
accountable for peer review
of team reports
Clinic advisors distributed peer review sheets to
Clinic team members. Students completed the
peer review sheets when they reviewed fellow
team members’ portions of Clinic report .
Example of Assessment Loop
• Step #1: Articulate Goals/Outcomes
– Goal: Graduates will possess strong
communication skills
• Outcome: Graduates will be able to write clearly
and effectively
• Step #2: Determine practices used to
achieve outcome
– Survey of required writing experiences in Core
and academic programs
Sem #
BIO
CHEM
CS
ENG
MATH
PHYS
One
4 papers
Two
Three
H/SS
Report in E4
1-2 papers in
Bio 101
Tech report
in Phys 28
Lab notebook
& paper in
Chem 53
4 papers
Tech report
in Phys 53
3 lab reports
Four
4 lab reports
1-2 papers in
Bio 108
Five
Six
Essays/paper
in Chem 52
Lab notebook
Exp abstracts
1-2 papers in
Bio 109
3 reports +
paper in 109
3 lab reports
Exp abstracts
1-2 papers
in Bio 113
Design
doc,
Report, &
User guide
in CS 121
Tech report
in Phys 54
Term paper
in Phys 52
Abstracts
in CS 110
Proposal &
mid-year
report OR
Project paper
in Chem 114
Essay in
CS 131
Clinic report
IE Paper
IE Paper
IE Paper
IE Paper
Proposal
Letter of Intro
3 Lab
reports
Prog report
Proposal +
midyear
report
Proposal +
midyear
report
Variable
Amount
Of
Writing
Tech report
in Phys 134
Seminar
paper
Seven
IE Paper
IE Paper
Seminar
Researc
h
Paper
Example of Assessment Loop, cont’d
• Step #3: Conduct measurements
– Higher Education Research Institute (HERI)
College Student Survey (CSS)
– National Survey of Student Engagement
– HMC Alumni Survey
Example of Assessment Loop, cont’d
• Step #4: Review & interpret results
– Faculty committees
• Assessment
• Curriculum
• Teaching & Learning
– Full faculty - workshop
Writing Skills
Frequency data from HERI, NSSE & Alumni Survey
80
70
60
Liberal
Arts
50
40
4-Yr NonSectarian
30
20
HMC
Seniors
HMC AITU
Seniors Seniors
1959-19871988-2001
Cohort Cohort
10
0
"Much stronger writing
skills than when entering
college"
"Institution contributed
very much to ability to
write clearly and
effectively"
"To what extent was an
emphasis on writing skills
present while you were at
HMC?" (5)
To what extent did experiences at your institution contribute
to your ability to write clearly and effectively?
Seniors’ Perceptions of Writing Skills
by Major (from NSSE)
4
3.5
3
2.5
2
1.5
1
Biology
CS
Engineering Mathematics
Physical
Sciences
Writing emphasis was "very much" present
Alumni Perceptions of Writing Emphasis
by Major (alumni graduating in last 10 years)
70
60
50
40
30
20
10
0
Biology
Chemistry
CS
Engineering
Math
Physics
Sem #
BIO
CHEM
CS
ENG
MATH
PHYS
One
4 papers
Two
Three
H/SS
Report in E4
1-2 papers in
Bio 101
Tech report
in Phys 28
Lab notebook
& paper in
Chem 53
4 papers
Tech report
in Phys 53
3 lab reports
Four
4 lab reports
1-2 papers in
Bio 108
Five
Six
Essays/paper
in Chem 52
Lab notebook
Exp abstracts
1-2 papers in
Bio 109
3 reports +
paper in 109
3 lab reports
Exp abstracts
1-2 papers
in Bio 113
Design
doc,
Report, &
User guide
in CS 121
Tech report
in Phys 54
Term paper
in Phys 52
Abstracts
in CS 110
Proposal &
mid-year
report OR
Project paper
in Chem 114
Essay in
CS 131
Clinic report
IE Paper
IE Paper
IE Paper
IE Paper
Proposal
Letter of Intro
3 Lab
reports
Prog report
Proposal +
midyear
report
Proposal +
midyear
report
Variable
Amount
Of
Writing
Tech report
in Phys 134
Seminar
paper
Seven
IE Paper
IE Paper
Seminar
Researc
h
Paper
Example of Assessment Loop, cont’d
• Step #5: Recommend actions
– Universal writing handbook
– Faculty workshop on writing instruction
– Revision to Core Curriculum
MULTIPLE CURRICULAR OBJECTIVES
IN HORIZONTALLY INTEGRATED
CORE CURRICULUM
•
Content
•
Skills
Context
•
•
•
•
•
•
•
disciplinary knowledge
discipline-related techniques
oral and written communication
critical thinking
teamwork and collaboration
project management
leadership
relationship of science or technology with
contemporary society
individual and comparative explorations of
cultural identity
QUESTIONS?