Closing the Gap:
From Data to Decision Making
Stefinee Pinnegar, Ph.D.
Pat Esplin, Ph.D.
Brigham Young University
academy.byu.edu
Closing the gap
from data to decision making
• Our world is confronted with problems that occur over and over and are difficult to address
Intractable • Providing solutions at a particular place and a particular time can inform others
Problems
• Organizational goals can be catalysts for identifying and responding to the intractable
problems within your setting
Overarching • There is power in translating your goals into questions that guide assessment and action.
Questions
• Identifying available data embedded within normal activities of the organization provides a
rich source of information
Assessment • Using an assessment cycle transforms data to actionable information
Strategies
Decision
Making
• Organizations become more strategic when they base actions on meaningful data
• Learning organizations seek improvement by gathering information about the effectiveness of
their actions in an assessment cycle.
Intractable
Problems
1.
2.
3.
4.
5.
6.
• What are some intractable human issues that
always underlie problem solving in education?
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
Intractable
Problems
• Our (student & society)
intractable problems
1.
2.
3.
4.
5.
6.
Literacy
Economic health/poverty
Diversity
Health
Family
Power
Overarching
Questions
• How can we effectively support
student success here?
• Underlying questions
– Who?
– What?
– When?
– Where?
– Why?—Mission statement
– How?
Conceptual framework indicating influences on student learning
The existing knowledge,
understanding, abilities, motives
and conceptions/ styles of learning
students already have
How students
approach learning
and studying
How students perceive
the teaching-learning
environment
Quality of learning
achieved
How course material is
selected, organized,
presented, assessed
Constructive
alignment
How a teaching-learning
environment is designed
and implemented
What students are expected to
learn and understand
University teachers’ ways of
thinking about teaching
Noel Entwistle, ESRC/TLRP ETL
report 3, March 2003
Overarching
Questions
1.
2.
3.
4.
5.
6.
• What are your institution’s/program’s
basic or overarching questions?
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
Assessment
Strategies
•Levels of on-going assessment
Classroom
Program/
Department
Institution
Assessment Cycle
Outcomes
(What do we want
to achieve?)
Improvement
(Now what? Policy,
resource changes?)
Assessment criteria
Your
institution
(How will we know if we
have done this well?)
Mission/Values
Learning – So what?
(What did we learn?
What does it mean?)
Assessment activities
(Design, carry out)
Assessment results
(What happened?)
Suite of BYU Instruments
Internal
BYU
Developed
BYU
Freshman
Surveys
Non-Returning
Student Survey
•Pre-arrival
•End-of-year
Senior
Survey
Employers of
BYU Graduates
Survey
Alumni
Questionnaire
AQ2
(10 year)
Aims of a BYU Education
Foundation
Timeline
Freshman Sophomore Junior
External
BCSSE
Key
Graduate
Student
Exit Survey
Internal /
In use
NSSE
Senior Grad Std
NSSE
Internal / In
Development
Internal /
Planned
External
Post-college
Assessment
Strategies
• What kinds of data do you already have?
• What kinds of data do you need?
Data you have
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
Data you need
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
Overarching
Questions
Assessment
Strategies
• How can we effectively support student success
here?
• What data do we already have?
• What data do we need?
• Note: each word in the question directs
our efforts:
•
•
•
•
•
•
•
How
Can
We
Effectively
Support
Student success
Here
Assessment
Strategies
• Who matters at my institution?
Students
Parents
Faculty
Freshman
Academy
Academic
Administrators
Support
Administrators
Assessment
Strategies
1.
2.
3.
4.
5.
6.
• Who matters at my institution?
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
Assessment
Strategies
•What matters to us?
Freshman Academy Mission Statement
Freshman Academy (FA) is a one-semester program that
helps first-year students make important connections within
the university during their first semester at BYU. By creating
the best teaching-learning environment possible, FA strives
to provide an academic environment that facilitates the
development of well-rounded students and bridges the gap
between high school and college. FA does this by helping
students connect with each other, faculty members,
academic majors, the University Core, The Aims of a BYU
Education, and other university resources.
Assessment
Strategies
1.
2.
3.
4.
5.
6.
• What matters at my institution?
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
_____________________________________
Freshman Academy Mission
Bridging the Gap
Student
Peer mentor
Aims of BYU Education
Academic
Experience
Faculty
Other
students
• Spiritually strengthening
• Intellectually enlarging
• Character building
• Life-long learning & service
Academic majors
University resources
Development of the total Person
What matters to stakeholders?
Cost Effectiveness
High impact
Low impact
High cost
Low cost
 Student retention
 Student performance (grades)
 Student learning
Expectations
When do things matter?
1 yr
2 yr
3 yr
Time spent at the university
4 yr
Assessment
Strategies
•When do you do things?
What things do you do when?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
What do you know about them?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
Assessment
Strategies
• What do you know about student
success at your institution?
What things do you know?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
How do you know them?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
Analysis of Existing Data
Criteria
Points in time
Pre-semester
Week 3
Demographics
Background
•BCSSE
Expectations
Follow-up 2nd
Semester
Learning Habits
•BCSSE
•Learning Goals
Inventory
•Boyer Model
•High school
engagement
•FA interactions
•Housing placement
•Approaches to study
•Time spent in high
school
•Time logs
•PM “How well do
you know”
•Meals with faculty
•Time logs
•Weekly surveys
•Classroom
observations
•Time logs
•Socio-grams
•Meals with faculty
•Weekly surveys
•Classroom
observations
•Peer mentor
reflection summaries
•Final learning papers
•Student reflections
•Peer mentor
reflection summaries
•Modified Temple
survey
•Use of campus
resources
•NSSE
•Peer mentor story
cycles
•Use of time
•(ETL)
•Peer mentor story
cycles
Mid-term
End of
Semester
Engagement
Connections
•End of semester
survey
•Student reflections
•Peer mentor
reflections
summaries
Performance
•ACT
•GPA/EGPA analysis
•Summer reading
reflection
•Student reflections
•Peer mentor
reflection summaries
Assessment
Strategies
Decision
Making
• What do you know about: How can we effectively
support student success here?
• What do we do now?
What do you know?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
What can you do?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
Assessment
Strategies
•Students who matter
Significant time
preparing for class
High ACT
High
HS GPA
Low
HS GPA
Low ACT
Low level
Engagement
High level
Engagement
Limited time
preparing for class
Assessment
Strategies
Decision
Making
• What do you know about: How can we effectively
support student success here?
• What do we do now?
What do you know?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
What can you do?
1.
2.
3.
4.
5.
6.
________________
________________
________________
________________
________________
________________
Using typical activities and timelines
Before
semester
•
•
•
•
•
Application
Registration
Autobiographies
Surveys
Contact
First 3 weeks
Mid-term
• Interviews
• How well do you
know?
• Faculty meetings
• Student
engagement
End of
Semester
• Interviews
• Transition ratings
• Student
engagement
Closing the gap
from data to decision making
• Our world is confronted with problems that occur over and over and are difficult to address
Intractable • Providing solutions at a particular place and a particular time can inform others
Problems
• Organizational goals can be catalysts for identifying and responding to the intractable
problems within your setting
Overarching • There is power in translating your goals into questions that guide assessment and action.
Questions
• Identifying available data embedded within normal activities of the organization provides a
rich source of information
Assessment • Using an assessment cycle transforms data to actionable information
Strategies
Decision
Making
• Organizations become more strategic when they base actions on meaningful data
• Learning organizations seek improvement by gathering information about the effectiveness of
their actions in an assessment cycle.