ASSESSEMENT METHODS
AND ACTIONABLE DATA
Richard Mahon, Riverside City College,
facilitator
 Lesley Kawaguchi, Santa Monica College
 Maggie Taylor, Fresno City College
 Linda Umbdenstock, member, ASCCC-RP SLO
Collaborative

AUTHENTIC ASSESSMENT

“. . . [A]uthentic assessment simulates a real
world experience by evaluating the
student’s ability to apply critical thinking
and knowledge or to perform tasks that
may approximate those found in the work
place or other venues outside of the
classroom setting.” SLO Terminology
Glossary: A Resource for Local Senates, pp.
2-3
SAMPLE STUDENT LEARNING OUTCOME:

History: “Demonstrate the ability to
interpret historical information by applying
analytical skills used by historians – such as
synthesizing evidence from both primary
and secondary sources, comparing and
contrasting multiple perspectives,
contextualizing information, and/or
identifying causes and effects of change and
continuity – to the course content.”
POSSIBLE WAYS TO ASSESS:
Essays
 Research Papers
 Midterm or Final Exams
 Oral Discussion and/or Presentation

RUBRIC DEVELOPED BY THE FACULTY
REGISTERED NURSING SLOS

Fundamentals of Nursing Skills Lab
SLO: “Provide safe, comprehensive and proficient
care for the adult client in a simulated environment.”
 Each procedure has a rubric.
 At end of course, a patient scenario is given which
involves a set of procedures.

Performance of procedure, including rationale
 Documentation


Core RN faculty discussed the rubrics, the
scenarios, and use the results to make any
changes.
ENGL 252: WRITING IMPROVEMENT
Assigned English faculty met several times,
developed rubric, and exam.
 Held several norming sessions to assist in
evaluating papers.
 When finished, had an overall critique of the
exam.
 Results will be used in the next semester.

IDENTIFYING ACCESSIBLE EXTERNAL
SOURCES OF DATA
Chancellor’s Datamart
http://www.cccco.edu/ChancellorsOffice/Divisio
ns/TechResearchInfo/MIS/DataMartandRepo
rts/tabid/282/Default.aspx
• NCES National Center for Education
Statistics
http://nces.ed.gov/
• CPEC California Postsecondary Commission
http://www.cpec.ca.gov/
• CalPASS - http://www.cal-pass.org/
•
IDENTIFYING INTERNAL DATA SOURCES
Program
completion
reported
Course and
student service
data reported
Data gets
into college
MIS
reporting
Actionable data
analyzed reported
STUDENTS PROGRESS IN A NON-LINEAR FASHION
DEFINING ACTIONABLE DATA – USING DATA
INTERACTIVELY




Data collection does not equate to action or improvement.
Even the most valid and reliable data are not a substitute for
action and will not by themselves motivate action.
Actionable data provide information that leads to improved
practice.
Actionable data result from “action-oriented” research.
ACTIONABLE DATA?
ARCC DATA
Indicator
Student Progress & Achievement
Completed 30 or more units
Fall to Fall Persistence
Voc Ed Course Completion
Basic Skills Course Completion
Basic Skills Course Improvement
ESL Course Improvement
Is this data actionable?
Statewide Rates
2009
2008
51.8%
51.2%
71.2%
70.4%
69.2%
68.3%
77.7%
78.2%
60.5%
60.5%
51.2%
50.0%
50.1%
44.7%
2010
52.3%
72.4%
68.7%
77.6%
61.5%
53.8%
50.2%
ACTIONABLE DATA?
Basic Skills Success Rates
African-American
Asian
Decline to state
Filipino
Hispanic
Native American
Pacific Islander
Unknown
White
Total
Is this data actionable?
31.3%
75.0%
68.4%
62.5%
56.1%
59.7%
ACTIONABLE DATA?
Is this data actionable?
ACTIONABLE DATA?
Basic Skills Success at Levels prior to Transfer
English English English
3 levels prior
Pre-Intermediate
2 levels prior
67%
Intermendiate
1 levels prior
40% 74%
High Intermediate
transfer
25% 46% 79% Transfer
16% 29% 42%
Is this data actionable?
ESL ESL ESL
Math Math Math
68%
Fundamental 65%
35% 69%
Algebra
32% 67%
22% 37% 72% Intermediate 11% 27% 72%
11% 13% 31%
5% 13% 31%
STUDENT LEARNING OUTCOMES DATA
ESL Level 1 Outcomes Assessments % passed

Listening/Speaking
SLO: Demonstrate understanding of frequently used words, phrases and questions in
familiar contexts. Engage in limited social conversations to communicate basic
survival needs.

Assessment = Interview

Outcomes = 84% passed successfully

Reading

SLO: Construct meaning from simplified print materials on familiar topics.

Assessment = CASAS Level A (A comprehensive standardized test)

Outcomes = CASAS Success rate 73%


Writing

SLO: Produce simple sentences in paragraph format and complete simple forms.

Assessment = Written Paragraph and Rubric for scoring

Outcomes = 74% pass rate
EXPLORING DATA - DATA 101
Principle 1 – Use longitudinal data when possible
Principle 2 – Use data in context
Principle 3 – Look for both direct and indirect data
Principle 4 – Do not oversimplify cause and effect of data
Principle 5 – Use appropriate levels of data for appropriate levels of
decisions
Principle 6 – Perception is the reality within which people operate
Principle 7 – Use of data should be transparent
Principle 8 – Consider carefully when to aggregate or disaggregate data
Principle 9 – Focus on data that is actionable
Principle 10 – Consider implications and the “What if?
CREATING INSTITUTIONAL PROCESSES FOR
CONDUCTING RESEARCH AND USING DATA
TO INFORM PRACTICE
-Information Capacity Challenges
-Research Processes and Procedures
-Research Agendas
-Action Research Guided Questions
INFORMATION CAPACITY CHALLENGES

Building an Evidence-based Infrastructure




Keeping Up with the Demand




Managing and responding to myriad requests
Maintaining quality and integrity of data process
Making data and information widely accessible
Responding to heightened accountability
mandates
Linking research to (resource) planning
Supporting data-driven decision-making
Turning Data into Action



Making data available and applicable at all levels
Making sense of and taking action on the data
Building a Culture of Inquiry
PROCESSES & PROCEDURES
Guidelines for use of data and information
 Protection of human subjects policy
 Review panels and committees
 Request and fulfillment procedures
 Criteria for prioritizing ad hoc requests
 Linking requests to broader goals & initiatives
 Creating and using Research Agendas

RESEARCH AGENDAS

College-wide Research Agenda
Supports major college-wide initiatives & activities
 Tied to college-wide plan (goals & priorities)
 Includes recurring requests


Topical Research Agenda
Focused on a single topic or group of interest
 Tied to a specific initiative or activity
 Fewer research activities than college-wide

ACTION RESEARCH GUIDED QUESTIONS
Developing the Research Agenda
1.
What and who will be researched?
2.
How is research tied to college plans, goals, initiatives and/or activities?
3.
How will the information be used, by whom and how often?
4.
Which methodology or approach will be used?
Turning Data into Information
1.
What do the data tell us?
2.
Which questions were fully answered by the research and which need more
exploration?
3.
What are reasonable benchmarks based on the research?
Taking Action on the Information
1.
What interventions or strategies do we need to deploy in order to move the needle?
2.
How should this information be shared and applied across the college?
DEVELOPING A CULTURE OF
COLLABORATIVE INQUIRY



-Operational Definition
-Building a Culture of Inquiry
-Data Integration & Inquiry Strategies
CHARACTERISTICS OF EVIDENCE
Good evidence used in evaluations has the following characteristics:
1.
It is intentional, and a dialogue about its meaning and relevance
has taken place.
2.
It is purposeful, designed to answer questions the institution has
raised.
3.
It has been interpreted and reflected upon, not just offered up in
its raw or unanalyzed form.
4.
It is integrated and presented in a context of other information
about the institution that creates a holistic view of the institution
or program.
5.
It is cumulative and is corroborated by multiple sources of data.
6.
It is coherent and sound enough to provide guidance for
improvement.
Source: ACCJC/WASC Guide to Evaluating Institutions,
2009
EVIDENCE AND INQUIRY
Culture of Evidence
Data are provided for
multiple purposes.
 Data demonstrate
performance &
outcomes.
 Data are accessible to
all constituency
groups.
 Data are continuously
distributed.

Culture of Inquiry
Dialogue centers
around data/evidence.
 Dialogue focuses on
taking action.
 Dialogue is open and
collaborative.
 Dialogue is continuous
and widespread.
 Dialogue is reflective
and dynamic.

BUILDING A CULTURE OF
INQUIRY
1.
2.
3.
4.
Make evidence and inquiry the paradigm.
Communicate often, widely and clearly.
Embed the cycle of collectively assessing,
planning, implementing, re-assessing and replanning…..
Create opportunities for continuous
collaboration with multiple and mixed
constituency groups.
WHAT MIGHT THIS LOOK LIKE?
ORGANIZING DATA: WEAVE, TRAKDAT,
TASKSTREAM, ELUMEN
“The New Guys in Town”
QUESTIONS?

Thank You
Please fill out the evaluations