Three Ways to Evaluate your
Program:
LOGIC MODEL
SWOT ANALYSIS AND ACTION PLAN
STUDENT LEARNING OUTCOMES
Dr. Patricia Fullmer
College Reading and Learning Association Annual Conference
November, 2011
Agenda
 Overview of Evaluation
 Logic Model
 SWOT Analysis and Action Plan
 Student Learning Outcomes
Learning Outcomes
 Gain an overview of evaluation.
 Create a logic model.
 Generate a SWOT analysis and action plan.
 Craft student learning outcomes.
Overview of Evaluation: Types
Summative:
 Document
the results of the program.
Formative:
 Improve
the program.
Why Evaluate?
 Required by Funding
Sources:



Government
Grants
University/College funds
 Required by Accrediting
Organizations:


Middle States, Southern
States, etc.
Council for the
Advancement of
Standards in Higher
Education (CAS)
 Self-Improvement:
 Determine how and where
to improve your program’s
services.
 Document Effectiveness:
 Demonstrate your
program is valuable and
worthwhile!
How do you conduct Evaluation?
 Evaluation Process:
1. Who does the evaluation?

Evaluation committee/staff
2. Evaluation Design:





Determine evaluation questions
Determine what information to collect and when
Determine the source, methods and
instrumentation of data collection
Examine information: Analysis and Interpretation
Generate results
3. Use of evaluation results
Characteristics of Good Evaluation
 Comprehensive
Uses
variety of methods
Uses multiple sources
Considers multiple perspectives
Uses comparisons
 Systematic
 Continuous
Methods and Instruments
Direct Measures
Indirect Measures
 Pretest/Posttest
 Self Reports:
 Projects
 Portfolios
 Professional subject
area assessments




Surveys
Interviews
Focus Groups
Questionnaires
 Grades
Collecting Data: Considerations
 Validity
 Reliability
 Quantitative
 Qualitative
 Rubrics
 Protocols
 Population Sampling
 Timing-When to collect data
Use of the Results of Evaluation
1.
2.
Demonstrate effectiveness
Make decisions for ongoing
improvement:
Modify goals and processes
Determine an
improvement/action plan
Allocate funds
Apportion resources
Logic Model
THE INITIAL STEP IN THE
EVALUATION PROCESS
What is a Logic Model?
 Describes your program visually
 Picture of the program’s: “Theory of Change”:
 Events & activities that are expected to create a change, resolve
a problem-describes the desired results
 Shows what leads to the desired results:
 Actions
Results
Inputs
Outputs
Outcomes
Benefits of a Logic Model
 Builds mutual understanding
 Helps initiate an evaluation:



Determine the wanted outcomes
Decide what to measure and assess
Establish at what point in time to conduct the evaluation and
collect the data
 Summarizes the program



Improves planning
Identify gaps in services
Guides use of resources and funds
Qualities of a Logic Model
 Group Process
 Meaningful
 Reasonable
 Realistic/Attainable
 Measurable
Components of a Logic Model
 Inputs
 Outputs
 Outcomes/Impact
 Assumptions
 External Factors
Where to Start?
AT THE END!!! THEN PROCEED BACKWARDS!!!
1.
2.
3.
4.
What are the desired outcomes?
Who will be the participants?
What will be the services and
activities?
What are the resources needed?
Learning Resource Center Logic Model
Inputs
Outputs
Activities
What we invest:
Participation
What we do: Who we
reach:
Funding:
University funds
Title III funds
Act 101 funds
Tutoring Labs
Online tutoring
programs
Staff:
Tutors
Peer Tutors
Secretary
Administrators
Review minilessons
Training for staff-CRLA
Certified
Writing Portfolio
assistance
Online programs and access
codes
Mini-Lesson plans and
exercises
Workshops
Individual
tutoring
Students
Students’ Families
Faculty
Administrators
Alumni
Summer Bridge
Peer Tutoring
Collaboration with academic
departments and
administrators
Outcomes -- Impact
Short
Medium
This
semester’s
results:
Next
Results in 4-6
semester/next years and more:
year’s results:
Students
Students
Students increase successfully
increase their their GPA
graduate Lincoln
skills to
with skills for
college level Students persist graduate school or
and return the
the fast track in a
Students earn next
corporation
a B or above semester/next
in their
year
Lincoln’s
course
graduation rate
Students become increases
Students
successful
increase life academically,
Lincoln’s
skills and
socially, and
reputation
coping skills emotionally at
increases
Lincoln
Students
More and better
adjust to
Lincoln’s
prepared students
college life
retention rate
are attracted to
increases
Lincoln
Act 101 Program
Assumptions:
Following best practices in our services will help students increase their skills.
Increased student skills will improve retention and graduation rates.
Increased graduation rates will help attract students and increase Lincoln’s reputation.
Perception of LRC by Faculty, Administrations, and Students
Long
Growth of Lincoln
External Factors:
Location
Economy
State and federal funding
University of Wisconsin-Extension (2003). Enhancing program performance with logic models. Retrieved from lmcourse@ces.uwex.edu
Writing Lab Logic Model
Inputs
Outputs
Activities
Participation
What we invest:
What we do: Who we reach:
Professional Writing Tutors
Tutoring Labs
Students
Mini review grammar
lessons and exercises
Writing Labs that
address grammar
Students’ Families
Faculty
Required tutoring
laboratories
Drop-in tutoring
Administrators
My Writing Lab software
Appointment
tutoring
Individual assistance for all
students in writing
Writing Portfolio
Assistance
Conducive lab environment
Student
development
workshops
Computer and Internet
Access
Alumni
Assumptions:
Following best practices in our services will help students increase their skills.
Increased student skills will improve retention and graduation rates.
Increased graduation rates will help attract students and increase Lincoln’s reputation.
Perception of LRC by Faculty, Administrations, and Students
Outcomes -- Impact
Short
This
semester’s
results:
Medium
Long
Next
Results in 4semester/next 6 years and
year’s results: more:
Students gain an Ability to construct
understanding of a thesis driven
the structure of essay
an essay
Students utilize the
Ability to
LRC on a regular
construct a topic basis
sentence and
thesis statement Expansion of
with areas of
students’
support
vocabulary
Students are able
to write
proficiently
across the
curriculum
Satisfaction of
writing portfolio
requirements
Ability of
students able to
General
Ability to conduct write a
knowledge of
research properly comprehensive
proper grammar
and coherent
and punctuation Understanding of response to
proper work
graduate level
Faculty
citation
exams
collaboration
External Factors:
Location
Economy
State and federal funding
University of Wisconsin-Extension (2003). Enhancing program performance with logic models. Retrieved from lmcourse@ces.uwex.edu
SWOT Analysis and Action Plan
AN INTERNAL AND EXTERNAL
PERSPECTIVE OF YOUR
PROGRAM
SWOT Analysis and Action Plan
SWOT Analysis
Action Plan
 Strengths
 Build on strengths
 Weaknesses
 Opportunities
 Threats
 Take advantage of
opportunities
 Address weaknesses
 Address threats
Writing Lab SWOT Analysis
Strengths: What is done well?
Evaluation of success indicators has resulted in collaboration on
content goals and agreement to move to a mastery system
addressing core skills of college English grammar.
Weaknesses: What could be improved?
Factors for poor student attendance include the following
potential causes:
1. Since students are not given credit for labs.
Tutors developed several writing workshops that were presented in
a broad range of academic disciplines.
2. The LRC continues to have a credibility issue.
Tutors make classroom visits at beginning of semester.
3. Students do not receive a letter grade for the lab.
Tutors continue to offer extended hours for make-up labs and dropins for students seeking professional tutoring.
4. There is no uniformity among professors teaching the
course about how best to incorporate lab work in the
overall grade for the course.
The Writing Lab has a professional staff that is dedicated to
nurturing student success.
Students with low proficiency in reading and writing are often
hesitant to participate in lab sessions.
All of the tutors have earned CRLA Level I & II certification and
continue training on a regular basis each semester.
One weekly 50-minute lab session is insufficient time.
Opportunities: What are the
opportunities open?
Threats: What are the obstacles?
Design student development workshops and increase in-class
presentations of workshops in order to improve students’ skills and
knowledge of the writing process and expand the visibility of the
LRC.
Creation of a mastery system addressing core skills of college
grammar, to improve students’ proficiency in reading and writing.
Lack of competitive salaries or salary raises contributes to low
tutor retention and high tutor turnover.
Some of the university’s faculty population holds a poor perception
of the LRC which results in lack of support from individual faculty
or entire departments.
Success students project a negative attitude towards the Writing
Lab which diminishes students’ motivation and results in a decline
in attendance during the semester.
Writing Lab Action Plan
Improvement
Evidence
The Writing Lab will focus on grammar mastery. The student will
be required to master 20 identified areas of the 26 total areas of the
Study Plan in My Writing Lab at 85% to be eligible to test out of the
lab.
Campbell, M. (1983). "Mastery Learning in the College Learning
Center." Paper presented at the National Association for
Remedial/Developmental Studies Conference, Little Rock,
AK, March 1983, 19 pp. (ED 247 592).
To test out of lab, students in English 098 and English 099 will be
The study found that students need to understand the task at hand,
required to score 85% on the post-test at midterm time. Students will the process, and the assessment involved in order for there to be
be allowed to take the post test twice.
successful learning.
Lab tutors will focus on sentence grammar areas in My Writing Lab
work.
Input from Lincoln University English Department Composition
Committee.
Tutors will review sentence grammar facts twice in the semester.
In weeks 6 and 11, the tutors will review previously covered
material.
Adult Literacy and Basic Skills Unit (1991). A self-directed
approach to English grammar. Kingsbourne House, 229231 High Holborn, London, England WC1V 7DA, United
Kingdom.
The Writing Lab will increase its hands-on activities to help
facilitate the mastery topic of punctuation in order to improve
students’ writing aptitude.
Boylan, H. (2002). What works: Research-based best practices in
developmental education. Boone, NC: Continuous Quality
Improvement Network/National Center for Developmental
Education.
Students will write sentences in lab.
Accommodate diversity through varied instructional methods. Use
technology with moderation.
Brophy, L. (1988). Educating teachers about managing classrooms
and students. Teaching and Teacher Education, 4(1), 3.
This study found that increased time on task improved students’
skills.
Information from the student survey.
Student Learning Outcomes
AN EVALUATION OF THE IMPACT OF
YOUR PROGRAM
Benefits of Student Learning Outcomes
 Fulfills accreditation requirements
 Fulfills funding requirements
 Establishes goals
 Tells us what success will look like!
SMART Outcomes
 Specific
 Measureable
 Attainable
 Results-oriented
 Timed
Writing Lab Student Learning Outcomes
Student Learning
Outcomes
Assessment
Method
Students will increase
their skills in the area of
sentence grammar.
Results of pre and For students who
post
attended six or more
diagnostic tests.
sessions:
English 098 students will
show a 50% improvement
from the pre test to the
post test in sentence
grammar for those who
attend six or more
sessions.
English 099 students will
show a 33% improvement
from the pre test to the
post test in sentence
grammar for those who
attend six or more
sessions.
Findings
English 098 students
show a 50%
improvement in
sentence grammar.
English 099 students
show a 47%
improvement in
sentence grammar.
Modified
Student
Learning
Outcomes
Action Plan
Students will increase
their skills in the area
of sentence grammar.
Students will improve
sentence grammar by:
English 098 students
will show a 53%
improvement from the
pre test to the post test
in sentence grammar
for those who attend
six or more sessions.
English 099 students
will show a 50%
improvement from the
pre test to the post test
in sentence grammar
for those who attend
six or more sessions.
Focusing on mastering
all sentence grammar
areas in My Writing
Lab.
Write sentences in lab.
Receive tutor
assistance with the
online program.
Tutors giving students
two review lessons
during the semester to
help reinforce grammar
facts.
References
Adult Literacy and Basic Skills Unit (1991). A self-directed approach to English grammar.
Kingsbourne House, 229-231 High Holborn, London, England WC1V 7DA, United Kingdom.
Boyd, H. (2002, November 22). Use comparison groups to strengthen your evaluation. (Audio
Conference). Retrieved from: http://www.uwex.edu/ces/pdande/resources/pdf/agenda.pdf
Boylan, H. (2002). What works: Research-based best practices in developmental education.
Boone, NC: Continuous Quality Improvement Network/National Center for Developmental
Education.
Brophy, L. (1988). Educating teachers about managing classrooms and students. Teaching and
Teacher Education, 4(1), 3.
Campbell, M. (1983). Mastery learning in the college learning center. Paper presented at the
National Association for Remedial/Developmental Studies Conference, Little Rock, AK,
March 1983, 19 pp. (ED 247 592).
References (con’t)
Council for the Advancement of Standards in Higher Education (2008). CAS Self-assessment guide for
learning assistance programs. CAS: Washington, DC.
Council for the Advancement of Standards in Higher Education (2008). History of CAS.
from http: //www.cas.edu
Dwyer, C. A., Millett, C. M., & Payne, d. G. (2006). A culture of evidence: Postsecondary assessment
and learning outcomes. Princeton, NJ: Educational Testing Service.
Kellogg, W. K. (2004). Evaluation handbook. . W. K. Kellogg Foundation: Battle Creek, Michigan.
Kellogg, W. K. (2004). Logic model development guide. W. K. Kellogg Foundation: Battle Creek,
Michigan.
Nichols, J. O. (2008). The administrative unit assessment handbook: Guidelines for planning and
implementing. Measuring student support services & administrative outcomes. Orlando, FL:
University of Central Florida. Retrieved from: http://oeas.ucf.edu.doc/adm_assess_handbook.pdf
Strategy: Create and implement the best strategy for your business, (2005). Boston, MA: Harvard
Business School Press.
References (con’t)
SWOT analysis II: Looking inside for strengths and weaknesses, (2006). Boston, MA: Harvard Business
School Press.
Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: Teaching and training guide.
Madison, WI: University of Wisconsin-Extension, Cooperative Extension, Program Development
and Evaluation.
University of Wisconsin-Extension (2003). Enhancing program performance with logic models.
Retrieved from lmcourse@ces.uwex.edu
Wargo, M. C. (2007). Handbook for program assessment in administrative/educational support units.
Western Carolina University Cullowhee, NC. Retrieved from:
http://www.wcu.edu/WebFiles/PDFs/AssessmentHandbook_AES.pdf
Weiss, C. H. (1972). Evaluation research: Methods for assessing program effectiveness. Prentice-Hall,
Inc.: Englewood Cliffs, NJ.