Student Learning
Outcomes: Stepping
Stones to Academic
Program Success
Dr. Mia Alexander-Snow, PhD
Day 2: Assessment Workshop
Florida Institute of Technology
May 25, 2010
1
Workshop Agenda
• Assessment Overview
• Program Assessment Process
• Program Assessment Strategy
• College Competencies & General Education
Assessment
• Program Major Learning Goals (Objectives)
• Outcome Statements: Think SMART
• Assessment Methodology is MATURE
• Data collection
• “Closing the Loop” Program Improvement
2
Workshop Goals
Participants will
• be able to identify the elements of a student learning outcome
statement;
• develop student learning outcome statements for a specific
program and/or course;
• be able to identify the elements of an assessment methodology
for an outcome;
• be introduced to curriculum and program mapping;
• develop an assessment methodology for a program or course
outcome;
• select direct measures for outcomes which have the potential to
provide useful data that lead to program improvement, and
• learn about closing the outcomes assessment loop.
3
Assessment & Accreditation
Overview
4
What is Assessment?
Assessment is …
the systematic collection, review, and use of information about
educational programs undertaken for the purpose of improving
learning and Development (Palomba, C.A. & Banta, T.W. (1999).
Assessment Essentials p. 4).
The results gathered in the Assessment process are not to be used to:
•
impact the individual evaluation process
•
undermine or impinge on academic freedom or professional rights
• replace or alter the collective bargaining process
5
Assessment is a Collective
Process
President
Key elements
•
Cultural change
•
Commitment from upper
administration
•
Use of existing organizational
structures to implement
planning and assessment
•
Participation from all levels
•
Long term commitment
•
Capability to measure
objectives related to
student learning outcomes
and related activities
•
Dissemination of results and
information
Vice
Presidents
College Deans
University
Assessment
Committee
Faculty, Directors,
Departmental/Program
/Unit Assessment
Coordinators, Staff
6
What is Accreditation?
• Accreditation is. . .
the procedure by which an authoritative body gives formal recognition that a body
or person is competent to carry out specific tasks. In a certification system, an
accreditation body will accredit or, in simpler language, approve a certification
body as competent to carry out certification
http://www.goldenwestcollege.edu/iec/assessment.html).
• Accreditation has two fundamental purposes:
1. to assure the quality of the institution or program, and
2. to assist in the improvement of the institution or program.
http://www.collegedegreeguide.com/articles-fr/accredited-collegeuniversity.htm
7
The Assessment Process supports
SACS Core Requirements and Comprehensive
Standards
8
SACS Core Requirements
Compliance is essential for gaining and maintaining accreditation with the
Commission on Colleges. To maintain accreditation, an institution must
meet all Core Requirements (2.1-2.11, including 2.12—QEP)
2.5
The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation
processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in
continuing improvement in institutional quality; and (3) demonstrate the institution is effectively
accomplishing its mission. (Institutional Effectiveness)
2.7.3
In each undergraduate degree program, the institution requires the successful completion of a general
education component at the collegiate level that (1) is a substantial component of each undergraduate
degree, (2) ensures breadth of knowledge, and (3) is based on a coherent rationale. For degree completion in
associate programs, the component constitutes a minimum of 15 semester hours or the equivalent; for
baccalaureate programs, a minimum of 30 semester hours or the equivalent. These credit hours are to be
drawn from and include at least one course from each of the following areas: humanities/fine arts,
social/behavioral sciences, and natural science/mathematic.
2.12 The institution has developed an acceptable Quality Enhancement Plan (QEP) that includes an institutional
process for identifying key issues emerging from institutional assessment and focuses on learning outcomes
and/or the environment supporting student learning and accomplishing the mission of the institution.
(Quality Enhancement Plan)
9
SACS Comprehensive Standards
The Comprehensive Standards are more specific to the operations of the
institution, represent good practice in higher education, and establish a level
of accomplishment expected of all member institutions. If an institution is
judged to be significantly out of compliance with one or more of the
Comprehensive Standards, its reaffirmation of accreditation may be
denied.
10
SACS Comprehensive Standards
3.3 Institutional Effectiveness
•
3.3.1 The institution identifies expected outcomes, assesses the extent to
which it achieves these outcomes, and provides evidence of improvement
based on analysis of the results in each of the following areas:
– 3.3.1.1 educational programs, to include student learning outcomes
– 3.3.1.2 administrative support services
– 3.3.1.3 educational support services
– 3.3.1.4 research within its educational mission, if appropriate
– 3.3.1.5 community/public service within its educational mission, if appropriate
3.3.12
The institution has developed a Quality Enhancement Plan that (1) demonstrates institutional
capability for the initiation, implementation, and completion of the QEP; (2) includes broadbased involvement of institutional constituencies in the development and pro- posed
implementation of the QEP; and (3) identifies goals and a plan to assess their achievement.
11
SACS Comprehensive Standards
3.4 Educational Programs: All Educational Programs (includes all on- campus, offcampus, and distance learning programs and course work) (See Commission
policy “Distance and Correspondence Education.”)
3.4.1
The institution demonstrates that each educational program for which academic
credit is awarded (a) is approved by the faculty and the administration, and (b)
establishes and evaluates program and learning outcomes. (Approval of
Faculty)
3.5 Educational Programs: Undergraduate
3.5.1 The institution identifies college-level competencies within the general education
core and provides evidence that graduates have attained those competencies.
(Competencies with General Education Core)
Federal Requirements
4.1 When evaluating success with respect to student achievement in relation to the
institution’s mission, the institution includes, as appropriate, consideration of course
completion, state licensing examinations, and job placement rates.
(Evaluation of Student Achievement)
12
The Assessment Process Supports
Student Learning
13
Type
Systemwide Assessment
Level
Community college, 4-Year, or University System
Goal
To provide public accountability
Type
Institutional Assessment
Level
Individual College or university
Goals
To provide accountability; to gain and maintain accreditation
Type
Level
Goals
Program Review and Assessment
School, program, or department
To gain and maintain accreditation;
to improve program quality and success
Type
Level
Goals
Classroom Research
Individual course
To understand and improve teaching and learning in that individual
course
Type
Level
Goals
Classroom Assessment
Individual class meeting or assignment within the course
To understand and improve teaching and learning of a single lesson
or unit
14
Developing an Effective
Assessment Process
15
9 Principles of Assessment Process
(http://www.aahe.org/assessment/principl.htm)
The Nine principles include:
1. The assessment of student learning begins with educational values.
2. Assessment is most effective when it reflects an understanding of learning as multidimensional,
integrated, and revealed in performance over time.
3. Assessment works best when the programs it seeks to improve have clear, explicitly stated
purposes.
4. Assessment requires attention to outcomes but also and equally to the experiences that lead to
those outcomes.
5. Assessment works best when it is ongoing, not episodic.
6. Assessment fosters wider improvement when representatives from across the educational
community are involved.
7. Assessment makes a difference when it begins with issues of use and illuminates questions that
people really care about.
8. Assessment is more likely to lead to improvement when it is part of a larger set of conditions that
promote change.
9. Through assessment, educators meet responsibilities to students and to the public.
16
Developing an Effective Assessment Process
The Assessment of Student
learning is a participative, interative
process that. . .
• Provides data/information you need on your students’ learning.
• Engages you and others in analyzing and using this
data/information to confirm and improve teaching and learning
• Produces evidence that students are learning the outcomes you
intended
• Guides you in making educational and institutional improvements
• Evaluates whether changes made improve/impact student
learning, and documents the learning and your efforts (The Higher
Learning Commission ).
17
Implementing an Effective Assessment Program
Strategy
(Nichols 5-Step Model)
• Program (unit) mission statement
• Program (unit) desired outcomes or objectives
• How many should be required ( 1, 2,3)?
• What number & type of outcomes (Operational and/or SLOs)?
• Measurement approaches
• How many per outcome? (1, 2, 3)?
• What type & number of methods is acceptable (1 indirect and 1 direct;
2 directs; 2 directs and 1 indirect)?
• Measurement results
• Planned use of results to improve program (unit)
• Implemented quality improvements
18
Definitions
Program Assessment:
setting and assessing outcomes that will promote program
improvement
Types of outcomes:
• Outcome: A specific, measurable statement that describes
desired performance.
• Operational outcome: A type of outcome that addresses
operational or procedural tasks, such as functions, demand,
resources, efficiencies, and satisfaction
• Learning outcome/Student Learning Outcome
(SLO): A type of outcome that describes
a.
b.
intended learning that students should meet on the way to
attaining a particular degree, called Student Learning Outcomes.
intended learning that participants meet as a result of a unit’s
program(s) or services(s), called learning outcomes
19
Steps 1-3: Defining Program Major
Mission, Vision, Goals
(Objectives), & Outcome
Statements
20
Step 1: Defining Program Major Mission & Vision
• Who are we?
– Name of the office, department, or unit
• What do we do?
– Your unit’s primary purpose
• For whom do we do it?
– The stakeholders or customers of your unit
22
Step 1: Defining Program Major Mission & Vision
Mission Statement
• Brief (75 words or less) and comprehensive
• It should make sense to someone who knows little or nothing
about your unit.
• It should rarely need revision and be able to endure changes
in leadership.
• It should lead to the development of goals, outcomes and
performance measures for those outcomes.
23
Step 1: Defining Program Major Mission & Vision
The mission structure:
“The mission of (your office name) is to (your primary
purpose) by providing (your primary functions or
activities) to (your stakeholders).” (You may add
additional clarifying statements.)
*NOTE: the order of the pieces of the mission statement may vary from
the above structure
24
Mission Statement Example
Office Name
The Mission of the UCF Office of Residence Life is to
provide
Primary
Stakeholders
Primary
quality
Purposehousing facilities and related services that are reasonably
priced, safe, comfortable, well-maintained and staffed by
friendly, caring, and efficient people to undergraduate students.
The department develops and promotes programs and staff
interactions that are conducive to student learning, support the
University’s academic mission, and encourage individual
responsibility within a community setting.
Primary
Functions
25
Step 1: Defining Program Major Mission & Vision
Vision
1.
2.
3.
What would you like your department to become?
Do you want to be the best at something and in what way?
What would you like to see the department be or do in the future?
Vision Statement: The vision of (name of office) is to become
(statement that will guide your future actions).
Ask:
•
Is this a statement that will pull you in a desired direction?
•
Is your vision statement inspirational?
26
Step 1: Example of a Vision Statement
The University of Central Florida’s Faculty Center for
Teaching will foster the model environment
where scholars innovate, invigorate, and explore
the art of teaching and the science of learning.
27
Step 2: Defining Program Major
Learning Goals (Objectives)
28
Step 2:
Defining Program Major Learning Goals (Objectives)
Describe an “ideal” student at various phases in your program, focusing on the
abilities, knowledge, values and attitudes that you feel that this student has
either acquired or have been supported as a result of your program. Ask:
• Cognitive skills: What does the student know?
• Performance skills: What can the student do?
• Affective skills: What does the student care about?
•
Describe how the students’ experiences in the program have contributed to their
abilities, knowledge, values and attitudes.
•
Describe the program alumni in terms of their achievements, such as career
accomplishments, lifestyles, and community involvement.
•
List the skills and achievements expected of graduates of the program.
29
Step 2:
Defining Program Major Learning Goals (Objectives)
Collect and review any existing goal or outcome statements such as those from:
• Catalog descriptions
• Program review reports
• Mission and vision statements
• External agencies (e.g., Southern Association of Colleges and Schools
Commission on Colleges [SACS - COC], National Council for Accreditation of
Teacher Education [NCATE], Association to Advance Collegiate Schools of
business [AACSB], and Accreditation Board for Engineering and Technology
[ABET])
Review other programs’ goals
• Review program goals and intended outcomes of other departments at your
university.
• Review program goals and intended outcomes of similar departments at other
universities.
30
Steps 2
Defining Program Learning Goals (Objectives)
List five to seven of the most
important goals identified in the
sources.
•Prioritize the goals depending on
their importance to your
department and their universality
(i.e., how well they apply to
different program tracks, if
applicable).
Goal
1
2
•Next, determine whether the
goal is best described as
knowledge, abilities, attitudes,
values, or a combination of these.
A goal can be described by more
than one term. See the following
illustration. (Adapted from
Assessment Handbook, 2003)
3
Knowledge Abilities Attitudes Value
s
X
X
X
X
X
X
31
Step 2: Example #1
State Level Core Learning Goals
Florida State mandate (Academic Learning Compacts): Students who
complete a baccalaureate degree in the State of Florida are expected
to demonstrate the following Core competencies:
All undergraduate programs at UCF develop 8–12 specific
Student Learning Outcomes that address:
•
Communication : reading, speaking, writing, editing, questioning,
listening, making presentations, and interpersonal relations.
Some programs have additional modalities and/or techniques
•
Critical Thinking: outcome activities that require analysis, synthesis
and evaluation
•
Discipline-specific knowledge, skills, attitudes & behaviors: vary by
discipline and can vary by level, major or program
Institutional links:
All FL Universities: http://www.flbog.org/about/cod/asa/university_alc.php
32
Step 2: Example #2
Institution/College (General Education) Learning Goals
(Objectives)
Students who complete a baccalaureate degree are expected to
demonstrate the following competencies:
•
COMMUNICATION: demonstrate ability to engage in cross-communication
styles—written, verbal, auditory, visual
•
QUANITATIVE : apply quantitative tools to solve real-world problems;
•
SCIENTIFIC: understand and apply relationships and theories of natural
sciences;
•
SOCIOCULTURAL: demonstrate knowledge of the human condition and the
importance of the arts;
•
GLOBAL SOCIOCULTURAL RESPONSIBILITY: Participate actively as
informed and responsible citizens in social, cultural, global, and
environmental matters
•
INFORMATION FLUENCY: Critically, gather, evaluate, and use technological
information in an appropriate and effective way.
Adapted from College Core Competencies from South Plains College, Levelland, TX:
http://www2.southplainscollege.edu/ & Florida Community College Statewide GenEd Requirements
33
Step 2: Example #3
College Competency/Program Major Learning
Goals (Objectives)
CRITICAL THINKING: Observation, reflection, analysis,
synthesis, application.
A graduate of ____ is able to…
• Gather and analyze data relevant to a problem or position (ability).
• Consider diverse perspectives and alternative points of view
(values).
• Predict and evaluate consequences and revise the thinking process
(ability).
• Select well-reasoned solutions and use sound evidence to justify a
position (ability).
• Apply knowledge to new situations and larger contexts (_______).
34
Step 2: Example #4
College Competency/Program Major
Learning Goals (Objectives)
COMMUNICATION: demonstrate ability to engage in crosscommunication styles—written, verbal, auditory, visual
A graduate of ____ is able to…
• Comprehend and interpret various types of written
information (knowledge).
• Communicate thoughts, ideas, opinions, information, and
messages in writing (ability).
• Compose and create documents with correct grammar,
spelling, and punctuation, as well as appropriate
language, style, and format (_______).
35
Step 2: Example #5
College Competency/Program Major Learning
Goals (Objectives)
INFORMATION LITERACY: Find, evaluate, organize, and
use information.
A graduate of ____ is able to…
• Locate and use appropriate and relevant information from print
and/or electronic sources (knowledge, ability).
• Identify, analyze, and evaluate information for relevancy, accuracy,
authority, bias, currency, and coverage (_______).
• Synthesize collected ideas and materials into original work in
appropriate formats (knowledge, abilities).
• Use information ethically and legally (values, ability).
• Use appropriate technology to manage information, solve problems,
communicate, develop products, and provide services (_______).
36
Step 3: Learning Outcomes
Are they S.M.A.R.T.?
37
Step 3: Define Outcome Statements
Think S.M.A.R.T.
• Specific
• Clear and definite terms describing the abilities, knowledge, values,
attitudes, and performance
• Measurable
– It is feasible to get the data; data are accurate and reliable; it can be
assessed in more than one way
• Aggressive and Attainable
– The outcome has the potential to move the program or unit forward.
• Results-oriented
– Describe what standards are expected from students or aspect of the
functional area being assessed
• Time-bound
– Describe a specified time period for accomplishing the outcome (e.g.,
by graduation, course completion)
38
38
Step 3: Define Outcome Statements
Outcome Statement: A specific, measurable statement that describes desired
performance.
– More precise, specific, and measurable than a goal
– Can be more than one outcomes related to each goal
Types of outcomes:
• Outcome: A specific, measurable statement that describes desired performance.
• Operational outcome: A type of outcome that addresses operational or procedural
tasks, such as functions, demand, resources, efficiencies, and satisfaction
• Learning outcome/Student Learning Outcome (SLO): A type of outcome that
describes
a. intended learning that students should meet on the way to attaining a
particular degree, called Student Learning Outcomes.
b. intended learning that participants meet as a result of a unit’s program(s)
or services(s), called learning outcomes
39
Student Learning Outcomes (SLOs)
•
Specific statements that describe required learning achievement that must
be met on the way to attaining the degree or earning the certificate and
meeting the goals of the program.
•
Ways statements can be framed:
– What does the student know? (cognitive)
– What can the student do? (psychomotor)
– What does the student care about? (affective)
40
40
Step 3: Defining Outcome Statements--Learning
Outcomes/Student Learning Outcomes (SLOs)
Attention to Bloom’s Taxonomy
•
Cognitive: Developmental Process
 Knowledge, Comprehension, Application, Analysis, Synthesis,
Evaluation
•
Affective: Emotional Growth
 Accepting, Responding, Valuing, Organize personal value system,
Internalize value system
•
Psychomotor: Movement patterns, skills, and behaviors
 Imitation, Manipulation, Develop precision, Articulation (combine,
integrate related skills), Naturalization (automate, become expert)
41
Step 3: Example #1
Defining Student Learning Outcomes (SLOs)
Speech Communications 1600/1016
Communication: Effective reading, writing,
speaking, listening
Outcome: Students who have completed SPC 1600
or SPC 1016 will understand and use
extemporaneous delivery techniques to enhance
information processing and persuasion of listeners.
42
42
Step 3: Example #2
Defining Student Learning Outcomes (SLOs)
Reading Education
Communication: Effective reading,
writing, speaking, listening
Outcome: Students will demonstrate culturally
responsive teaching strategies that meet the
literacy needs of students in a diverse,
multicultural classroom.
43
Step 3: Example #3
Defining Student Learning Outcomes (SLOs)
American History : 1877-Present
Communication: Effective reading, writing,
speaking, listening
Outcome: Students will demonstrate
knowledge of the chronology and significance of
major events and movements in American
history.
.
44
44
Step 3: Example #4
Defining Student Learning Outcomes (SLOs)
Art
Communication: Effective reading, writing,
speaking, listening
Outcome: Students will demonstrate good visual
literacy in their artwork (e.g., principles of design,
technical applications, systems of perception, color
applications, and conceptual issues).
45
Learning Outcomes: Rate the Examples
Are they S.M.A.R.T.?
46
Learning Outcomes: Rate the Examples
1.
Faculty will collaborate with M.Ed. Students to produce a successful
conference proposal.
2.
Tutors will demonstrate active listening techniques following the
Summer 2008 training.
3.
HVAC certificate students will accurately appraise systems likely to be
present in specified building types.
4.
After completing the A.S. program in Graphic Design, students will
successfully redesign products according to customer specifications.
5.
Graduates of the Hypothetical Engineering BS program will recognize,
integrate, and diagram essential components of engineering.
47
Learning Outcomes: Rate the Examples
1.
Students will develop the competencies needed for teachers of social
studies.
2.
After completing SLS 1101, students will show an increase in their ability
to use technological resources to conduct research.
3.
HVAC certificate students will accurately appraise systems likely to be
present in specified building types.
4.
After completing the A.S. program in Graphic Design, students will
successfully redesign products according to customer specifications.
48
Steps 4 -7: Assessment Methodology,
Course & Curriculum (Program)
Mapping
49
Step 4: Identifying Appropriate Assessment
Methods
• Consider these questions:
–
–
–
–
How will I know if this outcome is being accomplished?
What measures are currently available?
What method will provide me with this information?
What kind of assessment data: Formative or Summative?
51
Step 4: Identifying Appropriate Assessment
Methods
• Direct: direct examination or observation of knowledge,
skills, attitudes or behaviors to evaluate the competence
of students (participants) in the program
What does the student (participant) know?
– What can the student (participant) do?
• Indirect: concerned with documenting student
(participant) experiences, opinions, or perceptions
– What does the student (participant) report that
he/she knows?
– What does the student (participant) report he/she
can do?
52
52
Examples of Program Assessment Methods
for Program Level Operational Outcomes
(PLOs)
Direct Methods
•
•
•
•
Retention rates
graduation rates
degree attainment
percent of graduates continuing to
•
•
•
•
•
•
•
•
53
Indirect Methods
•
• stakeholder perception
• students
• administration and staff
baccalaureate programs (2-year
colleges)
graduate school (baccalaureate
programs
cost per student to deliver instruction
faculty qualifications
faculty productivity
employment statistics
reduction in errors (or other)
audit, external evaluator
written surveys and questionnaires:
• faculty
• employers
•
interviews
•
focus groups
Examples of Program Assessment
Methods for Student Learning Outcomes
(SLOs)
54
Direct methods
Indirect methods
•
standardized exams
•
•
locally developed exams
•
embedded questions
•
external examiner
•
oral exams
•
minute papers
•
portfolios (with rubrics)
•
behavioral observations
•
simulations
•
project evaluations
•
performance appraisals
written surveys and
questionnaires:
– student perception
– alumni perception
– employer perception of
program
•
exit and other interviews
•
focus groups
•
student records
54
Step 4: Identify Appropriate Assessment
Methods
Be M.A.T.U.R.E.
Matches
– directly related to the outcome it is measuring
Appropriate methods
– uses appropriate direct and indirect measures
Targets
– indicates desired level of performance
Useful
– measures help identify what to improve
Reliable
– based on tested, known methods
Effective and Efficient
– characterize the outcome concisely
55
55
Step 4: Examples Linking Student Learning
Outcomes with Assessment Methodology
COMMUNICATION
Outcome: Students who have completed SPC 1600 or SPC 1016
will understand and use extemporaneous delivery techniques to
enhance information processing and persuasion of listeners.
Measure: Student persuasive speeches will be videotaped. The
Speech Program’s Assessment Committee will evaluate 50 speech
samples using criteria in the program-adopted rubric. Standard:
Students observed will be rated as Satisfactory (70% or above) on
use of techniques that increase the likelihood of persuasion as
defined by a program adopted rubric.
56
Step 4: Examples Linking Student Learning
Outcomes with Assessment Methodology
COMMUNICATION
Outcome: Students will demonstrate knowledge of the chronology and
significance of major events and movements in American history.
57
Measure: Professors and instructors will collect portfolios of work
conducted by randomly selected students. Each portfolio will be
determined, grouped in one of three categories: "excellent,"
"satisfactory," or unsatisfactory. Faculty on the assessment team will
evaluate the examinations, essays, and other work to determine whether
each student demonstrates an ability to meet the "qualitative" factors of
this learning objective, specifically can students analyze information,
appreciate the significance of that information, and array it into cohesive
arguments or narratives demonstrating convincing relationships between
major events and movements. Target: All student portfolios will be
‘satisfactory’
57
Step 4: Examples Linking Student Learning Outcomes
with Assessment Methodology
COMMUNICATION
Outcome: Students will demonstrate culturally responsive teaching strategies
that meet the literacy needs of students in a diverse, multicultural
classroom.
Measure: Research papers and presentations in RED 2116, RED 2336, and
RED 2337 will demonstrate that students have acquired the knowledge
and skills required for culturally responsive teaching as judged by course
faculty. 90% of all student research papers and presentations completed in
these courses will receive a grade of "B" or higher from course faculty. A
random sample will be reviewed by external reviewers, using the same
rubric as course faculty. All papers reviewed will earn a “satisfactory” or
higher.
58
Step 4: Examples Linking Student Learning Outcomes with
Assessment Methodology
COMMUNICATION
Outcome: Students will demonstrate good visual literacy in their artwork
Measure: External juror reviews a sample of portfolios and/or exhibited
work to assess visual literacy. Juror assessment forms assess the
performance in: principles of design, technical applications, systems of
perception, color applications, and conceptual issues. The students will be
rated on a scale (excellent, very good, good, fair, poor) to assess and report
the department's strengths and weaknesses in this category. Based on
previous year’s baseline data: We expect to achieve at least 75% in the
“good,” “very good” or “excellent” categories.
59
Assessment Methods Exercise: Rate the
Examples
1. At least 85% of students surveyed will agree or strongly agree
that they feel competent in public speaking situations.
2. The average GPA of all graduating CS majors on assignments
requiring oral presentations or written technical documentation
shall be a 2.75 or better. The data used for this measure shall be
based on Course Content reports for all required CS courses
taken post-Foundation Exam.
3. Students will produce or direct a short film with a grade of B or
above In FIL 4228 Directing III.
60
60
Assessment Methods: Rate the Examples
Are they M.A.T.U.R.E.
61
Assessment Methods Exercise: Rate the
Examples
1. Graduates of Imaginary Science B.S. will demonstrate the ability to create
original designs for emerging technologies.
2. By the end of their sophomore year, students will score at least 85% on
the embedded questions in FLD 2742 that address flow patterns.
3. At least 85% of students will be able to administer shots accurately as
rated by the clinical preceptor.
4. Employers will rate graduates satisfactory or above satisfactory on the 6
targeted skills.
5. Graduates of the Graphics Arts AS Program will score at least above
average on the faculty developed rubric assessing their accuracy.
62
62
Step 4: Conduct Course Mapping
63
Step 5: Conduct Course Mapping
Bloom’s Taxonomy-Learning in Action
Cognitive Learning: Developmental Process
•
•
Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation
Learning Experiences: problem based learning, group work in diverse teams, classroom
teaching, readings and discussions
Affective: Emotional Growth
•
•
Accepting, Responding, Valuing, Organize personal value system, Internalize value system
Learning Experiences: inter-group dialogue, cultural festivals, academic/life planning,
community-based learning, service learning, identity group programming, programs on world
religions, interdisciplinary courses
Psychomotor: Movement patterns and behaviors
•
•
Imitation, Manipulation, Develop precision, Articulation (combine, integrate related skills),
Naturalization (automate, become expert)
Learning Experiences: web-based information search skills, drama, arts, and music groups
64
Step 5: Conduct Course Mapping
Course Mapping ensures that the course content is actually relevant to
program and student learning outcomes.
Course Level Analysis:
• Align course SLOs with program Goals/SLOs
• Identify level of Bloom’s Taxonomy
• Identify appropriate classroom assessment tools and align with Course SLOs
Course
SLO1
Program Goal
A1
Program Goal
A2
I
R
Define
Analysis
Research Paper
test
65
Step 6: Conduct Developmental
Curriculum (Program) Mapping
66
Step 6: Conduct Developmental Curriculum
(Program) Mapping
Curriculum (Program) Mapping ensures that formative and summative
assessment methods for program improvement
Program Level Analysis: Aligns formative and summative assessment methods for
program improvement
• Identify the Program Goals that the Department would like students to
possess at different points in the program curriculum
• Identify the cognitive level for each of the course outcomes Introduce
(I), Emphasize (E), and Reinforce (R) according to Bloom’s Taxonomy:
Addressing Multiple Cognitive Levels
• Develop and/or identify FORMATIVE and SUMMATIVE assessment
methods to assess student learning and cognitive level
development, and
• Develop and/or identify FORMATIVE and SUMMATIVE assessment
methods to assess how well the Department is meeting Program goals
67
Step 6: Example of Developmental Curriculum
(Program) Mapping
Program Goal
A1: an ability to apply
knowledge of
mathematics,
science, and
engineering
“I” : Introduce/ Identify
“E”: Emphasize/Apply
“R”: Reinforce/ Analyze
Program
Goal
A1
Program
Entry level
Prereq
Stats
Course
Junior
Level
Course
3000
Senior Level
Capstone
I
E
R
X
(formative)
X
X
(Summative)
“X”: Assessment (i.e., pre-post test;
program entrance /exit exams; capstone
courses; college entrance/exit exams
etc).
68
Step 7: Establishing Data Collection
Timeline
69
Step 7: Establishing Data Collection Timeline
Questions to consider
– When will data be collected?
– Before, during, after course or program completion
– Before, during, or after fall, spring, summer semesters
– Every academic year
– How will you “capture” students?
– Will you collect data from the population or a sample?
– Program Pre entry or graduation?
– First years, sophomores, juniors, seniors, alumni
– Will the resulting data provide information that could lead to
improvement?
70
Example of Curriculum (Program) Mapping, &
Assessment Timeline
Program Goals
A1
Required Course
Level 1
Capstone
Course
SLO 1: ability to apply
knowledge of
mathematics,
science, and
engineering
I
Define
Pre-assessment knowledge test
(Formative)
R
Analyze
Problem based project
(Summative)
PLO 1: Curriculum
adequately
prepares
students to pass
licensure exams
or program
comprehensive
exam
Administer program entry exam
or licensing exam by the 2nd
class meeting. Data used to
assess student preparation
and/or cohort characteristics
Re-Administer
licensing or program
exam. Review student
sub-scores & pass
rates to assess
curriculum and
student learning gains
71
Steps 8-10: Data Collection &
“Closing the Loop”
72
Steps 8 & 9: Conducting Data Analysis for
Program Improvement
Data Collection Procedures
• Who will organize and analyze it?
• Where will data and results be housed?
• With whom will results be shared?
Interpretation of Assessment Data?
• Do we need to adjust what we are teaching?
•
Will we need to change the methods we use to teach?
•
Do we need to change the methods we use to assess?
74
Step 10: “Closing the Loop,” Using
Assessment Results for Program
Improvement
• Changes to Curriculum
– Changes in pedagogical practices
– Revision or enforcement of prerequisites
– Revision of course sequence
– Revision of course content
– Addition of course(s)
– Deletion of course(s)
75
Step 10: “Closing the Loop,” Using
Assessment Results for Program
Improvement
•
Changes to Academic Processes
– Modification of frequency or schedule of course offerings
– Improvements of technology
– Implement discipline or pedagogical related professional
development
Changes to Method of Assessment:
– Examine assessment tool and method
– Examine data collection
– Examine criterion level
76
Assessment Process Model
77
Resources
Angelo, T. And Cross, P. (1993). Classroom Assessment Techniques: A
Handbook for College Teachers. San Francisco: Jossey Bass.
Banta, T., Lund, J. Black, K., and Oblander, F. (1996). Assessment in Practice:
Putting Principles to Work on College campuses. San Francisco: JosseyBass.
Maki, P. (2004). Assessing for Learning: Building a Sustainable Commitment
Across the Institution. Sterling, VA: Stylus Press (AAHE).
Morante, Edward A (2003). A Handbook on Outcomes Assessment for Two
Year Colleges. Palm Springs, CA: College of the Desert. C
Nichols, Karen and James. (2001). General Education Assessment For
Improvement Of Student Academic Achievement: Guidance for Academic
Departments and Committees. New York: Agathon.
78
Resources
Piuma, Francesca, et Al. “Measuring Moving Targets: Assessing General
Education and Program Outcomes across the Curriculum.” Paper presented
at the Annual conference of the North Central Association Commission on
Schools. Tempe, AZ April 6-9, 1997.
--- Using the Baldrige Quality Award for Program Assessment. Watwood,
Britt U.S.; Nebraska; 1996-02-00
Assessment resources available at websites
http://www2.acs.ncsu.edu/UPA/assmt/resource.htm
http://www.oeas.edu
http://www.bridgew.edu/AssessmentGuidebook
http://www.fctel.uncc.edu/pedagogy/assessment/9Principles.html
http://rubistar.4teachers.org/index.php
79
Questions?
Continue the conversation:
Dr. Mia Alexander-Snow, PhD
Phone:407-823-0182
Email: mdalexan@mail.ucf.edu
80
80
CoB Example of Student Learning
Outcomes for Course and Program
GENERAL BUSINESS: Demonstrate an understanding of fundamental business
processes and practices
Outcome: Students will demonstrate knowledge of essential business
concepts and principles in the core foundational disciplines of accounting,
economics, finance, marketing, and management (includes information
management).
Measure: ???
Direct Methods: Examinations, Discipline specific projects, ETS Exam
Courses: Economics, Accounting, Marketing, Finance, Management
81
CoB Example of Student Learning
Outcomes for Course and Program
COMMUNICATION: Effectively communicate in the business world.
Outcome: Students will produce written documents that are clear and
concise, grammatically correct, incorporate logical, complete, and
articulate thoughts, and retain a professional appearance.
Measure: ???
Direct Methods: Project reports,
Courses: Global Business, Organizational Behavior and Theory
82
CoB Example of Student Learning
Outcomes for a Course
COMMUNICATION: Effectively communicate in the business world.
Outcome: Students will produce written documents that are clear and
concise, grammatically correct, incorporate logical, complete, and
articulate thoughts, and retain a professional appearance.
Measure: ???
Direct Methods: Project reports,
Courses: Global Business, Organizational Behavior and Theory
83
CoA Example of Student Learning
Outcomes for a Course
COMMUNICATION: Effectively communicate in the business world.
Outcome: Students will produce written documents that are clear and
concise, grammatically correct, incorporate logical, complete, and
articulate thoughts, and retain a professional appearance.
Measure: ???
Direct Methods: Project reports,
Courses: Global Business, Organizational Behavior and Theory
84
CoB Example of Student Learning
Outcomes for a Course
COMMUNICATION: Effectively communicate in the business world.
Outcome: Students will produce written documents that are clear and
concise, grammatically correct, incorporate logical, complete, and
articulate thoughts, and retain a professional appearance.
Measure: ???
Direct Methods: Project reports,
Courses: Global Business, Organizational Behavior and Theory
85
ABET 2009-2010 Educational Objectives
(a-k) for Engineering Programs Only
(a) an ability to apply knowledge of mathematics, science, and engineering
(b) an ability to design and conduct experiments, as well as to analyze and interpret data
(c) an ability to design a system, component, or process to meet desired needs within realistic
constraints such as economic, environmental, social, political, ethical, health and safety,
manufacturability, and sustainability
(d) an ability to function on multidisciplinary teams
(e) an ability to identify, formulate, and solve engineering problems
(f) an understanding of professional and ethical responsibility
(g) an ability to communicate effectively
(h) the broad education necessary to understand the impact of engineering solutions in a global,
economic, environmental, and societal context
(i) a recognition of the need for, and an ability to engage in life-long learning
(j) a knowledge of contemporary issues
(k) an ability to use the techniques, skills, and modern engineering tools necessary for
engineering practice.
86
AACSB 2009 Assessment of Learning Standards
AACSB Standard 15 (Management of Curricula) states:
The school uses a well documented, systematic process to develop,
monitor, evaluate and revise the substance and delivery of the
curricula of degree programs and to assess the impact of the
curricula on learning.
Standard 16 (Bachelor’s or undergraduate level degree: Knowledge and skills)
states:
…Adapting expectations to the school’s mission and cultural
circumstances, the school specifies learning goals and demonstrates
achievement of learning goals for key general, management-specific,
4 and/or appropriate discipline-specific knowledge and skills that its
students achieve in each undergraduate degree program.
87
AABI Accreditation Criterion
(Baccalaureate Programs)
4.1 Program Criteria for Aviation Management (Air Carrier Management, Airway
Science Management, Airport Management, Flight Operations Management, and
Maintenance Management)
•
The aviation faculty MUST develop each program with advice from appropriate
industry associations and professionals in the field.
•
Each program MUST provide evidence that graduates possess the necessary
knowledge, skills, and attitudes to competently and ethically function as a
manager in the aviation industry. Each program may be oriented toward a
segment of the industry, such as airlines, general aviation, or airports; or towards a
specific area, such as flight operations management or aircraft maintenance
management, or may be of a general nature. Additionally, each program MUST
provide evidence that its graduates demonstrate competency in program
objectives.
88
AABI Accreditation Criterion
(Baccalaureate Programs)
4.2 Program Criteria for Aviation Maintenance (Aviation Maintenance/Manufacturing,
Aeronautical Technology, and Airway Science Maintenance)
•
•
•
•
The aviation faculty MUST develop each program with advice from appropriate
industry associations and professionals in the field.
Each program MUST provide evidence that graduates possess the necessary
knowledge, skills, and attitudes to competently and ethically function as a
maintenance professional in the aviation industry.
An Aviation Maintenance program is designed to prepare the graduate for a
position in aviation maintenance or manufacturing. Classroom and laboratory
topics MUST lead to appropriate national certification.
The program objectives MUST include certification/licensure as an Aviation
Maintenance Technician with airframe and/or power plant ratings, or national
equivalent.
89
AABI Accreditation Criterion
(Baccalaureate Programs)
4.3 Program Criteria for Aviation Electronics (Avionics Technology, Airway Electronics,
or Aviation Technology)
•
•
•
Each program MUST provide evidence that graduates possess the necessary
knowledge, skills, and attitudes to competently and ethically function as an
aviation electronics professional in the aviation industry.
An Aviation Electronics program is designed to prepare the graduate for a position
in general, commercial, or military aviation, aerospace, and aviation related
government agencies. Graduates MUST be able to apply science and technology
to current problems in the aviation and the aviation electronics industry. The
topical content of an Aviation Electronics program will depend on the area of
specialization.
Graduates of all specializations MUST demonstrate a basic foundation in the
electronics technologies.
90
Example of General Education Program (GEP)
Competencies
Students who satisfy UCF GEP requirements are expected to
demonstrate the following competencies
Communications Foundation: demonstrate the ability to understand
communication and speaking skills
Cultural and Historical Foundation: identify and deepen appreciation of
common human themes and the richness of diverse cultures.
Mathematical Foundation: be able to solve real-world quantitative problems.
Social Foundation: demonstrate an understanding of the interaction among
social, economic, and political structures and functions.
Science Foundation: Demonstrate understanding of scientific knowledge and
problem solving in a physical or life science.
http://www.catalog.sdes.ucf.edu/undergraduate_degree_requirements/general_education_program/gener
al_education_program_learning_objectives/Default.aspx
91
AABI Accreditation Criterion
(Baccalaureate Programs)
4.4 Program Criteria for Aviation Studies (Airway Computer Science, Aviation Science,
Air Traffic Control, Safety, Security, Atmospheric Science, Aviation Meteorology, or
Aviation Human Factors)
•
•
•
The aviation faculty MUST develop each program with advice from appropriate
industry associations and professionals in the field.
Each program MUST provide evidence that graduates possess the necessary
knowledge, skills, and attitudes to competently and ethically function as a
professional in the aviation industry.
The Aviation Studies option provides baccalaureate courses in a coherent
sequence to prepare the graduate for a position in the aviation industry and
aviation related government agencies, requiring either broad or specialized
educational preparation.
92
AABI Accreditation Criterion
(Baccalaureate Programs)
4.5 Program Criteria for Flight Education (Aircraft Systems Management, Flight
Operations, Career Pilot, Professional Pilot, or Aeronautical Science)
•
•
•
•
The aviation faculty MUST develop each program with advice from appropriate
industry associations and professionals in the field.
Each program MUST provide evidence that graduates possess the necessary
knowledge, skills, and attitudes to competently and ethically function as
professional pilots in the aviation industry.
Classroom and laboratory topics MUST lead to appropriate national certification.
The program objectives MUST include certification/licensure as a Commercial Pilot
with an instrument rating, and multiengine land rating or flight instructor.
93
AABI Accreditation Criterion
(Baccalaureate Programs)
Criterion 2.1 Students
The quality and performance of the students and graduates are important
considerations in the evaluation of an aviation program. The institution MUST:
a.
b.
c.
d.
e.
f.
g.
Evaluate, advise, and monitor students to determine its success in meeting
program objectives
Have and enforce policies for the acceptance of transfer students and for the
validation of courses taken for credit elsewhere
Have and enforce procedures to assure that all students meet all program
requirements
Publish standards for the selection and admission of students which are related
to the educational mission and purposes of the institution
Maintain recent examples of student work, to include examinations, homework
problems, laboratory exercises, and reports. These items will include evidence
of student competence in both subject matter areas and communications skills
Assess the effectiveness of its validation methods in granting credit for noncollegiate achievement
Produce records reflecting the employment or continuing education experience
of students graduating from the program during each of the preceding five years
94
AABI Accreditation Criterion
(Baccalaureate Programs)
•
4.0
•
Each program MUST provide evidence that its graduates demonstrate competency
in program objectives.
•
Each program MUST provide evidence of a significant culminating upper division
experience in aviation management. Examples of a culminating experience
include a capstone course, an internship, or a special project that builds on prior
course work. Evidence may include student portfolios and other records of
student achievement.
PROGRAM CRITERIA (4.1 – 4.5)
95
AABI Accreditation Criterion
(Baccalaureate Programs)
Criterion 2.3 General Outcomes
Aviation programs MUST demonstrate that graduates have:
• An ability to apply knowledge of mathematics, science, and applied sciences to aviation•
•
•
•
•
•
•
•
•
•
related disciplines
An ability to analyze and interpret data
An ability to function on multi-disciplinary and diverse teams
An understanding of professional and ethical responsibility
An ability to communicate effectively, including both written and oral communication skills
A recognition of the need for, and an ability to engage in, life-long learning
A knowledge of contemporary issues
An ability to use the techniques, skills, and modern technology necessary for professional
practice
An understanding of the national and international aviation environment
An ability to apply pertinent knowledge in identifying and solving problems
An understanding of business sustainability in aviation
96
AABI Accreditation Criterion
(Baccalaureate Programs)
Criterion 2.12 Continuous Assessment and Improvement
Each program MUST have an assessment process that includes a written plan with
documented results. This process MUST incorporate relevant evidence used to
regularly assess program objectives and outcomes, and to evaluate the extent to
which they are being met. Evidence that may be used includes, but is not limited
to, the following: student portfolios, including graded assignments and/or projects;
nationally normed subject content examinations; alumni surveys that document
professional accomplishments and career development activities; placement data
of graduates; and employer surveys. The results of the assessment MUST be used
to effect continuous improvement of the program.
97
AACSB 2009 Assessment of Learning Standards
Standard 18 (Master’s level degree in general management (e.g., MBA
programs: Knowledge and skills) states:
…Adapting expectations to the school’s mission and cultural
circumstances, the school specifies learning goals and demonstrates
master’s level achievement of learning goals for key management-specific
knowledge and skills in each master’s level general management program.
Standard 19 (Master’s level degree in specialized programs: Knowledge and
skills) states:
…Adapting expectations to the school’s mission and cultural
circumstances, the school specifies learning goals and demonstrates
achievement of learning goals in each specialized master’s degree
program.
98
AACSB 2009 Examples of Learning Goals and
Corresponding Objectives
Learning Goal: Our graduates will understand the importance of behaving
ethically in their professional lives (i.e., have an ethical perspective).
Corresponding Objectives:
• Our students will know the professional code of conduct within their
discipline.
• Our students will identify the activities/issues in their chosen profession
that may present ethical challenges, and will articulate the consequences
associated with unethical behavior.
• Our students will identify an ethical dilemma in a scenario case and apply
an ethics model or framework to propose and defend a resolution.
99
100