Office of Institutional Planning, Research and Effectiveness (OIPRE)
October 6 – 8, 2015
Drake LRC
1

1. Provide a rationale for engaging in outcomes assessment
2. Describe the concept of institutional effectiveness
3. Outline steps in the outcomes assessment process
4. Distinguish between student learning outcomes and program outcomes
5. Describe AAMU outcomes assessment process requirements
6. Outline annual outcomes assessment report timeline
2

• Outcomes assessment is a formative process
• Outcomes assessment is not an end; rather, it is a means for improving student learning
• Motivation to engage in outcomes assessment
• Intrinsic value
• External accountability demands
3

• External accountability demands
• Outcomes assessment is a requirement of our regional accreditor – SACSCOC
• Without regional accreditation our students are not eligible for federal student aid
• Intrinsic value
• Outcomes assessment allows us to determine how well we are living up to the implicit promise we make to our students when they enroll and where appropriate, make the necessary changes to curriculum and/or pedagogy to ensure students reach the expected performance levels
4

• Institutional effectiveness is the systematic, explicit, and documented process of measuring institutional performance against mission in all aspects of an institution (p. 16).
• A commitment to continuous improvement is at the heart of an on-going planning and evaluation process. It is a continuous, cyclical process that is participative, strategic, flexible, relevant and responsive (p. 16).
• This process [outcomes assessment] serves as the cornerstone of institutional effectiveness. Institutional Effectiveness focuses on the design and improvement of educational experiences to enhance student learning
(pp. 48 – 49).
Source: Southern Association of Colleges and Schools Commission on Colleges, Resource Manual for the Principles of Accreditation
5

University Mission Statement
Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, of knowledge and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and region. The university offers contemporary baccalaureate, master’s educational specialist and doctoral level degrees to prepare student for careers in the arts, sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a student-centered educational environment for the emergence of scholars, scientist, leaders and critical thinkers, who are equipped to excel through their contribution and leadership in a 21 st century national and global society.
Strategic Priority 1:
Enhance AAMU’s
Distinctiveness and
Competitiveness
Strategic Priority 2:
Strengthen
Structures,
Operations, and
Systems to
Promote and
Support
Organizational
Excellence and
Stewardship
Strategic Priority 3:
Upgrade
University
Infrastructure and
Facilities
Strategic Priority 4:
Secure the
University’s
Financial Future
Strategic Priority 5:
Enhance the
University’s Image and Recognition
Strategic Priority 6:
Enhance university
Engagement through Expanded
Outreach
PROGRAM 1
MISSION
PROGRAM 2
MISSION
PROGRAM 3
MISSION
PROGRAM 4
MISSION
PROGRAM 5
MISSION
6

• Comprehensive Standard 3.3.1 (Institutional Effectiveness – unit level)
The institution 1) identifies expected outcomes*, 2) assesses the extent to which it achieves these outcomes, and 3) provides evidence of improvement based on analysis of results .
*For academic programs this includes student learning outcomes.
7

1. Develop program mission statement
2. Develop outcomes
3. Develop appropriate assessment measures
4. Develop desired performance targets
5. Analyze results
6. Develop and implement improvement plans
8

Improvement
Plans
Step 6
Mission
Statement
Step 1
Outcomes
Step 2
Analysis
Step 5
Performance
Targets
Step 4
Assessment
Measures
Step 3

• Mission statements should…
• Be specific, distinctive
• Clearly state the purpose of the academic program
• Identify the program’s stakeholders
• State the values and principles that guide the curriculum/program goals
• Speak to what the program is currently doing
• Align with AAMU’s strategic goals
10

• The mission of <<enter program name>> is to <<program’s primary purpose(s)>> by providing <<program’s primary functions or activities>> .
11

• “The International Baccalaureate® aims to develop inquiring, knowledgeable and caring young people who help to create a better and more peaceful world through intercultural understanding and respect. To this end the organization works with schools, governments and international organizations to develop challenging programmes of international education and rigorous assessment.”
• “The mission of the Department of Physician Assistant Studies is to educate patient-centered medical providers of the highest quality who are critical thinkers, leaders, and life-long learners.”
12

• Student learning outcomes (SLOs)
• The knowledge, skills or abilities students acquire through the successful completion of a course or academic program (what do you expect your graduates to know, think or be able to do upon graduation?)
• Unit of analysis = students enrolled in your program
• Program outcomes
• Illustrate what you want your program to do. These outcomes differ from learning outcomes in that you discuss what it is that you want your program to accomplish ( Bresciani, 2001 )
• Unit of analysis = the program
13

• For the annual assessment report, academic programs must report on:
• Three (3) student learning outcomes
• Two (2) program outcomes
14

• SLO statements should…
• Specify what students are expected to know, think or be able to do upon program completion
• Address only one outcome per statement
• Be under the control/responsibility of the program
• Be measurable
• Yield non “Yes/no” results
• Be at the program level
15

• Graduates of the <<program name>> will be able to/will demonstrate
<<skill, knowledge, value>>.
16

• Graduates of the B.S. Nursing
Program will have published at least one article or presented a paper at a conference.
• The program has no control over a student’s paper or conference presentation proposal being accepted.
• Graduates of the B.S. Nursing
Program will be able to integrate evidence, critical thinking and patient preference in evaluating options for care.
17

• Graduates in Food and Beverage
Management will appreciate the value of customer service.
• Graduates in Food and Beverage
Management will demonstrate their understanding of the value of customer service.
• This outcome is not measurable.
18

• Graduates in Computer Systems will be able to develop web pages, write and debug programs in multiple languages, understand basic computer concepts, and be able to describe the impact of technology on society .
• Graduates in Computer Systems will be able to describe the impact of technology on society.
• Graduates in Computer Systems will be able to analyze technical requirements and select the most appropriate solution.
• This outcome is bundled – the statement addresses more than one outcome.
19

• Graduates in Social Work will have successfully completed an internship.
• Graduates in Social Work will be able to apply the knowledge and skills of a generalist social worker.
• This is a yes/no outcome that will not necessarily yield data for improvement. Also, completion of the internship is not under the program’s control.
20

• Program Outcomes measure things other than direct student learning
• Enrollment levels, # of graduates, course completion rates
• Alumni satisfaction, employer satisfaction, satisfaction with advising
• Licensure pass rates
• Job/graduate school placement rates
• Faculty productivity, research grant funding levels
21

• For the annual assessment report, academic programs must report on:
• Three (3) student learning outcomes
• Two (2) program outcomes
22

• Now that you have developed sound outcomes, you must
• Indicate how the outcome will be assessed
23

• Assessment measures should be designed to yield meaningful , actionable data that can lead to improvements .
24

• Should…
• Be directly related to the outcome
• Address all aspects of the outcome over time
• Provide measurable results
• Provide adequate data for analysis
• Provide actionable data
• Focus on student work at or near graduation
25

• Should…
• Outline a systematic way to assess the outcome
• Include the following details:
• What will be collected
• When the data will be collected
• What assessment tool will be used
• How will the data be collected
26

• Direct Assessments
• Are based on analysis of student work, performances or behaviors
• Can be
• Quantitative – correct/incorrect (e.g., multiple choice, true/false, problem sets, etc.)
• Qualitative – requires judgment on the part of the evaluator (e.g., term paper, short answer/essay exam, project, performance, exhibits, etc.)
27

• Indirect Assessments
• Based on reported perceptions of student learning (e.g., student survey)
• Only appropriate as a secondary assessment method
28

• When assessing a S tudent L earning O utcome
• Multiple (at least two) assessment methods should be used where possible
• Direct assessments should be the primary assessment method utilized
29

• Tests (completion, essay, matching, multiple choice, true-false)
• Embedded assignments and course activities
• Classroom assessment activities
• Capstone projects (e.g., senior theses)
• Homework assignments
• Class presentations
• Poster presentations
• Recitals/exhibitions
• Portfolios
30

• While grades are appropriate as class assessment measures, they are not acceptable as program assessment measures .
• Why is that?
• Grades take the individual student as the unit of analysis, not the program
• May include factors not related to student performance (e.g., attendance, participation)
• Don’t necessarily map neatly on to specific student learning outcomes and therefore don’t lead to actionable data
31

• Why do we need performance targets?
• Allows the program to identify desired performance levels
• Provide standard for determining success
• Puts data analysis in perspective
32

• Targets should…
• Use component (not student or respondent) as unit of analysis
• Relate to outcome and assessment method
• Be specific (“x% of responses will be correct for each question” or “y% of scores for each item on rubric or survey will be at least 4”
• Avoid vague words such as “most” or “majority”
• Avoid “all” or “100%” targets
33

• We will analyze final exams from the capstone course. 85% of our majors will pass the final with a score of at least 60%.
• This measure uses students rather than learning components as the unit of analysis. As such, the measure will not lead to data that can inform improvement plans.
• We will analyze final exams from the capstone course. 80% of the responses for each of the ten questions will be correct.
• In this revised measure and target, each of the ten exam items represents a learning component.
34

• We will keep a record of the number of majors who participate in internships. The criterion for success is that all majors will successfully complete an internship.
• Example of a 100% target. Also the measure – internship completion – is not under the program’s control.
• Each intern supervisor will be asked to complete an evaluation form that directly measures the intern’s performance.
At least 85% of the scores for each item on the evaluation form will be “satisfactory” or “highly satisfactory.”
35

• A random sample of lab reports from the sophomore lab courses will be graded using a rubric. The target is that 75% of lab reports will be judged as “satisfactory” or
“excellent.”
• The assessment measure is conducted early in the program rather than near graduation. Also, the performance target focuses on the assignment grade rather than the learning components evaluated in the assignment.
• A random sample of lab reports from the capstone lab course will be graded using a rubric. The target is that at least 75% of scores for each item on the rubric will be
“satisfactory” or “excellent.”
36

• A survey will be conducted of graduating majors during the spring semester. The average score on the question related to “overall satisfaction with the program” will be at 3.5 out of a possible 5 points.
• This is an indirect measure (should only be used in concert with a direct measure). Target uses averages which can be skewed by outlier scores.
Instead we should focus on the proportion of students meeting the performance criteria.
• A survey will be conducted of graduating majors during the spring semester. At least 80% of the response for each question on the survey will be “satisfied” or “highly satisfied.”
37

• Here we move from planning the assessment to conducting it
• Provides data on which you can base changes
38

Analysis should…
• Provide detailed data (avoid “a majority” or “most”)
• Use specific numbers (no rounding)
• Avoid technical language
• Be clearly and succinctly presented
• Align with outcome and target
• Support actions taken later to foster improvement
39

• Should…
• Address outcomes where the performance target was not met
• State what future actions will take place in the future to address the outcome
40

• Curriculum revision
• Additional assignment
• Course modification
• Instructional methodology change
• Sequence change
• Technology update
• Assessment method change
• Target adjustment
• Faculty development
• Procedure, process change
41

Student Learning Outcome
Biology graduates will demonstrate an understanding of cell structure and function.
42

Assessment Measure
• Students in BIOL 432 will submit a research project based on cell structure and function. A faculty panel will use a rubric that has a scale of 1 to 5 (where 5 is excellent) to assess the five components of the project. Panel scores will be analyzed by each component across all student work.
43

Target
• 80% of scores provided by the faculty panel for each of the five components of the Biology Project Rubric will be 4 (very good) or 5
(excellent).
We will revisit this in a few slides.
44

Biology Project Results
Component # Scores 4 or 5 Total # Scores % Scores 4 or 5
Sources 68 72 94.4%
Methods
Analysis
Conclusion
Organization
45
67
48
59
72
72
72
72
62.5%
93.1%
66.7%
81.9%
45

Analysis
The 24 biology projects were assessed by a three-member faculty panel for each of the five components of the rubric and for overall performance. The target was not met for the Methods (62.5%) and
Conclusion (66.7%) components of the projects.
46

Improvement Plans
Fewer than 80% of scores provided by the faculty panel were 4 (very good) or 5 (excellent) for the Methods and Conclusion components of the Biology project. The department will develop a short video detailing the process of writing a methods section and make this available to BIOL 432 students through the course Blackboard site. A new assignment requiring students to critique the conclusion sections of sample lab reports will also be introduced in BIOL 432.
47

University Mission: Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, and dedicated to intellectual inquiry. The application of knowledge and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and region. The University offers contemporary baccalaureate, master’s, educational specialist and doctoral level degrees to prepare students for careers in the arts, sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a studentcentered educational environment for the emergence of scholars, scientists, leaders and critical thinkers, who are equipped to excel through their contributions and leadership in a 21st century national and global society.
Program Purpose: Step 1
Student Learning
Outcomes
Step 2
Assessment Methods
Performance Criteria
Performance
Target
Step 3 Step 4
Assessment Results Improvement Plans
Step 5 Step 6
Improvements
Implemented
Step 6
Program Outcomes
Step 2
Assessment Methods
Performance Criteria
Performance
Target
Step 3 Step 4
Assessment Results Improvement Plans
Step 5 Step 6
Improvements
Implemented
Step 6
48

• Assessment Plans Due
• Steps 1 – 4
• Mission, outcomes, measures and targets
• Friday, November 13, 2015
• Assessment Reports Due
• Steps 5 – 6
• Findings, improvement plans
• May ??, 2016 (specific date to be determined)
49

Office of Institutional Planning, Research and Assessment
306 Patton Hall
James Walke, Director james.walke@aamu.edu
256-372-8876
Dianne Bozeman, Assessment Coordinator dianne.bozeman@aamu.edu
256-372-8644
50