Using IDEA for Assessment,
Program Review, and SACS
University of Alabama Birmingham
September 11, 2012
Shelley A. Chapman, PhD
Plan for this Session
• Program Evaluation & Assessment of
Student Learning
• Group Summary Reports
• Aggregate Data File
• Benchmarking Reports
• Accreditation Guides
What makes IDEA unique?
1. Focus on Student
Learning
2. Focus on Instructor’s
Purpose
3. Adjustments for
Extraneous Influences
4. Validity and Reliability
5. Comparison Data
6. Flexibility
Student Learning Model:
2 Assumptions
Assumption 1:
Types of learning
must reflect the
instructor’s purpose.
Student Diagnostic Form
Assumption 2:
Effectiveness
determined by
students’ progress on
objectives stressed by
instructor
Diagnostic Report Overview
 Page 1 – Big Picture

How did I do?

Page 2 – Learning Details

What did students learn?

Page 3 – Diagnostic

What can I do differently?

Page 4 – Statistical
Detail

Any additional
insights?
The Big Picture
Your Average
(5-point Scale)
A. Progress on Relevant
Objectives1
Four objectives were selected as
relevant (Important or Essential—see
page 2)
1If
Raw
Adj.
4.1
4.3
you are comparing Progress on Relevant Objectives from one
instructor to another, use the converted average.
Progress
On
Relevant
Objectives
4.3 + 4.3
4.1
4.2 4
3.6
5
Summary Evaluation: Five-Point Scale
Your Average
Score
(5-point scale)
Report
Page 1
Raw
Adj.
4.1
4.3
4.7
4.9
4.1
4.4
D. Average of B & C
4.4
4.7
Summary Evaluation
(Average of A & D)
4.3
4.5
A. Progress on Relevant Objectives
Four objectives were selected as relevant
(Important or Essential—see page 2)
Overall Ratings
B. Excellent Teacher
C. Excellent Course
50%
25%
25%
Individual Reports to Group
Reports
The Group
Summary Report
How did we do?
How might we improve?
Defining Group Summary
Reports (GSRs)
• Institutional
• Departmental
• Service/Introductory Courses
• Major Field Courses
• General Education Program
GSRs Help Address Questions
• Longitudinal
• Contextual
• Curricular
• Pedagogical
• Student Learningfocused
Adding Questions
Up to 20 Questions
can be added
• Institutional
• Departmental
• Course-based
• All of the above
Local Code
Use this
section
of the FIF to
code types of
data.
Defining Group Summary
Reports
• Local Code
• 8 possible fields
• Example: Column one – Delivery Format
• 1=Self-paced
• 2=Lecture
• 3=Studio
• 4=Lab
• 5=Seminar
• 6=Online
Example from Benedictine University
Example Using Local code
Assign Local Code
• 1=Day, Tenured
• 2=Evening, Tenured
• 3=Day, Tenure Track
• 4=Evening, Tenure
Track
• 5=Day, Adjunct
• 6=Evening, Adjunct
Request Reports
• All Day Classes
• Local Code=1, 3, & 5
• All Evening Classes
• Local Code=2, 4, & 6
• Courses Taught by
Adjuncts
• Local Code=5 & 6
Description of Courses Included in this Report
Number of Classes Included
Diagnostic From
Short Form
Total
Number of Excluded Classes
42
27
69
0
Response Rate
Classes below 65% Response Rate
Average Response Rate
2
85%
Class Size
Average Class Size
Page 1 of GSR
20
UAB Spring 2012
Page 1 of GSR
Assessment of Learning
What are our faculty emphasizing?
How do students rate their learning?
How do our courses compare with others?
How do our students compare with others (selfrated characteristics)?
What efforts can we make for improvement?
(How can we “close the loop”?)
UAB Core Competencies
Shared Vision for a
UAB Graduate
Communication
Possible IDEA
Learning Objectives
8, 11
Knowledge
1, 2, 4, 7
Problem Solving
3, 9, 11
Citizenship
10, 12, extra questions
Are we targeting “Core Competencies”
in the Core Curriculum?
IDEA Learning Objectives
UAB Core Curriculum
Courses
1
2
3 4 5 6 7
EH 101 English Composition
X X
BY 101 Topics in Contemporary
Biology
X X X
MA 105 Pre-Calculus Algebra
X X X
ARH 101 The Art Experience
X
HY 101 Western Civilization I
X X
PY 101 Introduction to
Psychology
X
8
9
10
11 12
X
X
X
X
X
X
X X
X
X
Are we targeting “Core Competencies”
in the Core Curriculum?
IDEA Learning Objectives
UAB Core Curriculum
Courses
1
2
3 4 5 6 7
EH 101 English Composition
X X
BY 101 Topics in Contemporary
Biology
X X X
MA 105 Pre-Calculus Algebra
X X
ARH 101 The Art Experience
X
HY 101 Western Civilization I
X X
PY 101 Introduction to
Psychology
X
8
9
10
11 12
X
X
X
X
X
X
X X
X
What are We Emphasizing?
Percent of Classes Selecting Obj. as
Important or Essential
Page 2
This Group
Institution
IDEA
System
Objective 1
16%
70%
78%
Objective 2
13%
59%
75%
Objective 3
41%
58%
75%
Objective 4
32%
35%
55%
Objective 5
23%
19%
32%
Objective 6
32%
14%
25%
Objective 7
22%
27%
27%
Objective 8
78%
43%
47%
Objective 9
19%
23%
41%
Objective 10
7%
11%
23%
Objective 11
68%
42%
49%
Objective 12
20%
23%
41%
3.7
4.2
5.7
Average # of Obj. Selected
UAB Spring 2012
What are We Emphasizing?
Page 9
Section B
Number
Rating
Percent indicating
amount required
None or
Little
Some
Much
Writing
66
2%
17%
82%
Oral Communication
66
6%
42%
52%
Computer Application
66
50%
44%
6%
Group Work
66
27%
59%
14%
Mathematics/Quantitative
Work
65
97%
3%
0%
Critical Thinking
66
0%
30%
70%
Creative/Artistic/Design
66
61%
33%
6%
How Did Students Rate their
Learning on Core Competencies?
Do Students’ report of learning meet our expectations?
Objective 1: Gaining factual knowledge (terminology, classifications, methods, trends)
Raw
Average
Adj.
Average
# of
Classes
This
Report
3.9
3.9
11
Institution
4.2
4.2
3,963
IDEA
System
4.0
4.0
31,991
Pages 5 and 6
Percent of Classes Rating
This report
At least 4.0
At least 3.75
At least 3.5
Institution
IDEA System
0
20
40
60
80
100
How do students rate their learning?
Page 3
Part 1: Distribution of Converted Scores
Compared to the IDEA Database
Overall Progress Ratings (Courses)
Page 3
Percent of Classes at or Above the
IDEA database Average
Overall Progress Ratings (Courses)
Page 4
Part 3: Percent of Classes at or Above
This Institution’s Average
90%
80%
70%
60%
50%
Raw
Adjusted
40%
30%
20%
10%
0%
PRO
Excellent
Teacher
Excellent
Course
Summary
Which teaching methods might we use to improve
learning?
Page 7
Teaching Methods and Styles
Stimulating Student Interest
#
Classes
Av.
s.d.
42
3.8
0.5
15. Inspired students to set and achieve goals which
really challenged them
Frequently
Infrequently
# 15
0
10
20
30
40
Assessing the QEP for UAB
UAB’s Core
Competencies
IDEA Learning Goals
Communication
8 Developing skill in expressing myself
2, 7, 8, 15, 16, 18, 19
orally or in writing
11 Learning to analyze and critically
evaluate ideas, arguments, and points of
view
1 Gaining factual knowledge (terminology, 1, 2, 4, 6, 7, 8, 10, 12, 13, 15, 19
classifications, methods, trends)
2 Learning fundamental principles,
generalizations, or theories
4 Developing specific skills, competencies,
and points of view needed by professionals
in the field most closely related to this field
7 Gaining a broader understanding and
appreciation of intellectual and cultural
activity (music, science, literature, etc.)
3 Learning to apply course material (to
2, 7, 8, 9, 13, 14, 15, 18, 19
improve thinking, problem solving, and
decisions)
9 Learning how to find and use resources
for answering questions or solving
problems
11 Learning to analyze and critically
evaluate ideas, arguments, and points of
view
10 Developing a clearer understanding of,
1, 2, 4, 6, 7, 8, 11,13, 15, 16, 18,
and commitment to, personal values
12 Acquiring an interest in learning more
by asking my own questions and seeking
answers
Knowledge
Problem solving
Citizenship
IDEA Teaching
Methods Associated
with Progress on
Learning Goal
Relationship
of Learning
Objectives to
Teaching
Methods
How do students view course work demands?
Page 8B Student Ratings of Course Characteristics
Diagnostic Form Item # & Item
33. Amount of
Reading
Average
% Classes
Below 3.0
% Classes
4.0 or Above
This Report
3.4
21%
24%
Institution
3.3
31%
19%
IDEA System
3.2
33%
15%
34. Amount of
work in other
(non-reading)
assignments
This Report
3.3
24%
10%
Institution
3.4
23%
20%
IDEA System
3.4
21%
18%
35. Difficulty
of subject
matter
This Report
3.2
19%
0%
Institution
3.5
13%
19%
IDEA System
3.4
20%
18%
Aggregate Data File
Allows you to
• Use Excel
Spreadsheet
• Use with SAS or
SPSS
• Ask other types
of questions
• Display data in
different ways
A. Progress of Relevant Objectives-Converted Score Expected
Category
Distribution
Much Higher (63 or higher)
10%
Higher (56-62)
20%
Similar (45-55)
40%
Lower (38-44)
20%
Much Lower (37 or lower)
10%
Fall Spring Fall
2010 2011 2011
Raw Raw Raw
6%
7% 7%
32%
34% 31%
46%
46% 44%
11%
9% 10%
5%
5% 8%
Expected
B. Excellence of Teacher-Converted Score Category Distribution
Much Higher (63 or higher)
10%
Higher (56-62)
20%
Similar (45-55)
40%
Lower (38-44)
20%
Much Lower (37 or lower)
10%
Fall Spring Fall
2010 2011 2011
Raw Raw
Raw
0%
0% 0%
35%
38% 34%
42%
43% 40%
12%
11% 12%
11%
9% 14%
C. Excellence of Course-Converted Score Category
Much Higher (63 or higher)
Higher (56-62)
Similar (45-55)
Lower (38-44)
Much Lower (37 or lower)
Expected
Distribution
10%
20%
40%
20%
10%
Fall Spring Fall
2010 2011 2011
Raw Raw Raw
9%
11% 8%
26%
26% 29%
41%
40% 40%
14%
13% 13%
10%
10% 10%
Summary Evaluation (Average of A, B, C)-Converted Expected
Fall Spring Fall
Score Category
Distribution 2010 2011 2011
Much Higher (63 or higher)
10%
5%
4% 4%
Higher
(56-62)
20% 33%
35% 33%
Similar
(45-55)
40% 45%
45% 44%
Lower
(38-44)
20% 10%
10% 10%
Much Lower (37 or lower)
10%
7%
6% 9%
Fall
Spring
Fall
Primary Instructional
2010
2011
2011
Approaches
Primary Primary Primary
Lecture
61%
60%
61%
Discussion/Recitation
7%
8%
10%
Seminar
7%
9%
9%
Skill/Activity
7%
6%
7%
Laboratory
8%
9%
9%
Field Experience
1%
1%
1%
Studio
2%
3%
2%
Multi−Media
0%
0%
0%
Practicum/Clinic
0%
0%
0%
Other/Not Indicated
6%
4%
2%
Number Classes Rating 787
677
859
Fall
Spring
Fall
2010
2011
2011
Secondary Instructional Secon- Secon- SeconApproaches
dary
dary
dary
Lecture
16%
13%
16%
Discussion/Recitation
30%
27%
30%
Seminar
2%
3%
4%
Skill/Activity
11%
13%
13%
Laboratory
3%
6%
4%
Field Experience
2%
3%
2%
Studio
0%
0%
1%
Multi−Media
2%
3%
3%
Practicum/Clinic
1%
1%
1%
Other/Not Indicated
34%
31%
28%
Number Classes Rating 787
677
859
Instructors’ Reports on Course Emphases:
Selected Pairings-Writing and Oral Communication
Course Emphases:
Writing
None or Little
Some
Much
Fall Spring Fall
2010 2011 2011
25% 26% 22%
46% 45% 44%
29% 30% 34%
100%
80%
60%
None or Little
Some
40%
Much
20%
0%
Fall 2010
Course Emphases:
Oral communication
None or Little
Some
Much
Fall Spring Fall
2010 2011 2011
36% 35% 32%
47% 47% 44%
17% 18% 24%
Spring 2011
Fall 2011
100%
80%
60%
None or Little
Some
40%
Much
20%
0%
Fall 2010
Spring 2011
Fall 2011
Instructors’ Reports on Course Emphases:
Selected Pairings-Critical Thinking & Writing
Course Emphases:
Critical thinking
None or Little
Some
Much
Fall Spring Fall
100%
2010 2011 2011
19% 19% 18%
80%
49% 44% 41%
32% 37% 41%
60%
None or Little
Some
40%
Much
20%
0%
Fall 2010
Course Emphases:
Writing
None or Little
Some
Much
Fall Spring Fall
2010 2011 2011
25% 26% 22%
46% 45% 44%
29% 30% 34%
Spring 2011
Fall 2011
100%
80%
60%
None or Little
Some
40%
Much
20%
0%
Fall 2010
Spring 2011
Fall 2011
Highlights for Sample U
• Remarkably similar profiles
across terms
• Overall response rates
ranged from 66% to 80%
• 1st term in which
administration was primarily
online achieved a 75%
response rate
• Transition from paper to
online (fall 2009 to fall
2010) does not show major
differences in profiles
• Sample U faculty focus on
4-5 outcomes as
essential/important
• Over the last 3 terms, a
significant increase on
several objectives has been
observed: application of
course material, oral and
written communication
skills, & analysis and critical
thinking skills (objectives 3,
8, & 11, respectively)
Benchmarking
Institutional and Discipline Reports
Benchmarking Reports
Comparison to
• 6-10 Peers
• Same Carnegie
Classification
• IDEA database
Comparison Groups
Your University
----------
Peer*
----------------------------
Carnegie
----------
National
----------
----------
----------
----------
* Peer group is based on 6-10 institutions identified by your institution
Benchmarking Reports
The student,
rather than the class,
is the unit of analysis
Percentage of positive
ratings is given
rather than
averages
Response
Rates
Your University: Student
participation is similar to that
of each comparison group
Your University=79%
Peer=77%
Carnegie=79%
National=75%
Students’ Perceptions
Graduate Students
Instructional Objectives
Selected by Instructors
Instructors’ Intentions/ focus
Students’ Self-Reported
Progress on Learning
Objective 3: Learning to apply course material (to improve
thinking, problem solving, and decisions)
IDEA
Objective 3
Learning to
apply course
material (to
improve
thinking,
problem
solving, and
decisions)
% of total classes where instructor selected objective as “Essential” or “Important”
IDEA
Objective 8
Developing
skill in
expressing
oneself orally
or in writing
Objective 8: Developing skill in expressing oneself orally or in
writing
% of total classes where instructor selected objective as “Essential” or “Important”
Teaching Methods and Styles
Reported by Students
(Diagnostic Forms Only)
• Fostering
Student
Collaboration
• Encouraging
Student
Involvement
Benchmarking Look at “General
Education”
Benchmarking Look at “General
Education”
Using Aggregate Data for
Assessment
Benchmarking:
Discipline Report
Benchmarking: One
Year or 3-5 Year Trend
Report
UAB
Core
Competencies
Group Summary
Report,
Include Extra
Questions
Course Learning
Outcomes
Program
Learning
Outcomes
Course Learning
Outcomes
Program
Learning
Outcomes
Course Learning
Outcomes
Program
Learning
Outcomes
Course Learning
Group Summary
Report
Course Learning
Outcomes
Outcomes
Accreditation
Guides
• SACS
• NCATE
• CACREP
Questions?