“When learners are at a distance, the
careful design of assessments is
particularly important, because
society somewhat unfairly imposes
higher expectations on the
assessment of online learners.”
Oosterhof, et al. (2008) Assessing Learners Online
Assessing Online Instruction at
Carteret Community College
?
?
?
?
?
?
Donald Staub, Title III Director
Carteret Community College
The Lineup
• DL at CCC
• The Need to Assess
• How We’re Doing It
• Conclusions & Solutions
• Discussion
DL @ CCC
• DL is 10 years old
• 34 programs @ CCC
• ~1700 enrollment SP09
• Fall 2008
• 348 total sections taught
• 270 seated sections
• 61 Internet
• 15 Hybrid
The Need to Assess
• SACS
• Common
Sense
• it’s
the right thing to do
• we
need some answers
“Comparability of distance education
programs to campus-based programs
and courses is ensured by the evaluation
of educational effectiveness, including
assessments of student learning
outcomes, student retention, and student
satisfaction.”
(CS 3.3.1)
COC Distance Education Policy Statement
http://www.sacscoc.org/pdf/081705/distance education.pdf
DL @ CCC
• 270 seated sections
• 73% success / 22% failed or withdrew (15% W)
• 61 internet sections
• 65% success / 33% D, F, WD (16% W)
• 15 hybrid sections
• 81% success / 18% D, F, WD (9% W)
• FA07 internet: 59 sections; 34% D, F, W
• FA06 internet: 48 sections; 34% D, F, W
How We’re Doing It
• QAP
• Grade Distributions
• ILLOs & PLLOs
• Same instructor
The QAP
(Quality Assessment Plan)
• Developed in-house (using models)
• Peer-reviewed by content specialist
• Stipends
• All current online courses have been reviewed
• All new online courses must be reviewed
http://ccctitle3.wordpress.com/
The QAP Score Card
http://ccctitle3.wordpress.com/
The QAP Score Card: Learning
Grade Distributions
(Fall 2008)
TOTAL
ENROL
Seat
IN
HY
A
B
C
D
F
WD
WD*
3643 0.31 0.26 0.14 0.72 0.04 0.07 0.06 0.09
1159 0.32 0.22 0.12 0.65 0.05 0.12 0.08 0.08
188 0.48 0.22 0.11 0.81 0.04 0.05 0.02 0.06
0.26
0.33
0.18
ILLOs
• Communication
• Computer Literacy
• Critical Thinking
• Humanities & Fine Arts
• Information Literacy
• Personal Growth & Responsibility
Computer Literacy - CIS 110
Outcome: Students demonstrate proficiency in the
use of computer technology including application
software, system software, and internet browsers.
Assessment: Final Exam in CIS 110. This exam
consists of a hands-on skills assignment for Word,
Excel, Access and PowerPoint and a 25 question
Multiple choice assignment with questions
regarding computer hardware, system software,
internet browsers and other related computer
terminology.
Computer Literacy
5 Sections: CIS 110 3-WB; 2-IN
• Attrition:

IN01: 35%

IN02: 26%

WB03: 17%

WB04: 28%

WB05N: 18%
• Cross-section comparison (i.e. one course,
different instructors)
Student Name
Access
pts.
(15%)
Word pts. Excel pts. PPT pts.
(20%)
(15%)
(10%)
Integratio Theory
n pts.
pts.
(15%)
(25%)
Final
Score
75%
A
19
14
10
12
15
15
85
1
B
17
14
10
10
13
21
85
1
C
17
12
10
15
15
25
94
1
D
10
7
0
0
0
12
29
E
15
15
10
14
15
16
85
1
F
19
15
10
15
15
23
97
1
G
H
I
J
K
L
M
N
16
19
20
20
10
19
15
14
15
15
12
12
15
15
15
11
10
10
10
10
10
10
10
10
12
15
10
13
15
15
15
12
15
15
12
15
8
15
15
15
20
24
22
25
20
24
16
24
88
98
86
95
78
98
86
86
1
1
1
1
1
1
1
1
16.42857
13.35714
9.285714
12.35714
12.92308
20.23077
85
82.14%
89.05%
92.86%
82.38%
86.15%
80.92%
Average
% of students >=75
92.86%
Word pts.
(20%)
IN01
IN02
WB03
WB04
WB05
80
91
91
92.5
82
Excel pts.
(15%)
73.8
95
75.5
89
89
PowerPoint
pts. (10%)
95.7
100
96
100
92.8
Access pts.
(15%)
64.7
79
78
85
82
Integration
pts. (15%)
83
94
88.5
92.7
86
Theory pts.
(25%)
81
81.9
80.8
77
80.9
Final Score
79
89
84.6
80.9
85
Computer Literacy - CIS 110
Same instructor teaching one section of each
<75%
seated
<75%
24
16
25%
15.75
9
16
12
25%
17.8
5
12.3
5
12.2
5
6
13.4
13.25
19.29
73.3
16
6
4
3
6
6.3
12.75
13.8
20.5
81.2
11
6
4
3
6
Spring 2008
online
# of Ss @ 10%
Final # of Ss
Attrition
Word (20)
Excell (15)
PPT (10)
Access (15)
Integration (15)
Theory (25)
Final (100)
Personal Growth
& Responsibility
ACA 115
Outcome: Students demonstrate the ability
to understand and manage self effectively.
Points
Understanding Self ~~ Grading Criteria
20
Points
1. Identify long term and sho rt term educa tiona l goa ls:
a. Two (2 ) Long Te rm: 1 yea r—5 years
b. Th ree (3) Short Term: 1 week—16 weeks
Completed Goal Deve lopment Form must be sub mitt ed.
2. Evaluate the results of the foll owing inven tories in
relation to your educa tiona l goa ls:
a. Career Inven tory
b. Lea rning Styles Inven tory
c. Persona lit y Inven tory
Responses should be written in paragraph form using complete
sentences and approp riate grammar.
3. Identify five va lues that shap e you r beli ef sys tem.
Expl ain why these values are impo rtant to you.
(4 pts. per
value with
explanation)
Responses should be written in paragraph form using complete
sentences and approp riate grammar.
30
Points
(6 pts. each)
20
Points
(5 pts. each
for results + 5
pts. total for
explanation)
Points
Earned
Points
Managing Self ~~ Gra ding Criteria
Advisement Status Sheet
15
Points
(3 pts. each)
1. Placement Test or SAT Scores
2. Distance Learning Orientation
3. Keyboa rding Test
4. Computer Lit eracy Test
5. Advisor/Program Coordinator Information
Completed Advisement Status Shee t must be sub mitt ed.
Two Semester Class Schedule
15
Points
(5 pts. each)
a. Complete the curriculum che ck shee t for you r
Program of Study (Major) by check ing cour ses
cur rently enro ll ed in and indicating the grade for
cour ses completed. Colle ge Preparation cou rses shou ld be
list ed on a separate piece of paper and attached to the
check shee t.
b. Create a trial course schedu le for Spring 2009 .
c. Create a trial cour se schedu le for Summer 2009 or for Fall 2009.
Personal Growth
& Responsibility
ACA 115
Personal Growth
& Responsibility
ACA 115
Advisement Project
• In the fall of 2008, 79% of all ACA students (10
sections) scored 80 or above.
• For online sections (2 sections), 70% (7 out of
10) scored 80 or above.
Personal Growth
& Responsibility
PSY 150
Row 1
Sec 04
IN06
IN07
Row 2
Sec 04
IN06
IN07
Row 3
Sec 04
IN06
IN07
2+
7/10 (70%)
4/6 (66%)
4/6 (66%)
2+
9/10 (90%)
1/6 (16%)
5/6 (83%)
2+
7/10 (70%)
3/6 (66%)
6/6 (66%)
3/3
2/10 (20%)
1/6 (16%)
3/3
5/10 (50%)
3/6 (50%)
3/3
5/10 (50%)
2/6 (33%)
1
2
3
Missing,
inadequate or
marg inally
compl eted
Acceptable or
adequately met
requi rements
Exceptional or
exceeded
requi rements
No action p lan.
Hint of action
plan withou t
specifics.
Plan do es not
relate to self
help book.
Lacks tim eli ne.
Action plan has
specifics and
relates to self
help book.
Action plan has
tim eli ne.
Action plan
meets acceptable
standard s as well
as include s
additi ona l
research beyond
self help book .
Obstacles not
identif ied.
Only one
obstacle
identif ied.
Obstacles
identif ied but
lacks how to
ove rcome them.
Studen t identifi es
at least one
obstacle and plan
for ove rcomi ng
obstacle, but
lacks depth.
Identified
multi ple
obstacles and
determi ned ho w
to overcome
them.
Fail ed to
Student
demonstrate
analyze d how
awareness of
his or her life
how life will be
wo uld be
im proved by
improved by this proposed
change.
chang e.
Studen t mentions
chang e, but
ana lysis and
exp lana tion are
cursory.
Demonstrated
awareness of
how life will be
im proved by
proposed change .
Student
developed an
action plan for
personal grow th
or responsibility.
Student
identified
obstacles which
could prevent
grow th or
change and
identified how to
overcome said
obstacles.
Points
earned
PLLOs
Same instructor teaching one section of each
(take a look at a common assignment from each section)
• ACA 115
• ACC 120
• BUS 110
• ECO 251
• EDU 119
ACA 115 Learning Styles Project
Students identify and explain the
strategy used to prepare for an exam.
<75%
Instructor 1 WB02
WB05
18%
Instuctor 2 WB01
IN10
41%
36%
IN09
6%
24%
ACC 120
BUS 110
ECO 250
EDU 119
4 test test items related to: working/communicating
with parents, parental rights, and regulations dealing
with parental contact/information.
Item # # incorrect
– online
4
1
21
0
22
2
23
1
n=20
# incorrect
- seated
0
0
2
1
n=12
Conclusions & Solutions
“The largest issue confronting online
assessments is that of the dynamics that
occur when the instructor and student do
not share the same immediate space.”
Milam, et al. (2004). In, Developing and Implementing
Assessment of Student Learning Outcomes
• Instructors who don’t treat online & seated
equitably (e.g. additional support to seated sections)
• Using the observation instrument correctly. Some
of this was a direct result of them using a document
that was scanned and they had to print/copy to use.
I believe some of the students had trouble with the
process. (EDU 119)
• Math & the fax machine
“The seated students did however have the
opportunity to submit their observation checks
sheets for revision if needed.”
“Students will no longer have to print the
inventory results. Instead, they will write a
narrative explaining their results.”
“Students will no longer need to submit a
curriculum check sheet (on-line students had
difficulty accessing them from the website).”
Discussion
“In many instances, the assessment of an
online course is not in alignment with
the type of instruction that may have
occurred in the course.”
Palloff & Pratt (2009) Assessing the Online Learner
Donald Staub
Title III Director
Carteret Community College
staubd@carteret.edu
252- 222-6010
http://ccctitle3.wordpress.com/