Program Outcome Assessment Summary Report
2011/2012 Academic Year
Department of Computer Science
Eastern Washington University
TABLE OF CONTENTS
1.
Summary of Actions Taken in Response to AY10/11 Assessment ........................................................ 2
2.
Summary of Program Outcome Assessment in AY11/12 ..................................................................... 3
3.
Advancement Programming Exam Scores ............................................................................................ 5
4.
Ethics Assessment ................................................................................................................................. 7
5.
4.1
Links between Program Outcomes and GenEd Objectives for PHIL 212 ...................................... 7
4.2
PHIL 212 Assessment Data ............................................................................................................ 8
Senior Project Rubric Scores ................................................................................................................. 9
5.1
Oral Communication ..................................................................................................................... 9
5.2
Teamwork ................................................................................................................................... 10
5.3
Technical Performance ............................................................................................................... 10
6.
Course-specific Outcome-related Learning Objectives....................................................................... 12
7.
Master’s Oral Communication Rubric Scores ..................................................................................... 13
8.
PHIL 212 Attachments......................................................................................................................... 14
1. Summary of Actions Taken in Response to AY10/11 Assessment
Following is a summary of actions taken on recommendations from the AY10/11 assessment summary:
1. The Advancement Programming Guidelines were modified to include explicit statements that (a) the
APE is intended to test implementation skills, as opposed to design and analytical skills, and (b) test
instructions should be kept as straightforward as possible.
2. The CS department Chair held preliminary discussions with the (now former) Director of the
Philosophy program, who is also an instructor for PHIL 212, Introduction to Ethics. It was agreed
that CS would investigate the possibilities of using the University’s General Education assessment for
Phil 212. The results of this attempt are detailed in Section 4.
3. Technical Performance and Teamwork rubrics were requested of all Senior Project teams in
AY11/12, in addition to the Oral Communication rubrics.
4. The Outcome-Related Learning Objectives (ORLOs) for CSCD 240 were revisited and revised in
consultation with Dr. Clark and Mr. Steiner, the two primary instructors for that course.
5. Instructors were asked to ensure that grade scales are clear (possible points) when citing student
performance in support of ORLOs.
6. Assessment of binary search trees was removed from CSCD 300. This subject is still assessed in
CSCD 320.
7. ORLOs were written for CSCD 327, which replaced CSCD 425 as a required course for all majors.
2
2. Summary of Program Outcome Assessment in AY11/12
The following table summarizes the assessment methods applied to Program Outcomes during the
2011/2012 academic year. Details of the assessment data are provided in the sections that follow.
Program Outcome
a) Students will have the ability to use current techniques, skills,
and tools necessary for computing practice.
b) Students will recognize the need for, and will have the ability
to engage in, continuing professional development.
c) Students will have the ability to analyze the local and global
impact of computing on individuals, organizations, and society.
d) Students will have the ability to communicate effectively with
a range of audiences.
e) Students will have an understanding of professional, ethical,
legal, security, and social issues and responsibilities.
f) Students will have the ability to function effectively on teams
to accomplish a common goal.
g) Students will have the ability to design, implement, and
evaluate a computer‐based system, process, component, or
program to meet desired needs.
h) Students will have the ability to analyze a problem and
identify and define the computing requirements appropriate to its
solution.
i) Students will have the ability to apply knowledge of
computing and mathematics appropriate to the discipline.
j) Students will have the ability to apply mathematical
foundations, algorithmic principles, and computer science theory
in the modeling and design of computer‐based systems in a way
that demonstrates comprehension of the tradeoffs involved in
design choices.
k) Students will have the ability to apply design and
development principles in the construction of software systems
of varying complexity.
Assessed Via
APE, CSCD 240, CSCD 300, CSCD
320, CSCD 327, CSCD 330, CSCD
340, CSCD 350, CSCD 370
Senior Project Oral Communication
Rubric, CSCD 340
Senior Project Oral Communication
Rubrics
PHIL 212
Senior Project Teamwork Rubric
APE, CSCD 260, CSCD 300, CSCD
320, CSCD 327, CSCD 330, CSCD
340, CSCD 350, Senior Project
Technical Rubric
APE, CSCD 240, CSCD 260, CSCD
300, CSCD 320, CSCD 350, Senior
Project Technical Rubric
APE, CSCD 240, CSCD 260, CSCD
300, CSCD 320, CSCD 330, Senior
Project Technical Rubric
CSCD 300, CSCD 320, Senior
Project Technical Rubric
CSCD 350, Senior Project Technical
Rubric
The following table summarizes assessment findings from each of the major assessment methods that
were used in AY11/12. The ORLO category only lists courses where concerns were found. See Section
6 for a summary of the ORLO report findings from each course.
Assessment Instrument
APE Exam
GECC Assessment of Phil
212
Result of Assessment
No concerns. No recommendations.
There is concern that the PHIL 212 Course Objectives are unable to help
cover Program Outcome (c), which was covered by the course that Phil
212 has replaced in the curriculum. Also, while some of the Phil 212
Course Objectives may help with Program Outcomes (d) and (e), as
3
Senior Project Oral
Presentation Rubric
Senior Project Teamwork
Rubric
Senior Project Technical
Performance Rubric
Course-specific OutcomeRelated Learning Objectives
(ORLOs)
Masters Oral Presentation
Rubric
detailed in section 4, the GECC schedule for assessment of these
objectives is presently inadequate.
No concerns. No recommendations.
This rubric was not applied to all senior project teams. It is
recommended that data from all senior project sections be collected
during AY 12/13.
This rubric was not applied to all senior project teams. It is
recommended that data from all senior project sections be collected
during AY 12/13.
All courses met objectives. No recommendations.
No concerns. No recommendations.
The following recommendations are made as a result of AY 11/12 assessments. These recommendations
will be reviewed by department faculty during Winter Quarter 2013:
1. It is recommended that the Assessment and Undergraduate Affairs committees collaborate to propose
a new course that will specifically address the societal impacts and ethical issues of computing. This
proposal should be communicated to the Director of the Philosophy program, the Dean of the College
of Science Health and Engineering, and the Dean of the College of Arts, Letters, and Education. The
course proposal should contain assessable outcome-related learning objectives that support program
outcomes (c), (d), and (e), with assessment of at least one category each year.
2. It is recommended that ORLOs be created for CSCD 110, 210, and 211, and that those ORLOs
include learning objectives that specifically address program outcome (e). These ORLOs should also
provide secondary coverage of the program outcomes covered by APE assessment.
3. It is recommended that the ORLOs for CSCD 340 modified to include learning objective(s) that
address program outcome (c).
4. It is recommended that the ORLOs for CSCD 350 be modified address program outcomes (c) and (e).
New instructor Dan Tappan should also review the entire set of CSCD 350 ORLOs.
5. The Teamwork and Technical Performance rubrics should be applied to all CSCD 490 project teams
in the AY12/13 cycle.
6. The Assessment Plan should be updated to reflect curriculum changes: PHIL 212 (or a new course)
replacement of CSCD 402, CSCD 327 replacement of CSCD 425, and the group of GUI courses
replacing CSCD 370.
4
3. Advancement Programming Exam Scores
APE scores support Program Outcomes (a), (g), (h), and (i). APE exams were administered during each
quarter of the academic year plus summer 2011. An 80% is required to pass the exam. Exam section
topics are:
(1)
(2)
(3)
(4)
Linked List Manipulation
Recursion
Data Abstraction and Class Design
General Programming
30%
20%
30%
20%
Summary data for AY 11/12 and the preceding two years:
Previous Year
Number of Exams Taken
109
Overall Average Score
75.1%
Overall Pass Rate
57.8%
Failures on 3+ Attempt
3.7%
Last Year
144
78.9%
64.6%
2.1%
This Year
138
79.0%
63.0%
4.0%
Fig 3.1 shows subsection scores along with pass rates for this year alongside the previous two years.
AY 11/12 vs. AY 10/11 and AY 09/10 APE Scores by Subsection
100.0%
90.0%
80.0%
70.0%
60.0%
AY 09/10
AY 10/11
AY 11/12
84.2%
78.9%
79.0%
75.1%
78.4%
75.3% 73.8%
73.6%
74.1%
71.0%
79.6%
82.8%
84.0% 82.1%
70.4%
64.6% 63.0%
57.8%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Pass Rate
Ave Score
Linked List
Recursion
Data Abstractopm
General
Fig. 3.1 – Summary APE Data by Exam Section
5
The average overall exam score was 79.0%, essentially unchanged since last year and an improvement
over the year before. The overall pass rate was slightly lower than last year. The proportion representing
failures on a third attempt rose to 4%, which is similar to the level of the year before last. The scores and
pass rates are acceptable.
Subsection scores were reasonably balanced, with little change in average performance. Gains from last
year in the General Programming area, which primarily covers File I/O and Exceptions, were maintained
this year. In previous years Java core instructors have expressed a fundamental tradeoff between recursion
and linked list performance, dependent on the class time devoted to each topic. This year’s data supports
that, with a rise in linked list performance occurring alongside a corresponding drop in recursion
performance. The performance in these two areas is reasonably well balanced, so no action is
recommended at this time.
6
4. Ethics Assessment
In previous years CSCD 402, Computing Ethics, supported Program Outcomes (c), (d), and (e). The
department was unable to continue delivering this course due to a rapid decrease in faculty resources
occurring at the same time as a rapid increase in enrollments. As a consequence, the department chose to
change the degree requirements to replace that course with PHIL 212, Introduction to Ethics. This is the
first assessment cycle on the effects of that replacement. PHIL 212 also satisfies a List 3 Humanities and
Fine Arts (HFA) requirement in the University’s general education structure, and is thus subject to
University assessment of course goals and objectives for the HFA list. The Computer Science department
agreed to analyze the possibility of using the University assessment of that course rather than ask for any
special assessment during the first cycle.
4.1
Links between Program Outcomes and GenEd Objectives for PHIL 212
Possible mappings between the CS Program Outcomes and General Education Goals and Objectives for
HFA courses are:
CS Program Outcome (c): Students will have the ability to analyze the local and global impact of
computing on individuals, organizations, and society.
This program outcome was supported by the CSCD 402 course, but no Goal or Objective for the
PHIL 212 course could be identified that specifically addresses the impact of Computing.
CS Program Outcome (d): Students will have the ability to communicate effectively with a range of
audiences.
HFA Goal 3: Students will develop reading skills as well as creative and analytic writing skills
appropriate to an introductory level in a discipline.
Objective 2: Students will produce written material consistent with introductory learning levels.
This course goal and objective supports the program outcome with respect to writing skills. Oral
communication skills are assessed elsewhere in the Computer Science curriculum (see section 5.1).
CS Program Outcome (e): Students will have an understanding of professional, ethical, legal,
security, and social issues and responsibilities.
HFA Goal 1: Students will employ the basic forms of criticism and expression in philosophy,
literature, and/or at least one of the arts, and they will understand the social and environmental
contexts of the artists and/or writers.
This course goal provides a partial, and somewhat weak, link to the program outcome in the sense
that it indicates that social context of the (ethics) philosophers is discussed in the course. It does not
specifically cover professional, legal, and security aspects of the program outcome.
HFA Goal 2: Students will understand the methods for interpretation and criticism appropriate to the
humanities and fine arts.
7
Objective 2: Students will draw competent interpretations based on given reasons and
evidence.
Objective 3: Students will demonstrate that differing interpretations of the same information
are possible and that different conclusions may be reached using the same evidence.
These course objectives support the program outcome in the area of analysis of different ethical
models and arguments.
4.2
PHIL 212 Assessment Data
Two sections of PHIL 212 were assessed according to University general education assessment
requirements during Winter quarter 2012. One of these sections was an online course. Both reports
(attached) assessed a single HFA objective:
HFA Goal 1: Students will employ the basic forms of criticism and expression in philosophy,
literature, and/or at least one of the arts, and they will understand the social and environmental
contexts of the artists and/or writers.
As discussed above, this assessment provides partial support of program outcome (e). Some important
conclusions of the two instructors from the attached reports are:
“… students are decreasingly well prepared to handle the textual analysis of difficult material that this
course relies upon.”
“ … while the majority of the students (30 out of 39) understood the material well enough to score around
or above the 70% mark, there is still more clustering than I would like to see at the top and the bottom of
the grade scale.”
The assessments from Phil 212 indicate mixed results in support of program outcome (e). The instructor
providing the first quote above has also made some personal comments to the CS department chair that he
has enjoyed having Computer Science students in his classes. Taken in combination with the quote from
the second instructor, we have to wonder if it wouldn’t be useful to ask for a separate assessment of CS
students in this course.
While there exist some reasonable possibilities linking course objectives to program outcomes (d) and (e),
there is no HFA goal or objective that appears to support program outcome (c), which was previously
covered by the department’s CSCD 402, Computing Ethics course. The department should therefore
identify another assessment opportunity for program outcome (c).
An additional difficulty with using the University’s assessment of HFA goals and objectives is that the
University’s assessment cycle for HFA has yet to be well-defined. The responsible university committee
(GECC) has been discussing ways in which to assess all general education objectives within a single
NWCCU accreditation cycle, but at present (AY 12/13), only a single general education category is being
asked to assess a single objective each year. It does not appear that there will be any assessment of HFA
courses during AY 12/13.
8
5. Senior Project Rubric Scores
Oral Communication rubrics were applied to senior project teams during Fall, Winter, Spring, and
Summer, in sections of both CSCD 488 and CSCD 490.
Teamwork and Technical Performance rubrics were applied to the work of CSCD 490 senior project
teams during Winter and Summer.
These assessments are tied to Program Outcomes as follows:
Oral Communication Rubric Scores: (b), (d)
Teamwork Rubric Scores : (f)
Technical Rubric Scores: (g), (h), (i), (j), (k)
5.1
Oral Communication
Fig 5.1 shows the average scores for presentations from the Fall and Spring offerings of CSCD 488, and
the Winter, Spring, and Summer offerings of CSCD 490. The Professional Development Awareness
score relates to Program Outcome (d), and the other scores relate to Program Outcome (b).
AY10/11 (N=13)
Capstone Oral Communication
AY11/12 (N=24)
4.00
3.62 3.69
3.66 3.64
3.40
3.50
3.50 3.40
3.49 3.41
3.36 3.47
3.48
3.34 3.47
3.26
3.50
3.47
3.30
3.40 3.49
3.25
3.25
3.00
2.50
2.00
1.50
1.00
0.50
or
e
s
Sc
es
e
er
ag
Av
De
v
Aw
a
re
n
ta
tio
en
Pr
of
Pr
es
El
oc
ut
io
n
n
t
ac
Co
nt
e
Ey
nic
M
ec
g
in
ga
g
En
ha
nc
die
Au
ag
M
an
e
Ti
m
s
e
t
em
en
s
al
Vi
su
to
se
on
Re
sp
Or
g
an
i za
tio
Q
n
&
ue
s
tio
ns
Fl
ow
0.00
Fig. 5.1 – Project Oral Communication Rubric Scores
9
The scores indicate that students’ are performing well against these learning objectives, and there appears
to be some improvement over the previous year in a couple of categories. No action based on Oral
Communication data is recommended.
5.2
Teamwork
Fig 5.2 shows the average scores for the teamwork performance of 9 senior project teams from the Winter
and Summer offerings of CSCD 490. These scores relate to Program Outcome (f).
AY 10/11 (N=4)
Teamwork Performance
AY 11/12 (N=9)
4.00
3.64
3.62
3.50
3.42
3.50
3.50
3.70
3.50
3.53
3.61
3.49
3.56
3.20
3.00
2.50
2.00
1.50
1.00
0.50
0.00
Involvement
Division of Effort
Communication
Group Dynamics
Troubleshooting
Average Score
Fig. 5.2 – Project Teamwork Rubric Scores
The scores indicate that students’ are performing well against these learning objectives. No specific
action based on Team performance data is recommended.
It is apparent, however, that this rubric was not applied to all senior project teams. It is recommended that
data from all senior project sections be collected during AY 12/13.
5.3
Technical Performance
Fig 5.3 shows the average scores for the technical performance of 9 senior project teams from the Winter
and Summer offerings of CSCD 490. These scores relate to Program Outcomes (g), (h), (i), (j), (k).
10
AY 10/11 (N=4)
AY 11/12 (N=9)
4.00
3.65
3.79
3.58
Capstone Technical Performance
3.71
3.55 3.59
3.55
3.63
3.50 3.52
3.50
3.56
3.50
3.53
3.62
3.37
3.23
3.00
2.50
2.00
1.50
1.00
0.50
er
ag
Av
D
oc
u
m
e
n
en
ta
tio
ni
ng
m
an
rfo
r
Pe
Pl
an
ce
n
Im
pl
em
en
ta
tio
gn
es
i
D
si
al
y
An
D
ef
in
iti
on
s
0.00
Fig. 5.3 – Project Technical Performance Rubric Scores
The scores indicate that students’ are performing well against these learning objectives. No specific
action based on Team performance data is recommended.
It is apparent, however, that this rubric was not applied to all senior project teams. It is recommended that
data from all senior project sections be collected during AY 12/13.
11
6. Course-specific Outcome-related Learning Objectives
Course-specific assessments were conducted for the following courses. Here we summarize the findings
for each course. Full reports are available in the /faculty_reposit/ABET/ directory for the course.
Course
CSCD
240
CSCD
260
CSCD
300
Term
Spring
12
Fall 11
Outcomes
(a), (h), (i)
Result of Assessment
Objectives met.
(g), (h), (i)
Objectives met.
Fall 11
CSCD
320
Fall 11
CSCD
327
Winter
12
CSCD
330
CSCD
340
CSCD
350
Spring
12
Fall 11
(a),(g),(h),(i),(j) Objectives met, including some of the binary tree objectives
that have been removed from the ORLO. Additional time
dedicated to debugging strategies as a result of conversion to
5 hours per week seems to have helped as instructor is
experiencing fewer office visits.
(a),(g),(h),(i),(j) Objectives met, although instructor intends to add examples
and perhaps slow down the pace in an attempt to improve
student weaknesses in analysis and algorithm design.
(a),(g)
Objectives met. Instructor plans to provide more examples
and in-class exercises to help students better digest
normalization concepts.
(a),(g),(i)
Objectives met.
CSCD
370
(a),(b),(g)
Objectives met.
Spring
12
(a),(g),(h),(k)
Winter
11
(a)
Objectives were met. Instructor has difficulty measuring
student abilities with use cases in this course and believes
the ORLOs should be reviewed with new Software
Engineering faculty member that is starting in Fall 12.
Objectives met.
12
7. Master’s Oral Communication Rubric Scores
The Oral Communication rubric was also applied at the defense of 7 Master’s degree candidates during
AY 11/12. The rubric was applied by audience members present at the exam. The results are summarized
in Fig. 7.1.
AY10/11 (N=7)
Master's Oral Communication
AY11/12 (N=7)
3.98
4.00
3.77
3.75
3.83
3.67
3.57
3.57
3.81
3.76
3.71
3.57
3.88
3.77
3.74
3.54
3.53
Presentation
Prof Dev
Awareness
3.45
3.50
3.78
3.75
3.59
3.00
2.50
2.00
1.50
1.00
0.50
0.00
Organization Response to
Questions
& Flow
Visuals
Time
Management
Engaging
Audience
Mechanics Eye Contact
Elocution
Fig. 7.1 – Master’s degree Oral Communication Rubric Scores
Although not generally as high as the preceding year, all of the scores are acceptable. No action based on
Master’s Oral Communication scores is recommended.
13
8. PHIL 212 Attachments
14
15
16
17
18
19
20
21
22
23