2009-2010
Annual Program Assessment for the
MS Experimental Psychology Graduate Program
Final Report
Submitted to Dr. Stephanie Stein, Chair
Department of Psychology
by
Dr. Wendy A. Williams
Program Director
June 14, 2009
1
Table of Contents
I.
Introduction and Overview……………………………………………………….….3
II. Program Enrollment …………………………………………………………..……4
Admissions, Enrollment Changes, Enrollment Trends,
Recruitment, Degrees conferred, Attrition and Retention
III. Assessment of Student Learning Overview …………………………………….….9
Review of Record Keeping Procedures, Number of Enrolled Students
IV. Major Milestones ……………………………………………………………….…12
General Academic Assessment Data, Course of Study Forms, Option
Approval Forms, IACUC/HSRC Approvals, Theses Accepted, Degrees
Conferred, Conference presentations
V. Time to Completion Data …………………………….……………………………15
Thesis Defense Meetings/Time to Completion
VI. Programmatic Course Offerings …………………………………….…………….17
VII. Quarterly Survey of Student Progress …………………………………………....18
Academic Performance Summary, Cumulative GPA,
Itemized Assessments:
• Content knowledge
• Skills mastery
• Quantitative analytical skills
• Logic-based analytical skills
• Comprehension
• Written and verbal expressive skills
•Appreciation of empirical evidence
• Ability to succeed in the experimental psychology program
• Disposition consistent with the field of experimental psychology
VII. Assessing the Quarterly Survey ………………………………………………….27
Previous Limitations, Results of solutions Implemented, Limitations to the
new 2009-2010 Assessment
IX. MS Experimental Psychology Program Assessment (Student Version) ………….29
Overview, Breakdown by Program, Breakdown by Year, Open-ended
Comments
X. Concerns raised in Previous Assessments (2007-2008; 2008-2009) ………………33
XI. Conclusions ………………………………………………………………………37
2
2009-2010 Annual Program Assessment 3
MS Experimental Psychology
2009-2010 Annual Program Assessment for the
MS Experimental Psychology Graduate Program
Final Report
Introduction and Overview
The 2009-2010 academic year was spent continuing to improve the MS Experimental
Psychology program assessment procedures, making important changes to the ABA
specialization program, developing a new colloquium course for all MS Experimental
Psychology students, moving the assessments to an on-line format, and recruiting for the
2009-2010 school year. A significant change in assessment procedures was issued on
May 20, 2009. Program and departmental assessment data must be received by Dr. Tracy
Pellet on or before June 15th. Every effort has been made to gather all data available for
Spring 2010, however, some data was not available and was therefore not included.
Moreover, it makes more sense to talk about our program in terms of the academic year
because it keeps cohorts of students together. Program changes and university changes
make it difficult to lump students together across academic years. Therefore, the
qualitative nature of this report will refer specifically to the 2009-2010 academic year.
All evaluations are based on grades or formalized assessments submitted before June 14,
2010. Averages may not reflect responses from all faculty, due to the early deadline.
2009-2010 Annual Program Assessment 4
MS Experimental Psychology
Program Enrollment
Admissions
Thirteen new students were admitted to the MS Experimental Psychology program this
year. Three females and ten males. Eight were accepted into the General Experimental
program; five were admitted to the ABA specialization. Their stats are outlined in Table
1.
Table 1. Average GPA and GRE scores (verbal and quantitative) for 2009-2010
incoming graduate class in the MS Experimental psychology program.
2008-2009
2009-2010
N
6
13
GPA
3.40
3.53
GRE-Verbal
487
472
GRE-Quant
625
585
The range for undergraduate GPA (last 90 credits) for the thirteen applicants was 2.883.98; the range for GRE-verbal scores was 350-620; the range for GRE quantitative
scores was 450-720.
Applicant Targets. The MS Experimenal psychology program’s minimum entrance
requirements are:
1) BA/BS in psychology or a related field
2) GPA of 3.0
3) combined GRE score (verbal+quantitative) of 900.
This year, one student fell below the 3.0 GPA cutoff. This student was a CWU graduate;
well known in the department; had earned a 3.89 in winter 2009; and was working to
complete an honors thesis. Based on his reputation, his most recent work, and on his
interest to continuing his work with his honors thesis mentor, he was admitted. Thus far
he has proved to be highly successful in the program. The range for the combined GRE
score was 860 – 1340. Two students’ GRE total scores fell below the 900 cutoff. Both
student earned GPA scores well above the minimum. The overall average GRE combined
score was 1058 – significantly above the minimum requirement. All applicants also need
to have a research interest that aligns with at least one of our faculty members. This was
true for all thirteen students admitted to the program.
Enrollment changes
During the course of the 2009-2010 school year, the status of several students’ enrollment
changed. Three students announced they were dropping from the program. The program
director met several times with each student to discuss their individual issues. None of
the students left the program for reasons related to the curriculum, funding, staffing, or
performance. All three students left for personal reasons. Two left after deciding that
graduate school was not a good fit for them. The third has decided to remain active but to
2009-2010 Annual Program Assessment 5
MS Experimental Psychology
not enroll. She will be searching for a MS program in Organizational Development, a
program that is no longer available at CWU.
Also during the 2009-2010 school year, two students changed their program affiliations.
One student changed from the General Experimental program to the ABA program. The
other student changed from the ABA program to the General Experimental program.
Both students changed their specializations under the guidance of (and approval from)
their academic advisors, the program coordinators and the program director.
Enrollment Trends for 2009-2010
Enrollment in the MS Experimental Psychology programs increased from 5 students in
2007-2008 to 6 students in 2008-2009 to 13 students in 2009-2010. This year (20092010) the enrollment in the MS Experimental Psychology program more than doubled
from last year, and was greater that the two previous years combined.
This year (2009-2010) was the second year since the creation of the Primate Behavior
and Ecology Program, which was previously expected to significantly reduce enrollments
in the Psychology MS Experimental Psychology program. Only one student enrolled with
an interest in primate behavior in fall 2009, but he changed his interest to social
psychology after the first quarter. Hence, our reliance on primate behavior (CHCI)
students has decline from more than half (3/5) in 2007-2008 to one third (2/6) in 20082009 to zero (0/13) in 2009-2010. This change is noteworthy in that primate behavior
has historically been the primary draw to the MS Experimental Program for over 15
years. The addition of new faculty and the new ABA program have more than
compensated for the elimination of CHCI as part of the psychology department.
Similarly, 2009-2010 was also the second year for the Applied Behavior Analysis
specialization within the MS Experimental Program. Enrollment in the ABA
specialization more than double from 2 students in 2008-2009 to 5 students in 2009-2010.
Furthermore, the Ms Experimental Psychology program appears to be growing as it
reaches out to potential students with a more diverse faculty interest base (with research
interests that include physiological psychology and neuroscience, human cognition and
memory, applied behavior analysis, social psychology, and animal learning and
behavior). This broadening of research interests and the development of curricula that
meet national standards for licensure (in ABA and Animal Behavior) appears to be
having a dramatic influence on enrollment. While we don’t expect every year to result in
such a dramatic increase in enrollment, it does suggest that we are on the right track in
terms of offering the kinds of programs that are desired by recent college graduates.
Enrollment Targets. There was no explicit program targets for enrollment in 2009 when
the admissions process was underway. Since that time, it has become our goal to admit a
minimum of 1-2 students per faculty member. In 2008-2009, we had 9 regular tenuretrack faculty members on staff so that would place our new targets between 9-18
students. In 2008-2009 academic year, we admitted 6 students. In 2009-2010, our
enrollment jumped to 13 – well within our target goal. It seems that our recent attempts to
2009-2010 Annual Program Assessment 6
MS Experimental Psychology
strengthen the MS Experimental Psychology program through new hires and the addition
of the ABA specialization has been successful.
Next year we will see the loss of one tenure-track faculty member (Dr. Ryan Zayac, ABA
program coordinator), although we hope to search to replace that position in Fall 2010.
Also, Dr. Elizabeth Street will rejoin the MS Experimental Psychology Program in winter
2011 which may help to offset the loss of Dr. Zayac. Her background in applied behavior
analysis will help us through the transition to finding a new program coordinator who is
BCBA certified. Because the ABA program is in flux, we do not anticipate growing the
program immediately. Rather, we anticipate a temporary stabilization in recruitment,
enrollment and retention resulting in a total enrollment in the ABA specialization
somewhere between 10-15 students over the next two years. However, changes to the
ABA faculty and programming should result in another period of growth sometime
around 2012-2013.
Recruitment for 2010-2011
This year, the MS Experimental Psychology program set target goals for recruitment and
enrollment. Our goal was approximately 1-2 students per faculty member. With a
reduction in staffing by 1.5 faculty members (Drs. Anthony Stahelski and Marte
Fallshore), we were left with seven full-time tenure-track faculty members. Thus, our
goal for admissions for 2010-2011 was for between 7-14 students.
In 2009-2010, the responsibility of recruitment and admissions was given to Dr. Kara
Gabriel. She worked collaboratively with the MS Experimental psychology faculty to
match students with individual faculty. We implemented an acceptance policy that
requires at least one full-time faculty member with a shared research interested to accept
responsibility as an applicant’s academic advisor. Also, at least one other faculty
member must agree to take over in the event a change of advisor is needed.
Dr. Ryan Zayac, the Advisor for the Applied Behavior Analysis Specialization attended
the Association of Behavior Analysis International Conference in San Antonio, Texas to
promote the ABA specialization in MS experimental psychology at CWU. As a result f
the combined efforts of Srs. Gabriel and Zayac, 8 new graduate students accepted
positions in the MS Experimental Psychology program (3 in General Experimental and 5
in the ABA specialization). This is the first year that students were turned away in the
ABA program. Enrollment in the ABA specialization is limited by the supervision
requirements. Last year Dr. Zayac was our only fulltime, tenure-track Board Certified
Behavior Analyst (BCAB) faculty member. Some assistance was provided by Season
Almasen, M.A, and a BCBA from Children’s village (one our practicum sites). With only
Dr. Zayac able to conduct supervision, a cap of 5 students was placed on enrollment.
Now that Dr. Zayac is leaving, Ms. Almasen will be teaching courses in the ABA
specialization as an ABD doctoral candidate and adjunct, and Dr. Elizabeth Street is
beginning the process of submitting her paperwork for her BCBA. She has all the
educational and supervised hours qualifications but she must sit for the Board exam. Dr.
Wendy Williams has agreed to complete her ABA supervised hours under Dr. Street in
2009-2010 Annual Program Assessment 7
MS Experimental Psychology
2011-2012. Additionally, it is anticipated that a search for Dr. Zayac’s replacement will
be approved for Fall 2010. It is anticipated that the cap on enrollment in the ABA
specialization will be raised for 2012-2013, after the staffing concerns have been
resolved.
Incoming students. Of the students slated to arrive in September 2010, three are coming
from other universities (2 from out-of-state, plus one international student). and the
remaining 5 are CWU graduates. Five students are joining the ABA specialization and
will be working with Drs. Street and Williams; one student is coming from Ghana to do
social/health psychology research with Dr. Williams. One student will be working with
Dr. Gabriel; another will be working with Dr. Polage. Drs. Greenwald, Lonborg,
Fallshore, and Matheson will not be advising new students in the fall but the admission’s
process for the MS Experimental Psychology (General Experimental specialization)
remains open throughout the year. It is not unusual for us to accept students on an
ongoing basis.
A comparison of enrollment trends between 2007-2008, 2008-2009 and 2009-2010 is
noted in Table 2.
Table 2. Enrollments for 2007-2009 and offers made and accepted for the 20092010 academic year.
Academic Advisor
2007-2008
2008-2009
2009-2010
20102011
K. Gabriel
R. Greenwald
S. Lonborg
M. Matheson
D. Polage
E. Street†
W. Williams†
R. Zayac
1
n/a
0
0
n/a
n/a
0
n/a
0
0
0
1
0
n/a
1
2
2
2 (-1)
2
1 (-1)
1
n/a
2
3
1
0
0
0
1
(5)
6
n/a
M. Jensvold (CHCI)
S. Schepman*
M. Fallshore (.5)
A. Stahelski (.5)
3
n/a
0
1
2
n/a
0
0
0
n/a
0
0
0
n/a
0
retired
*Dr. Schepman works primarily with students in the School Counseling. School
Psychology and Mental Health counseling programs.
† Dr. Williams will share thesis advisor duties with Dr. Street for all of the ABA
specialization students from 2008-2009 to 2010-2011. Her additional students work in
general experimental psychology.
Note: All MS Experimental faculty may also supervise thesis projects for student in the other Psychology
graduate program. Hence, not all faculty members’ thesis advisees are noted in Table 2.
2009-2010 Annual Program Assessment 8
MS Experimental Psychology
Degrees Conferred in 2009-2010
Last year the program director, Dr. Wendy Williams, made serious overtures to Nth year
students (3rd year and beyond) who had lost momentum in their programs, or taken leaves
of absence. Her goal was to clarify each student’s standing and intention, as well as to
attempt to find ways to help them return and complete their programs in a timely fashion.
Ten students were contacted. Of those, five students have completed their degree
programs. Two students completed their coursework and successfully defended their
theses within the 2-year timeframe set as the goal to completion in both specializations.
An additional three students are poised to defend their theses this summer.
Attrition and Retention
Three students (1st-Nth year) resigned from the program this year. Of the students who
did not complete the program, three are poised to defend their theses this summer. The
remaining students four appear to have stagnated and will approached again in the fall by
the program director to see what we can do to facilitate completion of their programs. All
remaining students continue to be actively working toward degree completion.
2009-2010 Annual Program Assessment 9
MS Experimental Psychology
Assessment of Student Learning Overview
In 2007-2008, the MS Experimental (MS EXP) Psychology Graduate Program
Committee met to revise the Student Learning Objectives (SLOs) and SLO Assessments.
At the time, the Psychology Department had charged the MS Experimental Psychology
Program Committee with reviewing and updating the SLOs and assessments in order to
bring them more in line with the assessment procedures of the other graduate programs
within the department. The work on the SLOs and Program goals was met with strong
approval by the CWU Assessment Team.
In 2007-2008, a new tracking system was developed that included individual files, survey
materials, and summary data. At the time, only a preliminary report based on a narrow
application of those tools was available. Last year, the tools were implemented more
comprehensively and thoroughly. This year, an electronic survey system was developed
in an attempt to get greater cooperation from the faculty in terms of completing the
forms. The assessment of student learning for the 2009-2010 academic year is based on
these new, electronic assessment tools. direct comparisons between 2008-2009 and 20092010 will be made whenever possible. However, it is clear that the electronic form
requires minor changes to ensure that the data are gathered in ways that allow such
comparisons.
Review of Record Keeping Procedures
Supplemental Individual Files for ALL current students in the MS Experimental
Psychology Program (going back to 2004-2005) are now created for every graduate
student and maintained in the Director’s office. Much of the data in these files was
gleaned from the departmental records and/or graduate school records, and was then
summarized within each file. (See attached Sample Tracking Form). As each student
progresses through the program, copies of the course of study forms, option approval
forms, IACUC and/or HSRC approvals, updated annual transcripts and other relevant
documents are placed in this file. Major milestones like proposal meetings, thesis defense
meetings, theses submissions and approvals are also noted at the end of the year.
Individual assessment reviews by course faculty, academic advisors and thesis chairs are
maintained here as well. Initial fall assessments are averages across instructors; winter
and spring are then added into that average across the year. By late June we have an
overall assessment for each student that encompasses one entire academic year (quarterly
averages are not currently being computed but could be if an individual case warranted
such fine grain analyses) This year, electronic data collection was implemented in an
attempt to streamline this system, increase user friendliness, and to provide printed
summaries for the students’ files.
Strengths. The student files have centralized important information like names, ID
numbers, course of study, major advisor, contact information. The use of the summary
forms has allowed us to focus on the progress and needs of individual students. We are
able to check off completed courses and note progress in a timely manner by cross-
2009-2010 Annual Program Assessment 10
MS Experimental Psychology
referencing quarterly transcripts with their official course of study forms. This has been
easily implemented because the information is available on-line to the director.
Furthermore, the new assessment data have allowed us to directly address many of the
issues raise by the CWU Assessment committee last year, namely assessments based on
measures other than grades and academic progress. These summary forms also make it
easy to contact students and to maintain a record of conversational notes as a way to
make sure that students are getting what they need in terms of resources, support and
guidance.
Limitations. Tracking student milestones continues to be difficult. In the past, duplicate
copies of the Course of Study forms were sent to Dr. Williams before being filed in the
department office. This has become less than reliable due to budget cuts and reductions in
staffing. Furthermore, proposal meetings, option approval forms, defense meetings, and
other types of information do not have a regular track that funnels through the Director’s
office. As such, tracking relies on self-report by advisors, or via manual review of
departmental files. This system is too slow and requires repeated checks (almost on a
weekly basis by the Director). We failed to develop an acceptable alternative tracking
system with the graduate school that would pass through the Program Director’s office.
Hence this year, the data were gathered by hand and only at the end of the year. Some
faculty did report milestones electronically, others did not. The goal for next year is to
make minor changes to the electronic assessment surveys that will make reporting easier
(e.g. check boxes rather than narrative boxes). We anticipate even less departmental help
as we have lost one office manager, and the secretary’s hours have been reduced.
Number of enrolled students
An overview of students who have yet to complete the MS Experimental Psychology
program is shown in Table 3. It includes data from last year for comparison purposes. As
you can see 9 students completed the program in 2009-2010. At the end of 2009-2010, a
total of 24 students are enrolled in the program (with an addition of 13 students this year).
Eight additional students will join the program in the fall bringing the total to 32.
Table 3. Comparison of the number of graduate students currently enrolled in the
MS Experimental Psychology program (2009-2010) and the data from last year
(2008-2009) by the year accepted into the program (current/last year)
20092010
13
20082009
4/6
20072008
3/5
20062007
0/2
20052006
3/4
20042005
1/3
20032004
0/1
Sixty percent of 2007-2008 first-year students had prerequisites that they need to fulfill
before completing the program. 33% of 2008-2009 first-year students had prerequisites
that they need to fulfill before completing the program. Only 23% of the 2009-2010
students had prerequisites that they need to fulfill before completing the program.
2009-2010 Annual Program Assessment 11
MS Experimental Psychology
Below, in Table 4, are the numbers of students needing each specific prerequisite course.
Beginning in 2010-2011, prerequisite courses changed to Psychology 300 (Research
Methods) and the two statistics courses only. Psychology 301 (Learning) will only be
recommended for ABA students. Psychology 461 (History and Systems in Psychology)
will no longer be required.
Table 4. Number of students entering the program and prerequisite courses needed.
2007-2008
2008-2009
2009-2010
PSY 300
N/A
N/A
0
PSY 301
3
2
0
PSY 362
0
0
0
PSY 363
1
1
2
PSY 461
2
1
N/A
Nth year Students. We began this year with 10 Nth year students. Five have completed
all course work and thesis data collection and have defended their theses. Degrees were
conferred on all of them since the last assessment. Of the remaining 5 students, one is
slated to defend this summer. Three of the four remaining Nth year students have failed to
make adequate progress (coursework and/or thesis work) this year; one has completed his
data collection. The Program Director will be contacting these four students in the fall to
see what we can do to help them complete their programs/theses.
Three students decided to leave the program. One left to join the military (Emmett FL09).
Another left for personal reasons (Fulton, FL09). And one has merely made her
intentions to leave the program known but continues to remain officially active
(Zukowski SP08) while she considers MSOD programs elsewhere.
Program Target: Our program target is to graduate 100% of students who are admitted.
Whenever we loose students to jobs, alternative careers or other programs, the student,
his/her advisor and the director meet to discuss whether the loss is due to a
selection/admission problem, a programmatic problem, or whether it is an isolated
incident. In the case of the three students lost this year, the students, the advisors and
director agreed that the students’ decisions to leave were restricted to the individual
interests of these particular students, and were unlikely to be related to the nature of the
program.
2009-2010 Annual Program Assessment 12
MS Experimental Psychology
Major Milestones
General Academic Assessment Data
Table 5. Number of students meeting departmental and graduate school
benchmarks and percent of program targets met
2007-2008
# of students enrolled
# of Course of Study
Approvals on file √
# of COS forms
approved in 1st qtr
# of Option Approvals
on file†
# of Thesis Defense
meetings*
# of Theses Accepted
by Grad School £
# of students who
presented @
Conferences
20082009
Nth year
students
16
(100%)
16/16
(100%)
n/a
6
(100%)
5/6
(83%)
5/6
(83%)
2/6
(33%)
1/6
(33%)
0/6
(0%)
unknown
15/16
(94%)
6/16
(37.5%)
6/16
(37.5%)
35/35
(100%)
34/35
(97%)
18/19
(95%)
17/22
(77%)
8/22
(36%)
6/22
(27%)
2/13
(15%)
0/6
(0%)
0/16
(0%)
2/22
(27%)
2009-2010
13
(100%)
13/13
(100%)
13/13
(100%)
0/13
(0%)
n/a
Total/Target
† Two 2nd year students will complete their proposal meetings and option approval forms this summer.
IACUC forms for these two students are blanket approvals at CHCI (using archival videotaped data).
£ One 2nd year student has defend her theses and is waiting for the degree to be conferred.
* Another 2nd year student will defend her theses on July 13th.
√Note that academic advisors responsible for those students who have yet to submit their course of study
forms to the graduate school have been notified and asked to help their advisees complete and submit this
paperwork immediately.
2009-2010 Annual Program Assessment 13
MS Experimental Psychology
Course of Study Forms.
By the end of the 2009-2010 academic year, 97% of all students had completed,
submitted and gained approval from the graduate school for their Course of Study forms.
one student has not yet submitted and had her course of study accepted by the graduate
school. This will be corrected before fall quarter.
Program Target: Our program goal is to have course of study forms completed
preferably during the fall quarter of each student’s first year, and no later than the end of
the first year. For the 1st and 2nd year students, all but one student had an approved course
of study by the end of their first quarter in the program. Ninety-seven percent of all
students have approved course of study forms on file.
Option Approval Forms. Of the 22 students who entered the program in 2008-2009 or
earlier, 17 have successfully proposed a thesis and submitted their option approval forms.
Of the five remaining students, four should hold proposal meetings early next fall and
submit the Option Approval forms at that time. One Nth year student had continued to
delay making progress since last year. The Program Director met with him last year and
helped him to revise his course of study. Unfortunately, his thesis advisor has reduced her
affiliation with the department (dropping to 50%). I suspect he is not getting the support
he needs. I will contact him over the summer to see what I can do to help him move
forward.
None of the first year students managed to develop a thesis proposal in time to appoint a
committee and complete the Option Approval form. Ideally, we would like to see all first
year students reach this point in the thesis process by the end of their first year, but it is
not unusual for them to propose in the fall of their second year. This year all of the 1st
year students are expected to achieve that milestone by the end of Fall 2010.
Program Target: Last fall, the MS Experimental psychology faculty agreed to set a
realistic program target for proposing. Based on the current patterns of behavior, setting a
program target for Option Approval Form completion by the end of fall quarter of each
student’s second year seemed realistic and achievable. However, we also agreed that
under ideal circumstances, proposal meetings (and option approval form submissions)
would occur at the end of the 1st year. We agreed to revisit the discussion next fall after
the data for this variable are in. It may be that the faculty will decide to move this target
goal forward to the end of the 1st academic year. If so, we will need to discuss
programmatic changes that will make such a change a realistic and achievable target.
IACUC/HSRC Approvals.
The goal of obtaining IACUC/HSRC approvals for thesis research were not assessed. It is
difficult to reliably obtain these data when the approval letters are issued to the student
directly. Although the goal remains, alternative methods of obtaining reliable, timely and
accurate data on this goal will be explored directly with the IACUC and HSRC directors
next year.
2009-2010 Annual Program Assessment 14
MS Experimental Psychology
Program Target: The program goal is for students to get HSRC/IACUC approvals
within one quarter of their proposal meeting.
Theses accepted/ Degrees conferred.
It is the goal that 100% of thesis submitted to the CWU Graduate School be accepted,
and 100% of degree filings be conferred. This goal has been met. Revisions are common
but relatively minor (formatting, spelling, etc.). We would also like to see the delay
between thesis submission and degree confirmation reduced significantly. At present, the
delay ranges between 3-9 months. This is something that must be addressed by the CWU
Graduate School, as the delay is occurring there. This delay is hurting the school’s
relationship with graduate alumni. Changes have been implemented at the level of the
graduate school but it will take time to determine how effective these change are.
Program Target: The program goal is to have 100% of thesis submissions accepted by
the Graduate School with minor revisions, and for 100% of degree filings to be conferred
within one quarter.
Conference Presentations.
At least two of our graduate students presented original research at regional conferences.
Program Target. The program target for conference presentations is 100% for second
year+ students. Again, this goal may seem high, but efforts to see this number grow are a
program priority. Moreover, this goal is directly tied to thesis data collection. Delays in
data collection can functionally delay a conference presentation by an entire year. This
too will be part of the 2010-2011 MS Experimental Psychology program committee’s
agenda.
2009-2010 Annual Program Assessment 15
MS Experimental Psychology
Time to Completion Data
Currently, the program director has data going back to Fall 2003 and time-to completion
data for ten students. See Table 6. An additional five students are expected finish within
the next 3-6 months (depending on the approve time for the graduate office). The
average overall time to completion is 4.325 years.
Table 6. Time-to-Completion data for students in the MS Experimental
psychology program.
Student/Advisor(s)
Manjarrez
(Eubanks/Street)
Todd (Fallshore)
Buckner (Jensvold)
Tate (Stahelski)
Redfield (Williams)
Sherrill (Matheson)
Stadner (Jensvold)
Jones (Matheson)
Rutledge (Gabriel)
Crnich (Matheson)
pending
Program
60-credit
Gen Exp
60-credit
Gen Exp
60-credit
Gen Exp
60-credit
Gen Exp
60-credit
Gen Exp
60-credit
Gen Exp
60-credit
Gen Exp
60-credit
Gen Exp
60-credit
Gen Exp
45-credit
Gen Exp
Year
enrolled
Fall 2003
Year
Completed
Summer 2008
Time-toCompletion
5.25 years
Fall 2003
Spring 2008
4.75 years
Fall 2003
Fall 2009
6.25 years
Fall 2004
Spring 2009
5.75 years
Fall 2005
Winter 2008
2.50 years
Fall 2005
Spring 2010
5.75 years
Fall 2006
Winter 2010
4.50 years
Fall 2006
Winter 2010
4.50 years
Fall 2007
Summer 2009
2.25 years
Winter
2009
Spring 2010
1.75 years
2009-2010 Annual Program Assessment 16
MS Experimental Psychology
Thesis Defense Meetings/Time to Completion.
This year seven students defended their theses. This represents only 36% of students
those that should be nearing the completion of their programs- a marked improvement
over last year (13%). Six of these students were Nth year students; the other completed
the program in four quarters plus a summer (a noteworthy accomplishment). We expect
another second-year student to defend on July 13, 2010.
Program Target. Our program target is for all students to defend their thesis at the end of
their second year. As you can see from the table, this is a goal that needs work on our
part. Many students extend their enrollment into a third year and some even well beyond
that. Because the credit load for the MS Experimental Psychology (General Experimental
specialization) degree was reduced from 60 to 48 credits in 2007-2008, we believe that a
2-year target for degree completion should be possible for most students. While there
may be exceptions due to thesis data collection restraints, the two-year target goal is
worthwhile for these students. The program committee will address this issue in the Fall
2010 in an attempt to identify way to assist students achieve this goal.
Students enrolled in the ABA program have a much higher course load requirement (68+
credits). Further, the combined goals of additional hours, extensive
internships hours and thesis completion creates a difficult situation for these students in
terms of time to completion. None of the ABA specialization students have finished their
program so no data on time-to completion are available. Both of our 3rd year students in
the program have yet to propose. The MS Experimental Psychology faculty will meet in
fall 2010 to discuss this goal. It is likely that two separate goals are needed for these two
programs.
2009-2010 Annual Program Assessment 17
MS Experimental Psychology
Programmatic Course Offerings
MS Experimental Psychology students are expected to maintain an overall GPA of 3.0,
and receive no grade lower than a C in required core content courses listed on their
Course of Study. In 2009-2010, eleven graduate courses in the MS Experimental
Psychology program were offered. Nearly all students (95%) passed their classes and the
overall average GPA in each of the course ranged from 3.67 to 4.0. Table 7 outlines the
number of students in each class, the number and percent that passed the class, the
average GPA for the class, and a breakdown by year in the program. Note that a course
offering may reflect more than one section offered across multiple quarters by various
instructors. Also, what appears to be courses with low enrollments are courses that are
offered to students in other programs as well.
Table 7. Content course offerings, percent of students passing and average GPA
Core
Course
Psy 504
Psy 525
Psy 541
Psy 542
Psy 544
Psy 550*
Psy 551
Psy 553
Psy 554
Psy 555
Psy 558*
Psy 562
Psy 565
Psy 576
Psy 578
Psy 580
Psy 684
Course
Offered?
Y
N
Y
Y
Y
Y
Y
Y
N
Y
Y
N
N
N
Y
Y
Y
# of MS
Exp
Students
11
# of
Exp
students
who
Passed
11
Percent
Passed
100%
Average
GPA
N/A
1st year
students
passing
11
2nd year
students
passing
0
Nth year
students
passing
0
2
2
1
5
12
7
2
1
1
4
12
7
100%
50%
100%
80%
100%
100%
3.85
2.0
4.0
2.54
3.73
4.00
2
1
1
4
12
7
0
0
0
1
0
0
0
1
0
0
0
0
8
8
100%
4
8
0
0
13
12
3
12
11
3
92%
92%
3.42
3.56
N/A
12
11
0
1
1
3
0
0
0
100%
*Spring 2010 sections of these courses wee taught but the grade rosters were unavailable at the time the
report was due.
Program Targets: All but two graduate students enrolled in MS Experimental
Psychology courses in 2009-2010 have maintained GPAs higher than 3.0, nor did they
have any grades lower than a C. The two students with failing grades are the ones who
chose to leave the program. They both maintained a 3.0 and passed their courses until the
quarter they left. Despite advising to the contrary, neither student officially withdrew
from courses, so they failed.
2009-2010 Annual Program Assessment 18
MS Experimental Psychology
Quarterly Survey of Student Progress
(Fall 2009, Winter 2010, Spring 2010)
As part of the our annual program assessment, the MS Experimental Program gathers
quantitative and qualitative data on 1st and 2nd year students related to the general skills,
knowledge, behaviors and dispositions that characterize professionals in experimental
psychology. In addition to the GPA results and the benchmark achievements noted in the
Student Learning Objectives (SLOs – see attached), faculty are surveyed and asked to
assess student qualitatively on a quarterly basis. Most of the qualitative assessments
provide insight into the requisite skills that contribute to grades. However, several
qualitative assessments address issues that contribute only indirectly to grades (if at all)
such as expressive skills, appreciation for empirical evidence, potential for success within
the field, and attitudes and dispositions. It is important to track each student’s progress in
these areas to ensure that our program provides appropriate and timely feedback as part
of their training, and makes data-driven improvements designed to address particular
needs. This year, the survey was administered on-line to help facilitate responding, as
well as data analyses. Faculty members in the MS Experimental Psychology program
who have had experimental psychology graduate students in their graduate classes during
the previous quarter are asked to complete a survey that rates students on a variety of
measures. Faculty were asked to complete the survey for each student in each class they
taught this year. Eighty-one surveys were returned.
Academic Performance Summary. Faculty were asked to assess overall academic
performance (which includes assignment completion, class participation, work effort,
organization, problem solving, critical thinking skills, data-based conceptualization and
analyses, and overall performance). Table 8 outlines the results of the ACADEMIC
PERFORMANCE summary item.
Table 8. Academic Performance Summary Table
RANKING
5. Superior performance
4. Good performance
3. Adequate performance
2. Less that adequate performance
1. Unsatisfactory performance
0. Insufficient knowledge to comment or
question is irrelevant to the course
Responses
33
26
15
3
2
2
Percent
41.8%
32.9%
19.0%
3.8%
2.5%
0%
Nearly ninety-eight percent (93.7%) of the rankings in Table 8 suggest adequate or better
progress in academic performance. Nearly 75% of all rankings met the program’s
criterion goal of a 4 (good performance) or better. Only 6.3% of rankings were less than
adequate or worse. These low rankings were given to 5 of the first-year students. Two of
them have left the program. The remaining three had apparent difficulty in only one class.
All three individuals were assessed by other instructors and ranked as making adequate
progress academically.
2009-2010 Annual Program Assessment 19
MS Experimental Psychology
Cumulative GPA. Table 9 shows the Cumulative GPA rankings for both 1st and 2nd year
students. Students are required to maintain a 3.0 GPA throughout their graduate careers.
Note that 85% of the first year students were able to maintain the 3.0 minimum-GPA by
the end of the first year; 100% of the second year students had GPAs over the 3.0minimum at the end of two years. Note that the two first-year students (15%) with GPA
scores lower that 3.0 (italicized) are the two students who chose to leave the program.
However, it is important to note that both students maintained an adequate GPA until the
quarter they decided to leave. Neither of them completed their courses during their final
quarter, which negatively affected their overall GPA. The average GPA for the remaining
first year students is 3.86.
Table 9. Cumulative GPA Scores for First and Second Year Students
Cumulative GPA
First Year Students
2.46
1.90
3.41
3.96
3.56
3.91
4.00
3.93
3.76
3.95
4.00
4.00
4.00
Cumulative GPA
Second Year Students
3.94
3.86
3.98
3.80
4.00
3.65
Mean = 3.60
Mean = 3.87
2009-2010 Annual Program Assessment 20
MS Experimental Psychology
Itemized Assessments Overview
MS Experimental Psychology program faculty members were asked to rate students in all
of their classes for the year on 10 individual performance measures related to the SLO
and suggestions from the University Assessment Team, including:
A.
B.
C.
D.
E.
F.
G.
H.
I.
J.
Content knowledge mastery
Skill mastery
Quantitative analytical skills
Logic-based analytical skills
Comprehension skills
Written expressive skills
Verbal expressive skills
Appreciation for empirical evidence
The instructor’s view of the student’s ability to succeed in the program
The instructor’s opinion of whether the student display’s an attitude or
disposition that is consistent with the field of experimental psychology
Each skill is rated on a scale from 1 to 5 (1 being the lowest and 5 being the highest).
Once the surveys are returned, they are compiled, and individual forms are placed in the
student’s file. These ten measures are tied to our outcomes goals, and we would like our
students to maintain a score of 4.0 as an overall average score for each item.
Table 10 shows a summary of those data from Fall 2009-Spring 2010 for the 10
qualitative items listed above (A-J). Twenty-three students were included in the analysis
and included first, second, and Nth-year students. The number of students at each ranking
is listed, along with a calculated percentage of the total responses. Scores reflect
assessments by class instructors in Psychology courses relevant to the students’ courses
of study. The data in Table 10 represent all the responses submitted for all students.
There were a total of 81 responses submitted across 6 faculty members. Rankings were
provided by MS Experimental Psychology program faculty only. Courses offered outside
the Psychology Department were not surveyed. As such, individual students are assessed
(and represented) more than once as a result of being in multiple classes across 3
quarters.
2009-2010 Annual Program Assessment 21
MS Experimental Psychology
Table 10. Number of students (and percent) ranked at each level on the 10 itemized
assessments noted above. “Unable to Observe or Irrelevant to the Class (0)” is the
alternative option. Faculty could also skip questions. Assessments scoring 0 or SKIP were
deducted from the total assessments for calculation purposes.
Rank
5
4
3
2
1
0
Skip
%@
Target
A
B
C
D
32
(40%)
26
(33%)
18
(23%)
2
(2%)
2
(2%)
1
(0%)
0
(0%)
73%
33
(41%)
25
(31%)
18
(23%)
2
(2%)
2
(2%)
1
(0%)
0
(0%)
73%
7
(31%)
14
(64%)
1
(4%)
0
(0%)
0
(0%)
59
(0%)
0
(0%)
95%
36
(52%)
22
(32%)
8
(12%)
1
(1%)
2
(3%)
12
(0%)
0
(0%)
84%
Assessment Items
E
F
G
39
(50%)
23
(29%)
13
(16%)
1
(1%)
2
(3%)
3
(0%)
0
(0%)
79%
28
(42%)
25
(37%)
11
(16%)
1
(1%)
2
(3%)
13
(0%)
1
(0%)
79%
25
(33%)
35
(46%)
11
(14%)
4
(5%)
1
(1%)
5
(0%)
0
(0%)
79%
H
I
J
33
(54%)
25
(40%)
3
(5%)
1
(1%)
0
(0%)
19
(0%)
0
(0%)
93%
35
(44%)
31
(38%)
9
(11%)
3
(5%)
2
(2%)
1
(0%)
0
(0%)
83%
7
(8%)
52
(64%)
17
(22%)
4
(5%)
1
(1%)
0
(0%)
72%
Program Target. Our program goal is for all (100%) students to maintain ratings at or
above 4.0 (good performance) on each measure. Our obtained measures range between
73% - 96%. Four of the ten assessments show that 80% or more of the assessments
ranked our students as having good or superior performance. Three of the remaining six
items all had 79% of all assessment rankings as good performance or better. The
remaining three items had over 70% of assessments falling in the target range (4.0 or
higher).Only a very few individual assessments (5% or less in any cell) were ranked at
the less than adequate or unsatisfactory levels.
It is possible that our target goal is overly ambitious since it requires that all students
perform more than adequately on all measures. (Note that adding in rank level 3
(adequate performance) would substantially raise the percentages to over 80% for all
assessments. Students enroll in graduate school to learn things they do not already know.
And not all students have completed the relevant course by the end of their first year. The
assessments presented above are not broken down by year(s) in the program. (Also note
that the two students who left the program were included in the data set. Their
assessments represent several of the low ranking in each case.) Although our target is set
very high, it is currently the view of the graduate faculty within the program that such
high goals provide motivation to improve program delivery. Failure to meet that goal is
not seen as failure. Rather it provides a rubric for assessing how we are doing at delivery
the best possible program.
Last year’s indicators suggested that 2nd year students outperformed the 1st year students.
As such, it is unlikely that this year’s grouped data will even allow us to reach our goal of
2009-2010 Annual Program Assessment 22
MS Experimental Psychology
100% at or above 4.0 (good performance) on each measure. Currently the data analysis
system does not gather information that would allow such an analysis. However, an item
(for office use only) can be added next year so that the data can be sorted by year in the
program. Unfortunately, it cannot be added to completed surveys. But the correction can
be made over the summer and future assessments will include those analyses. In the
meantime, our primary targets for improving scores will continue to be 1st year students.
Furthermore, I will be identifying the individual students who were assessed below level
3 (adequate performance) to explore ways to improve those skills.
Comparison of Student Averages. Last year, the data were analyzed at the individual
level and average scores per student were presented. This year the amount of data was
tremendous (81 assessments x 10 items x 22 students) with a total of 193, 820 individual
data points for the itemized assessment alone. Due to the short timeline, averaged data are
not presented here. But it is the goal of the program director to analyze those data over
the summer so that comparisons can be made with last year’s data.
Itemized Assessment Review
Content knowledge (Item A)
This item is linked directly to Student Learning Objective 5: Describe and contrast major
theoretical and practical concepts in core content areas (see attached SLOs).
Seventy-three percent of assessments of content mastery were rated at or above the target
level of 4.0. This suggests that instructors of courses relevant to our students’ courses of
study felt that the students are making good to superior progress on the skills addressed in
those classes. Furthermore, 40% of the assessments received perfect scores (5.0) while
another 33% earned good performance scores (4.0). Twenty-three percent of the
assessments earned scores of 3.0. These data suggest that our most of our students
demonstrate reasonable mastery of the content knowledge in their classes; still there is
room for improvement. Faculty may wish to consider alternative or supplemental
teaching methods that encourage content mastery.
Skills mastery (Item B)
This item is linked directly to Student Learning Objectives 2: Describe and perform data
analyses, 3: Describe, assess and utilize research designs in psychology, 4: Design,
conduct and report psychological research findings, and 7: Perform professional skills in
a supervised practice setting (see attached SLOs).
Seventy-three percent of assessments of skills were rated at or above the target level of
4.0. This suggests that instructors of courses relevant to our students’ courses of study felt
that the students are making good to superior progress on the skills required in those
classes. Furthermore, 41% of the assessments received perfect scores (5.0) while another
31% earned good performance scores (4.0). Twenty-three percent of the assessments
earned scores of 3.0. These data suggest that our most of our students demonstrate
2009-2010 Annual Program Assessment 23
MS Experimental Psychology
reasonable mastery of the skills required in their classes; still there is room for
improvement. Faculty may wish to consider additional practice to encourage skill
mastery.
Quantitative analytical skills (Item C).
This item is linked directly to Student Learning Objectives 2: Describe and perform data
analyses, 3: Describe, assess and utilize research designs in psychology, and 4: Design,
conduct and report psychological research findings (see attached SLOs).
This item had more faculty members opting not to assess student performance than any
other item. Of the 81 assessments return, 59 responded that they were unable to assess or
the question was not relevant to the course they were reviewing. This implies that the
faculty members are not simply marking a single score for all items for every student. It
shows that they are attending to the question in a meaningful way and are only reporting
assessments when they are relevant. Quantitative analyses are not conducted in every
class in the program. However, many classes do deal indirectly with data and as such a
conversation with the faculty is warranted regarding whether this item could be assessed
in ways other than through direct data analyses. As for the assessments that were
reported, 95% of assessments met the level 4.0 ranking target. Therefore, when the
students are actively engaged with data analyses, they are showing excellent proficiency.
(Note that by adding in the single level 3 ranking, the percentage jumps to 100%).
Logic-based analytical skills (Item D).
This item is linked directly to Student Learning Objectives 2: Describe and perform data
analyses, 3: Describe, assess and utilize research designs in psychology, 4: Design,
conduct and report psychological research findings, and 5: Describe and contrast major
theoretical and practical concepts in core content areas Perform professional skills in a
supervised practice setting (see attached SLOs).
Eighty-four percent (84%) of the assessments achieved the level 4.0 minimum target
goal. An additional 12% reached the 3.0 level. Again, we see nearly 100% of assessments
(minus 3 single rankings) suggesting that our students are capable of doing adequate or
better work. For those students scoring below the 3.0 rank, brief conversations with either
the instructional faculty or with the students’ advisors may shed light on the difficulty
and provide ideas for remediation.
Comprehension (Item E)
This item is linked directly to Student Learning Objectives 3: Describe, assess and utilize
research designs in psychology, 4: Design, conduct and report psychological research
findings, and 5: Describe and contrast major theoretical and practical concepts in core
content areas of research (see attached SLOs).
2009-2010 Annual Program Assessment 24
MS Experimental Psychology
Most graduate courses require a great deal of reading of professional literature. Hence,
there is a need for advanced comprehension skills. Here too, our students did well.
Seventy-nine percent of assessments earned ratings at or above our target, with 50%
earning perfect ratings of 5.0. Another 29% scored in the 4.0 range. A total of 92% of all
assessments fell in the adequate or better range (3.0 or higher). Again, here is an item that
should show a marked difference between first year students and more experienced
students. Next year’s, breakdown by year of admission to the program will shed more
light on this prediction.
Written Expressive Skills (Item F)
This item is linked directly to Student Learning Objectives 1: Write in the language of the
discipline, using the elements of style described in the APA Publication Manual. 3:
Describe, assess and utilize research designs in psychology, 4: Design, conduct and
report psychological research findings, and 6: Engage in scholarly and professional
activities including presenting research at formal and/or informal professional events
(see attached SLOs).
Experimental psychologists, whether in the field of basic research or applied science,
need to be articulate – particularly in writing. Written language using the APA writing
format is the primary mechanism of dissemination of research findings in experimenal
psychology. Seventy-nine percent of the student assessment items met the program goal
of 4.0 or higher. These data suggest that written expression is being mastered by most of
the graduate students across many settings. If we consider rankings of 3.0 or higher
(adequate performance) the number of assessments jumps to 95%. These data suggest
that most of our students could benefit greatly from writing practice and feedback across
many courses. Such practice and feedback can come in various forms. This assessment
item warrants discussion among the faculty to identify creative instructional and thesisbased opportunities for students to write and receive meaningful feedback. Some courses
in our program might benefit from additional learner objectives that target writing.
Furthermore, students scoring below the 3.0-level will be approached to discuss
additional resources for assistance with writing (e.g. CWU writing center, Psych 300).
Verbal Expressive Skills (Item G)
This item is linked directly to Student Learning Objectives 2: Describe and perform data
analyses for particular data sets; 3: Describe, assess and utilize research designs in
psychology, 4: Design, conduct and report psychological research findings, 5: Describe
and contrast major theoretical and practical concepts in core content areas of research
and 6: Engage in scholarly and professional activities including presenting research at
formal and/or informal professional events and 7: Perform professional skills in a
supervised practice setting (see attached SLOs).
Whether teaching a college class, presenting research findings as a conference, or
interacting with the parents of a child with autism, verbal skills are key to professional
2009-2010 Annual Program Assessment 25
MS Experimental Psychology
success. Seventy-nine percent of the assessments met the target goal of 4.0 or higher
(with 33% scoring in the superior range). That number jumps to 90% if adequate
performance is considered as well. It is the goal of the MS Experimental psychology
faculty to give our students ample opportunities to speak publically in both formal and
informal settings. Some courses in our program might benefit from additional learner
objectives that target and public speaking. the lowest scoring individual may benefit from
taking a communications/speech class. Such suggestions could come from instructional
faculty or from academic/thesis advisors.
Appreciation of empirical evidence (Item H)
This item is linked directly to Student Learning Objectives 2: Describe and perform data
analyses for particular data sets; 4: Design, conduct and report psychological research
findings, and 5: Describe and contrast major theoretical and practical concepts in core
content areas of research (see attached SLOs).
It is very important to note that the program faculty instructors view our students as
placing high value on empirical evidence and data. This is particularly noteworthy given
the diversity of interests within the program. Not all of our students will pursue careers in
fields dominated by statistical inference. Student in applied behavior analysis often rely
on single-subject designs, case studies or pure description. Ninety-three percent of the
assessments related to the appreciation of empirical evidence met the program criterion of
4.0 (good appreciation) or higher. These students have clearly developed their analytical
skills and an appreciation for data-driven approaches that are appropriate to their own or
to other sub-disciplines. The inclusion of adequate appreciation (3.0) raises the total to
98%.
Ability to succeed in the experimental psychology program (Item I)
This item is indirectly linked to all of the Student Learning Objectives (1-7) 1: Write in
the language of the discipline, using the elements of style described in the APA
Publication Manual; 2: Describe and perform data analyses for particular data sets; 3:
Describe, assess and utilize research designs in psychology ; 4: Design, conduct and
report psychological research findings, 5: Describe and contrast major theoretical and
practical concepts in core content areas of research; 6: Engage in scholarly and
professional activities including presenting research at formal and/or informal
professional events; and 7: Perform professional skills in a supervised practice setting
(see attached SLOs).
Predicting academic success is a bit like predicting the weather. We take great pride and
great credit in our successes, and we blame our failures on powers beyond our control.
Yet it is something that all of us are asked to do regularly as professionals when we make
guesses about undergraduate and graduate school admissions. Although it is difficult to
quantify, it is something that experts within the field recognize in others. As such, we
decided to ask the MS Experimental Psychology program faculty directly about whether
2009-2010 Annual Program Assessment 26
MS Experimental Psychology
they thought the graduate students were likely to complete the program. Eight-four
percent of the assessments were consistent with the program goal of 4.0 (good likelihood
or better). Recall that one assessment predicted one student was unlikely to succeed in
the program and that very student dropped out. The other student who dropped out was
repeatedly assessed as having between adequate to superior chances of completing the
program. Oh well, looks like rain or shine – you choose. In terms of testing the accuracy
and validity of such assessments, we will simply have to wait to see who completes the
program and who does not.
Disposition consistent with the field of experimental psychology (Item J)
This item is directly linked to all of the Student Learning Objectives (1-7) 1: Write in the
language of the discipline, using the elements of style described in the APA Publication
Manual; 2: Describe and perform data analyses for particular data sets; 3: Describe,
assess and utilize research designs in psychology ; 4: Design, conduct and report
psychological research findings, 5: Describe and contrast major theoretical and
practical concepts in core content areas of research; 6: Engage in scholarly and
professional activities including presenting research at formal and/or informal
professional events; and 7: Perform professional skills in a supervised practice setting
(see attached SLOs).
One of the most difficult attributes to measure is a person’s disposition toward their
chosen field of study. To do so, we must ask ourselves whether the person behaves and
interacts in a manner consistent with the values, opinions and approaches of the field. We
asked the faculty directly about whether they thought the students seemed to be
developing a sense of identity with the field of experimental psychology. The responses
were very consistent with our other measures (probably because items A-H are necessary
for such dispositions to develop). Ninety-two percent of the assessments suggested that
the program faculty believe that most of our students were adequately (3.0 or higher)
demonstrated the dispositions consistent with the field. Seventy-two percent of the
assessments met the program criterion of 4.0 or higher. Again it is important to note that
2 students left the program because they, themselves, felt that experimental psychology
was a poor fit for them. Of the assessments that met the criterion, only 8% received the
5.0 ranking (similar to a professional in the field). The remaining 64% of were described
as holding a attitude/disposition that is very consistent (4.0) with the field. It would
appear that there is substantial room for improvement. Again, this item warrants
discussion among the program faculty to identify opportunities to model/encourage
activities and behaviors that typify professionals in the field of experimental psychology.
2009-2010 Annual Program Assessment 27
MS Experimental Psychology
Assessing the Quarterly Survey
The quarterly survey of student progress has addressed several of the concerns raised in
last year’s final report. Specifically, it allows us to more directly assess the link between
student behavior, learning objectives and program goals. Until 2007-2008, we had relied
exclusively on grades and program milestones (such as time to thesis completion, HSRC
and IACUUC approvals, and grades). This quarterly assessment allows us to capture a
more dynamic view of the development of these important skill sets. Moreover, progress
can be tracked on an individual basis, and intervention by advisors or the director can
target individual student needs. Last year’s survey did, however, have some limitations.
Previous Limitations.
1) Return rates are not 100%.
2) Perspective is based solely on instructor point of view; the student’s point of
view of how the program is working for them is missing.
3) The current system is unwieldly and relies too much on xeroxed handouts that
can be misplaced or lost.
4) Integration with academic data (grades, progress) is not yet possible due to the
paper-based nature of all the data. There is no single database where all the
information can be entered.
Results of solutions implemented.
1) An electronic version of the assessment was created and implemented. As a
result, return rates were much higher. More faculty members participated,
even though the number of students being evaluated increased dramatically.
Hundreds of thousands of data points were gathered. There is now a rich
source of information from which to draw when considering improvements to
the program.
2) An electronic assessment of the MS Experimental Psychology program was
developed for students to complete. Fourteen students completed the survey.
The survey was greeted with enthusiasm.
3) The problem with reliance on Xerox copies has not been solved. A formal
attempt was made to modify the course of study form and to then redirect the
paper path from the graduate school through the program director’s office,
which was rejected by the graduate school. An attempt was made to make
arrangements for the secretarial staff to provide copies but it was
inconsistently completed.
4) Integration of assessment results with academic data (grades, progress)
continues to be a problem. However, the difficulty is now an electronic one.
There is still no single database where all the information can be entered. No
formal efforts were made to resolve this problem this year because the focus
was on developing electronic assessments.
2009-2010 Annual Program Assessment 28
MS Experimental Psychology
Limitations to the new 2009-2010 Assessment
1) The newly adopted timeframe for assessment activites creates barriers to a
thorough analyses of the data and completion of the assessment.
Accommodations must be found that will allow the process to begin earlier. If
three major assessments are to be completed (Quarterly, Annual and Student),
time must be available to compile and interpret the results. Consultation with
Dr. Pellett is recommended.
2) Data analyses need to be refined. This will surely result in changes to the
survey instrument. Some changes may preclude comparisons across academic
years. This problem should begin to diminish as the survey instruments are
perfected.
3) A solution to the paper trail issue is paramount. At present, we have SLOs that
require the tracking of IACUC/HSRC forms, proposal and defense meetings,
etc. for which there is no documentation. If these non-documented activities
continue to be part of the SLOs, a method for documenting them is essential.
4) Clarification is needed to really understand the CWU Assessment Teams
request for a clearer picture regarding how outcomes, methods and results fit
together. Example reports will be requested.
5) Requests for assessment data to be posted on the website still needs to be
discussed among the faculty. Which data? Directed at whom? The current
assessment data may not reflect changes that are put in place for the
subsequent year and hence, may be misleading. Solutions need to be found
that will allow us to make available data that are relevant, valid and useful.
2009-2010 Annual Program Assessment 29
MS Experimental Psychology
MS Experimental Psychology Program Assessment (Student Version)
This year, the MS Experimental Psychology students were invited to take part in an online survey/assessment of their graduate program (See attached copy of the blank survey).
Fourteen students completed the survey. The following are the results from the five
questions designed to address demographics, academic advising, thesis advising, and
satisfaction with the program.
Table 11: Demographics: In which program are you enrolled?
Program
MS Exp Psych (General)
MS Exp Psych (ABA)
Response Count
7
6
Response Percent
53.8%
46.2%
Table 12: Demographics: In what year did you start your graduate program?
Year
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
Response Count
0
1
0
0
5
8
Response Percent
0.0%
7.1%
0.0%
0.0%
35.7%
57.1%
Table 13: How would you rate the academic advising (re: course, adjustment to the
program, pre-thesis advising) that you have received?
Rating
Excellent
Good
Fair
Poor
Response Count
5
4
4
1
Response Percent
35.7%
28.6%
28.6%
7.1%
Table 14: How would you rate the thesis advising you have received?
Rating
Excellent
Good
Fair
Poor
No Thesis Advisor yet
Response Count
5
5
1
2
1
Response Percent
35.7%
35.7%
7.1%
14.3%
7.1%
2009-2010 Annual Program Assessment 30
MS Experimental Psychology
Table 15: How satisfied are you with the training in the MS Experimental psychology
Program thus far?
Rating
Very Satisfied
Satisfied
Unsatisfied
Very Unsatisfied
Response Count
2
4
6
2
Response Percent
14.3%
28.6%
42.9%
14.3%
Breakdown by Program
Students in the two programs (General Experimental Psychology and Applied Behavior
Analysis) responded differently to the 3 substantive questions. Generally speaking, the
view of academic advising is very mixed for both groups. This issue warrants review by
the Program faculty in terms of seeking methods to standardize academic advising
somewhat.
Table 16. Responses x Program: Academic Advising
Ranking
Excellent
Good
Fair
Poor
MS Exp Psy (Gen)
42.9% (3)
28.6% (2)
14.3% (1)
14.3% (1)
MS Exp Psy (ABA)
33.3% (2)
33.3% (2)
33.3% (2)
0.00% (0)
Thesis advising is less consistent across programs. The MS Exp Psy (Gen) program
appears to be more satisfied with the thesis advising in general (although one student was
very unhappy). It would appear that who the thesis advisor may be having a direct on this
item. The MS Exp Psy (ABA) students are more satisfied in general but the range is still
broad. More ABA students gave thesis advising a rating of “good” while the largest
number of GEN students rated their thesis advising as “excellent”.
Table 17. Responses x Program: Thesis Advising
Ranking
Excellent
Good
Fair
Poor
MS Exp Psy (Gen)
42.9% (3)
14.3% (1)
0.00% (0)
28.6% (2)
MS Exp Psy (ABA)
33.3% (2)
50.0% (4)
16.7% (1)
0.00% (0)
Training Satisfaction is also spread broad across both programs. More MS Exp Psy
(ABA) students have negative things to say about he program n general when compared
with the MS Exp Psy (GEN) students.
2009-2010 Annual Program Assessment 31
MS Experimental Psychology
Table 18. Responses x Program: Training Satisfaction
Ranking
Excellent
Good
Fair
Poor
MS Exp Psy (Gen)
14.3% (1)
28.6% (2)
57.1% (4)
0.00% (0)
MS Exp Psy (ABA)
16.7% (1)
16.7% (1)
33.3% (2)
33.3% (2)
Breakdown by Year
Responses also varied as a function of time spent in the program. Second year and Nth
year students appear to be fairly satisfied with the academic advising. First year students
overwhelmingly rate the academic advising as fair to good.
Table 19. Responses x Year: Academic Advising
Ranking
Excellent
Good
Fair
Poor
2005-2006
100.0% (1)
0.00% (0)
0.00% (0)
0.00% (0)
2008-2009
60.0% (3)
20.0% (1)
0.00% (0)
20.0% (1)
2009-2010
12.5% (1)
37.5% (3)
50.0% (4)
0.00% (0)
Second year and Nth year students appear to be fairly satisfied with the thesis advising.
However, one of them still does not have a thesis advisor. This is concerning. Six of the
eight first year students rate the academic advising as good to excellent.
Table 20. Responses x Year: Thesis Advising
Ranking
Excellent
Good
Fair
Poor
No thesis Advisor Yet
2005-2006
100.0% (1)
0.00% (0)
0.00% (0)
0.00% (0)
0.00% (0)
2008-2009
40.0% (2)
20.0% (1)
0.00% (0)
20.0% (1)
20.0% (1)
2009-2010
25.0% (2)
50.0% (4)
12.5% (1)
12.5% (1)
0.00% (0)
Students from every year appear to have mixed feelings about the program. But first and
second year students seem less satisfied. This may reflect the new and somewhat rocky
history with the ABA program. The program will loose its director next year. This has
creates angst among the first-year students and the data reflect this. The data from the
second year students is less clear. Two of these students are in the ABA program so that
could account for some of the low ratings. But overall, these data suggest that we need to
discuss and reflect on how we can help our students to be more satisfied. We may also
want to gather more detailed data next year to see what aspects of the program are
causing students to feel less than satisfied.
2009-2010 Annual Program Assessment 32
MS Experimental Psychology
Table 21. Responses x Year: Training Satisfaction
Ranking
Excellent
Good
Fair
Poor
2005-2006
0.00% (0)
100.0% (1)
0.00% (0)
0.00% (0)
2008-2009
40.0% (2)
0.00% (0)
60.0% (3)
0.00% (0)
2009-2010
0.00% (0)
37.5% (3)
37.5% (3)
25.0% (2)
It is important to note that the number of respondents is quite small because there are few
graduate students in the MS Experimental Psychology program overall. However, the
data do seem to suggest that discussions among the program faculty regarding academic
advising, thesis advising and training satisfaction are warranted.
Open-ended comments
Students were also asked to respond to three open-ended questions. The primary purpose
of these questions was to identify information that could be added to the Graduate MS
Experimental Psychology Handbook. The three questions were:
1. What do new students need to know about the MS Experimental Psychology
program? What advice would you give them?
2. What are the best aspects of the MS Experimental Psychology program?
3. What recommendations do you have that might help us make the MS
Experimental Psychology program better?
The students provided candid, honest and thoughtful responses. (See attached Response
Summary).
Recommendations.
1. Address these three issues with the program faculty in the fall with the goal of
standardizing our advising through the development of uniform advising materials.
2. Consider adding clarifying questions to the survey in an attempt to identify relevant
components of advising (e.g. frequency of meetings with advisor, level of feedback
provided, etc.)
3. Have the program director meet the incoming graduate student at their first meeting to
clarify the role of advisors, and the flexibility available to students to change advisors.
4. Also advise students that they can come to the Program Director for advice regarding
advising, or with concerns about the program.
2009-2010 Annual Program Assessment 33
MS Experimental Psychology
Concerns raised in Previous Assessments (2007-2008; 2008-2009)
The 2008-2009 CWU Assessment Team provided several points for consideration.
What student outcomes are being assessed this year and why? How were they assessed?
What methods were used? Who assessed? When was it assessed?
Much of this question has been answered above. Changes have been put in place that
involve quarterly and annual assessments. The assessment documents were translated
into an on-line electronic data gathering system. Additional modifications have
become apparent and will be implemented next year (e.g. identifiers for year in the
on-line program).
In reality this question more accurately addresses what isn’t included in the
assessment. For example, some course grades were unavailable. Changes to the
deadlines for assessment data have made the development of a complete picture
difficult. Rather than exclusively relying on grades, individual assessments that are
tied to the SLOs are now being made by academic advisors, thesis advisors, and
course instructors. But it takes time to complete these assessments. The assessments
for academic/thesis advisors was not completed in time to include in this year’s
report. Next year, the Annual thesis advisor survey will be administered mid-quarter
to avoid overloading the faculty at the end of Spring quarter.
Finally, a new program assessment survey for graduate students was developed and
implemented this year in the interest of gaining the students’ perspective. It was
greeted enthusiastically by the students in the program.
Is there a way to assess students at the beginning and ending of their program to
determine what was learned?
Due to the nature of Experimental Psychology, students arrive with a variety of
different degrees and skills. The distinction between basic and applied research is
large and will directly impact students’ training. Some students even arrive without
an undergraduate degree in psychology. One student may plan to be trained to work
for the US Census Bureau and need strong statistical skills, while a future applied
behavior analyst needs minimal statistical and stronger single-subject design skills.
As such, attempting to derive a standard set of skills for students in this broad area of
graduate training would not necessarily reflect the diversity of the program. Nor
would all students meet a given standard even at the end of the program. There are
not national standards or norms for experimental psychology. There is no licensing
entity. The Director plans to discuss this idea with the chair of the undergraduate
assessment committee within psychology.
2009-2010 Annual Program Assessment 34
MS Experimental Psychology
What general improvements in the program were implemented this year?
1. ABA specialization committee met to discuss the possibility of having the students
complete their coursework in the first year, leaving the second year for internships.
This is the model used in the mental health counseling program, the school
counseling program and the school psychology program. It has been successfully
employed in those programs and seemed attractive or the ABA progam. with
coursework out of the way, students would be free to complete internships are a wide
set of agency sites – some that have BCBA professionals on staff that could supervise
some of their hours. After careful review of the course requirements, credit-loads,
course programming and faculty workloads, we were able to redesign the program
such that the incoming class (2010-2011) will be able to complete their coursework
by the end of their first academic year.
2. All MS Experimental Psychology students now agree to a background check as a way
of protecting vulnerable populations with regard to internships and thesis work.
3. The Assessment process has begun to move to electronic formats that should facilitate
data collection and analyses. Also, faculty assessment surveys are being returned in
higher numbers as a result of the new format.
4. A year-long colloquium class was developed to assist students with aspects of
experimental psychology that are not traditionally part of any currently taught
courses, such as grant writing, conference presentation preparation, internship
searches, and professional discourse about one’s own research.
5. All faculty are more involved in recruitment and admissions as a result of the new
policies put into place (2 faculty members committed seeing every student through
the process).
6. A new program assessment survey for graduate students was developed and
implemented this year in the interest of gaining the students’ perspective.
Are there discipline specific rubrics or standards that could be used to evaluate student
progress?
Experimental psychology does not have a governing agency that sets standards or
establishes certification criteria. As such, students in the General Experimental
Psychology specialization do not have a set of standardized goals or outcomes set by
the APA or any other national board. Applied behavior analysts are required to take a
Board Examination before being granted their certification. At this time, board
certification is not offered through CWU. Currently, we provide only the course work
and the practicum hours required for certification. Students must arrange to take the
certification exam elsewhere. However, it is our long-term goal to have the CWU
psychology department’s ABA specialization board certified. At that point, we will
2009-2010 Annual Program Assessment 35
MS Experimental Psychology
be able to offer the exam as part of our program, which will allow us to keep track of
those data. (Currently, none of our students have progressed far enough to take the
exam.) The timeframe for program certification is between 5-8 years. Until then, we
do not have a nationally standardize measure of student progress beyond successful
course and program completion.
Have you considered admission interviews, standardized advising, portfolios?
As yet we have not begun to formally interview candidates for the MS Experimental
Psychology Program. Dr. Gabriel did encourage faculty to contact applicants with
overlapping interests. But a formal interview process has not been established. This
year, we have had some issues with professionalism and maturity among some of the
new students. As a result, a section on professionalism is being added to the MS
Experimental Psychology handbook. It also appears that a discussion regarding
formal interviews would be useful. It will be part of the agenda for Fall 2010. Finally,
interviews may be particularly necessary for the ABA specialization since
interpersonal skills are mandatory for the career.
It seems clear form the student versions of the assessment survey that a serious
discussion about advising among the program faculty is warranted. It will be high on
our list of agenda items for fall 2010.
The department of psychology has only recently implemented the use of portfolios for
the undergraduate students. Formal portfolios are rare in graduate school but not
unheard of. This too is worth a discussion by the faculty.
Have we considered developing a rubric for thesis defense?
This is on our list for discussion next year. If approved, a subcommittee will develop
a draft rubric that will be reviewed/modified by the entire program committee. The
program committee will bring the finalized draft version to the department chair for
feedback and final approval.
Have your considered making your assessment data available on-line?
Given the radically changing nature of the program at this time, making such data
available would not be meaningful to program applicants. We would prefer to wait
until 2012-2013 before placing our assessment data on-line so that we have time to
gather sufficient longitudinal data as to be meaningful, and not confusing.
Do you have a MS Experimental Psychology Handbook?
In 2008-2009, Dr. Williams develop a 60+ page handbook (with inserts) for new
incoming graduate students in the MS Experimental Psychology program. It includes
important information regarding the faculty, program objectives, prerequisites,
2009-2010 Annual Program Assessment 36
MS Experimental Psychology
internships, malpractice and liability insurance, the handling of complaints, and thesis
and non-thesis regulations and guidelines. As part of this project, she also had the
program committee discuss important issues like mandatory criminal background
checks and fingerprinting for any graduate students that plan to conduct research or
work directly with children or adults with developmental disabilities. A copy of the
handbook was supplied to the assessment team last year.
Next year, handbook inserts will be developed for the ABA specialization, and an
advising handbook will be need to be developed for Internship Supervision (to be
used by faculty and practicum/internship supervisors). But this might be best held
until Dr. Zayac’s replacement has been hired. This year, we have had some issues
with professionalism and maturity among some of the new students. As a result, a
short section on professionalism in the workplace is being added to the MS
Experimental Psychology handbook.
Have we made comparisons across programs?
In 2009-2010, the four graduate program directors and program coordinators met to
ensure that our programming would accommodate courses that overlap across
programs in ways that maximized efficiency. There are several classes that are
required in multiple programs. We wanted to make sure that course offerings would
be offered in ways that meet several criteria:
1. Graduate classes have sufficient enrollment (6 students minimum)
2. Graduate classes are not over-enrolled in ways that are detrimental to the
content.
3. Graduate classes are offered at programmatically appropriate times for each
cohort.
Mutually beneficial solutions were agreed upon and course programming was based
on our recommendations.
Beyond course offerings, we have not attempted to compare the MS Experimental
Psychology assessment data with that of other programs within the CWU College of
the Sciences, within the Psychology Department, nor with similar Experimental
Psychology programs at other universities. The CWU Psychology Department does
conduct a self-study every three years with an outside reviewer. Currently, it is
unclear what it would cost to establish a similar program for our graduate program
alone. Next year, Dr. Williams hopes to meet with the Directors of Mental Health
Counseling, School Counseling and School Psychology to see how they conduct their
assessments.
2009-2010 Annual Program Assessment 37
MS Experimental Psychology
Conclusions
Accomplishments. Overall, the assessment data suggest that as a program we are doing
well, and our new assessments provide a deeper and more thorough view of how our
students are progressing through the MS Experimental Psychology Program. Changes in
recruiting styles have led (in part) to a substantive increase in enrollments over the last 2
years. Many current first and second year students are progressing nicely toward timely
completion of their degrees. The reduction in credits for the MS General Experimental
degree program may help to facilitate this. Several Nth year students returned to the
program and completed their degrees. Between 5 and 10 students should have their
degrees conferred by this time next year. Communication and tracking of student
progress by the Director continues to improve but still requires changes to the way in
which paperwork is tracked. Of particular interest is the need to better track
IACUC/HSRC documentation.
Another important change has been the improved connection between assessment
documents and the student learning objectives for the program. The development of a MS
Experimental Psychology Graduate Handbook assisted new incoming students in
adjusting to the MS Experimental Psychology program and to CWU in general. New
regulations for working with human and animal subjects were established including
requiring all incoming student participate in a background check (and mandatory
fingerprinting for those planning to work with sensitive populations).
The creation of Psychology 505 Colloquium should help facilitate improved progression
through the thesis process; introduce new topics like grant writing, and encourage
conference participation.
The modification of the ABA specialization to allow all course work to be completed in
the 1st year, followed by supervised internships in distant locations should help to
increase the number of BCBA-certified supervisors without having to hire new faculty. It
will also allow students to focus intensely on their practicum skills without having to
divide their attention between internship and coursework. This change will also allow
students to gather thesis data at their internship sites that should improve time-tocompletion somewhat. Finally, the ABA specialization faculty are planning to begin a
conversation about migrating the ABA specialization into an independent master’s degree
program in Applied Behavior Analysis. The evidence is clear that the program has great
potential for growth, but that potential is limited by the time-to-completion. Competing
programs
Concerns. Concerns to be addressed in the future include faculty compliance with
assessment measures, thesis review delays at the graduate office, and the mediation of
student problems and/or concerns in a timely fashion. Plans for addressing and/or
ameliorating these concerns are in place for next year. Additionally, the Director would
like to meet with the CWU Assessment Team to discuss inter-program exchanges where
successful and creative assessment ideas could be shared.
2009-2010 Annual Program Assessment 38
MS Experimental Psychology
Closing Comments
As noted last year, it continues to be necessary to allow the new and updated programs to
become well-established, and to give new tracking and assessment systems sufficient
time to glean enough information to make relevant comparisons across academic years.
To see the effects of current and future modifications to the programs, it will be necessary
to allow students to progress far enough into their programs for meaningful data to be
gathered. Similarly, the program faculty also face with changes to old routines; it takes
time to make new advising and assessment instruments salient, meaningful and
important. We are beginning to see the fruits of our efforts on these fronts. We have also
begun to see data-driven, programmatic changes within the MS Experimental Psychology
degree program that result directly from these ongoing assessment activities. Of course,
our best data will come when we can look across multiple years to see how well the
changes that we implement produce meaningful change for the program and for our
graduate students, for our faculty and for the MS Experimental Psychology Program.
Respectfully submitted,
Wendy A. Williams, PhD.
MS Experimental Psychology Program Director
Department of Psychology
Central Washington University
2009-2010 Annual Program Assessment 39
MS Experimental Psychology
Appendix A. Summary of issues to be addressed by the MS Experimental Psychology
program committee and others in 2010-2011.
1. Thesis Rubric Development
2. How to get students to attend/present at conferences
3. Course of Study form completion in Fall / standardized advising
4. Thesis proposal and Option Approval forms completed by end of first year?
5. Dialog with IACUC and HSRC to improve student progress
6. A handbook for mentoring new graduate faculty (Academic and Thesis Advising)
7. Assessment completion compliance by faculty
8. Benchmark reporting by faculty (discuss with grad school)
9. Formalized internships for both specializations.
10. Revise Program Webpage
11. Time-to-completion: How can we get it closer to 2 years?
12. Inter-program assessment conversations within the department
13. Inter-program assessment exchanges across the College of the Sciences
14. Development of an ABA Handbook Insert for the new MS Exp Psy Graduate
handbook.
2009-2010 Annual Program Assessment 40
MS Experimental Psychology
Appendix B. Student Learning Outcomes for the M.S. Experimental Psychology
Program established in 2007-2008.
1. Write in the language of the discipline, using the elements of style described in the
Publication Manual of the American Psychological Association.
2. Describe and perform data analyses for particular data sets such as traditional
descriptive statistical analyses, inferential statistical analyses, sequential analyses,
single-subject designs and/or multivariate analyses.
3. Describe, assess and utilize common research designs in psychology.
4. Design, conduct, and report psychological research.
5. Describe and contrast major theoretical and practical concepts in core content areas
of research including behavior analysis, cognitive psychology, comparative
psychology, and/or physiological psychology.
6. Engage in scholarly and professional activities, including presenting research at
formal and/or informal professional events.
7. Performing professional skills in a supervised practice setting such as research
teams, internships, and graduate research courses.