1
MSW Evaluation Report: 2013-2014
Submitted by: W. Jay Gabbard, MSW, Ph.D., WKU MSW Program Assessment Coordinator
To: Dr. Dean May, Department Head-Social Work-Western Kentucky University
Dr. Amy Cappiccie, MSW Program Director
Date: 6/16/2014
Contained in this report is a summary of evaluation data gleaned from the MSW program for the academic year 2013-2014. This
information will be utilized to supplement departmental and college reports, as well as for CSWE Accreditation purposes. As well,
these results will be shared with the faculty and larger social work community, in order to continually receive input and improve the
quality of the MSW program at WKU.
ACAT
In order to assess the mastery of foundation level knowledge of MSW students at WKU, the ACAT assesses standard curricular areas
within accredited graduate social work programs in the United States. Curriculum A, the one administered to “traditional” WKU
MSW students, assesses knowledge in the following curricular areas:








Human Behavior in the Social Environment
Social Policy
Social Work Practice
Research Methods
Diversity
Populations at Risk
Social and Economic Justice
Values and Ethics
Results from the ACAT are provided to individual students in the form of standard scores for each curricular area as well as for
overall performance for the cohort taking the test. These curricular areas and the standard scores associated with them can be linked
directly to current program objectives.
The ACAT specifically assesses the following explicit curriculum standards from the 2008 CSWE Educational Policy and
Accreditation Standards (EPAS): 2.1.1-2.1.10.
2
ACAT Results-2 Year Traditional Cohort (Class of 2015)
As is done annually, the ACAT was administered to WKU MSW students at the end of their foundation year of courses on April 26,
2014. The exam is carefully proctored and traditional students are required to sit for the exam. Below are the results of this latest
administration:
Area
Standard Score
%ile
Reference Group Size
Diversity
490
46
7187
Populations at Risk
504
52
7187
Social/Economic Justice
518
57
7187
Values and Ethics
531
62
7187
Policy & Services
536
64
7750
Social Work Practice
575
77
7750
HBSE
499
50
7750
Research Methods
567
73
7750
OVERALL PERFORMANCE
539
65
7187
The results of this administration indicate that overall, this cohort of traditional MSW students performed at the 65th percentile
compared to similar MSW students nationwide. Thus, per the official ACAT report, based on a comparison group of 7187 MSW
students taking the ACAT with 8 areas evaluated, 65% would be expected to score at or below WKU’s overall performance score and
35% would be expected to score higher.
3
Per the 2013-2014 WKU MSW score report, “ACAT scores range from 200 to 800 with an average of 500 and a standard deviation of
100. Nationally, 68 percent of the scores in any given year fall between approximately 400 and 600. Year to year variations in the size
of the reference groups will cause scores to fall outside these limits. The content area scores are compared with a reference group of
other examinees taking the same content area. The overall performance score is compared with other examinees taking the ACAT in
this discipline with the same number of content areas.”
Students in this cohort scored highest in the areas of social work practice and research methods, just as they have consistently for
most of the ten years the test has been administered at WKU. They also fared well in the areas of Policy & Services and Values &
Ethics, two areas where they have traditionally scored lowest. The lowest mean scores were in the domains of diversity and HBSE.
The diversity results were interesting, given the heavy infusion of diversity content in foundation courses and flagship diversity course
required during the first semester. However, these results will be reviewed at a faculty retreat (see below) to see if additional diversity
content needs to be added or modified in the curriculum.
In terms of overall performance, scores increased 6 points since the last administration in 2013. The areas of greatest growth were in
Values & Ethics (plus 13 percentile points) and Social Work Practice (plus 11 percentile points). The increase in the Ethics domain is
notable due to the fact scores in this area have often been the lowest with foundation level MSW students at WKU. Practice has
usually been the highest rated domain on the ACAT with our students, so this additional increase in scores is auspicious.
The areas of most significant decline were in Diversity (minus 8 percentile points) and Social and Economic Justice (minus 5
percentile points). Diversity has traditionally been one of the lowest rated domains on the ACAT with our foundation students, even
though they consistently (see below) manifest statistically significant gains in cultural self-awareness on the Lum’s inventory.
Interestingly, one student in this foundation cohort attained the highest overall percentile score ever in the MSW program (96th
percentile) while another earned one of the lowest scores ever (10th percentile). Percentile wise, this group’s performance was one of
the highest in the history of the ACAT at WKU, which is certainly a positive development.
As in prior years, the results of this administration will be discussed at the MSW Faculty Retreat on August 20, 2014 and a strategy
devised to maintain strengths and address perceived deficiencies in the aforementioned areas. As well, ACAT results will be shared at
the next Departmental Advisory Committee (DAC) meeting during the 2014-2015 academic year, in order to solicit feedback from
community members and stakeholders on how to improve in weaker areas. This helps close the feedback loop with a group heavily
vested in the success of the program.
4
Individual results were also provided to students, in order that they might assess areas of strengths and weaknesses as they move
forward into their concentration year of studies and prepare for the Practice Readiness Exam (PRE) exam. Their academic advisors
play an integral role in this process.
Lum’s Cultural Awareness Inventory
One of the primary goals of the MSW program at WKU is that graduates emerge with the knowledge, skills, and self-awareness
required for culturally sensitive practice (EPAS 2.1.4; Gabbard, Starks, Cappiccie, & Jaggers, 2011). Towards this end, both advanced
standing and traditional students complete flagship diversity courses (SWRK 612 or SWRK 501) to master diversity and cultural
competency content. This material is also infused throughout the curriculum and in many electives such as Homelessness, Forensic
Social Work, Expressive Therapies, and Alternative Therapies. In order to assess student’s cultural self-awareness, the MSW program
at WKU utilizes an instrument that has been validated and employed in numerous diversity evaluative studies nationwide.
The Cultural Awareness Inventory is adapted from Lum’s Social Work Cultural Competencies Self-Assessment (2003). The
Cultural Awareness Inventory consists of a number of sections, including an introduction, background information, 44 statements that
are answered on a 4-point scale, and two open-ended items at the conclusion of the instrument.
The majority of the instrument is made up of the 44 self-reported statements that encompass the cultural domains of: cultural
awareness, knowledge acquisition, skill development, and inductive learning. Students respond to each statement based on the
following directions:
Rate yourself on your level of competency on a scale of 1 – 4:
1 = Unlikely
2 = Not very likely
Circle the appropriate number.
3 = Likely
4 = Definitely
5
Given EPAS 2.1.4 and the MSW program’s stated mission, “To educate and prepare students for professional social work practice to
meet the needs of increasingly diverse rural populations,” the Cultural Awareness Inventory is used to assess a critical component of
this program standard and mission. This instrument measures self-perception of cultural awareness while two additional instruments
(ACAT and PRE) assess more concrete diversity concepts learned throughout the program. It is this combination of knowledge, skills,
and self-awareness that constitutes the preparation of culturally competent social work practitioners (Gabbard, Starks, Cappiccie, &
Jaggers, 2011). The program wants to produce graduates who know the traditions, histories, and cultural norms of multicultural groups
and can put this knowledge into practice by designing culturally sensitive interventions.
Cultural Awareness Inventory-How the Instrument and Data are Used:
The Cultural Awareness Inventory is employed as a pre-and post-test assessment procedure, statistically evaluated with dependent ttests. It is administered at the beginning (orientation) and conclusion of the student’s MSW program. The goal is to assess for
significant changes (or lack thereof) in students’ perceived level of cultural competency and cultural self-awareness. While selfperceived change does have its obvious limitations (diversity content is also measured more concretely on the ACAT and PRE), it
does allow for students to reflect back on the knowledge and skills they have acquired in courses and in fieldwork, which is a valuable
exercise for them to engage in at the conclusion of their study. This, coupled with a self-reflection paper they write in their flagship
course, helps them to identify biases, stereotypes, and schemas they must come to grips with in serving multicultural (often oppressed)
clients.
Results-Class of 2014 (Advanced Standing and Traditional)
Below are descriptive statistics on all individuals who completed the pre and post-tests in the 2015 graduating class. Following this
data are dependent t-test results of this inventory for advanced standing and traditional graduates in 2014. While the scores are
approximately normally distributed (per the primary assumption of parametric tests), a nonparametric Wilcoxon Sign test was also
run for comparative purposes, given the presence of some outliers and relatively low correlation between scores. It is important to note
there were a number of students who did not complete instruments, who were not present for the pre-test and/or post-test when they
were administered or who did not enter the last four digits of their student number (for pre-post comparisons) who are not reflected in
the data below.
6
Descriptive Statistics
N
Range
Minimum
Maximum
Mean
Std. Deviation
Variance
age
36
30
22
52
32.89
8.834
78.044
pre cc score
36
56
113
169
137.61
13.217
174.702
post cc score
36
63
113
176
155.44
14.038
197.054
Valid N (listwise)
36
gender
Frequency
Percent
Valid Percent
Cumulative
Percent
male
Valid
5
13.9
13.9
13.9
female
31
86.1
86.1
100.0
Total
36
100.0
100.0
ethnicity
Frequency
Percent
Valid Percent
Cumulative
Percent
Valid
european american
23
63.9
63.9
63.9
african american
10
27.8
27.8
91.7
latino american
2
5.6
5.6
97.2
asian american
1
2.8
2.8
100.0
36
100.0
100.0
Total
The average age of respondents was 33 years old, with a range of 30 years and a standard deviation of 9 years. Similar to the
demographic constitution of the MSW program, most of the respondents were female. In terms of ethnographic makeup, the majority
of respondents were Caucasian, followed by African-American, Latino, and Asian-American respondents.
7
Paired Sample t-Test Results (2014 Graduates)
Paired Samples Statistics
Mean
N
Std.
Deviation
pre cc score
137.61
36
13.217
Pair 1
post cc score
155.44
36
14.038
Paired Samples Correlations
N
r
pre cc score & post cc
36
.286
Pair 1
score
Std. Error
Mean
2.203
2.340
Sig.
.090
Paired Samples Test
Paired Differences
Mean
Pair 1
pre cc score - post cc
score
-17.833
Std. Deviation
16.295
Std. Error
Mean
2.716
t
df
Sig. (2-tailed)
95% Confidence Interval of the
Difference
Lower
-23.347
Upper
-12.320
-6.567
35
.000
8
Looking at the results of the dependent t-test for the 2014 graduates (both traditional and advanced standing) who completed both the
pre and post ratings, there was a significant (at the .01 level) increase in MSW students’ cultural competency scores from when they
started the program at orientation to the end of their course of study. In terms of correlation between the two variables, it was not
significant at the .05 level (r =.90) and displayed a low, positive relationship between the variables. Due to the low correlation
between the scores and presence of some outliers in the data, a Wilcoxon Sign test (the nonparametric equivalent of the dependent t
test) was also run, for comparative purposes. This test also a significant result at the .01 level.
The average increase of 18 points on the inventory yields evidence that students (from their perspective) became more sensitive to
culturally diverse groups and skilled in providing culturally sensitive interventions in practice during their time in the MSW program
at WKU. This is an important finding, as one of the primary goals of the social work profession and MSW program at WKU is to
produce graduates who are highly skilled in assessing and intervening with diverse client populations.
Next are displayed the results and a graph of the dependent t-test for 24 advanced standing students in the graduating 2014 cohort
who completed the pre and post Lum’s assessment.
Paired Samples Statistics
Mean
N
Std.
Deviation
pre cc score
139.96
24
12.178
Pair 1
post cc score
159.21
24
11.740
Std. Error
Mean
2.486
2.396
9
Paired Samples Correlations
N
Correlatio
n
pre cc score & post cc
24
.141
Pair 1
score
Pair 1
pre cc score - post cc
score
Sig.
.510
Paired Samples Test
Paired Differences
Mean
Std. Deviation
Std. Error
95% Confidence Interval of the
Mean
Difference
Lower
Upper
-19.250
15.674
3.199
-25.869
-12.631
t
-6.017
df
Sig. (2-tailed)
23
The results of the sample of students in the advanced standing cohort’s dependent t-test also indicated a statistically significant (at the
.01 level) increase in self-reported cultural sensitivity from orientation to graduation. The average increase for this cohort (19 points)
was higher than the overall gain for the combined graduating class. Similar to the combined results, the correlation between the two
variables was not significant at the .05 level and low positive. The results of the nonparametric Wilcoxon Sign test yielded similar
results with a significant difference between the scores at the .01 level.
.000
10
The significant increase in cultural competency for advanced standing students supports the program and professional goals of
producing graduates who are sensitive to the needs of their diverse clients and skilled in providing culturally sensitive interventions.
The infusion of diversity content throughout the curriculum and flagship diversity courses (for these students, SWRK 612),
contributes to students’ continual growth in these areas while in the program.
Finally, below are the dependent t-test results for 12 students in the 2 year traditional cohort who graduated in 2013 (pretest was
obtained at orientation in 2012). Please note that there were a number of students who did not complete the pre or post-test, did not
enter the last four digits of their student number for comparison purposes, or who left the program (or shifted to part-time status),
which contributed to lower completed pre/post numbers with this cohort.
Paired Samples Statistics
Mean
N
Std.
Deviation
pre cc score
132.92
12
14.482
Pair 1
post cc score
147.92
12
15.687
Paired Samples Correlations
N
Correlatio
n
pre cc score & post cc
12
.303
Pair 1
score
Std. Error
Mean
4.181
4.528
Sig.
.338
Paired Samples Test
Paired Differences
Mean
Pair 1
pre cc score - post cc
score
-15.000
Std. Deviation
17.833
95% Confidence Interval of the
Difference
Lower
Upper
5.148
-26.330
-3.670
t
df
Sig. (2-tailed)
Std. Error
Mean
-2.914
11
.014
11
This sample of traditional program graduates manifested a lower average increase in cultural competency scores (15 points) than their
advanced standing colleagues and the combined class result that was statistically significant at the .05 level.
Similar to the overall and advanced standing correlation results, the r value was statistically insignificant at the .05 level and low
positive. Per the Hypothesis Test Summary displayed above, the nonparametric Wilcoxon Sign test also yielded a statistically
significant result at the .05 level. However, the fewer number of individuals in this cohort should be taken into consideration when
evaluating the results of these tests.
Historically, in past graduating cohorts, there has been a more pronounced increase in cultural competency scores for traditional
students (compared to advanced standing ones) that might have to do with the fact they take more courses with diversity content in
them at the graduate level. As well, their flagship diversity course (SWRK 501) is solely dedicated to cultural diversity and cultural
competency. As well, many of these students may have (unlike Advanced Standing students) had less exposure to these concepts in
their undergraduate degree programs, so their gains in this area would likely be more pronounced than those with a prior history of
exposure to diversity in BSW programs.
However, this cohort was atypical in the fact that they actually displayed a notably lower increase in self-awareness scores.
Interestingly, two individuals in this traditional cohort actually rated lower on their post-scores and one rated no change at all. This is
definitely an atypical finding that should be noted. It will also be discussed at the MSW retreat in Fall 2014 in order to assess if any
additional considerations need to be considered with this cohort.
In sum, students in both graduating cohorts, as well as the overall group, indicated a statistically significant (at the .01 and .05 levels)
increase in self-reported cultural awareness during their tenure at WKU. These results indicated a degree of success in reaching a
stated goal of the MSW program, with regards to cultural competency and diversity:
12
“More specifically, dependent t-tests (students put the last few digits of their student id numbers on the instruments so pre-post results
can be paired) after graduation each year. An increase in overall positive trend, confirmed by a p value of less than benchmark
standard of .05 (traditional in social science research) is the rejection level for statistically significant findings and an indication of
success in this area.”
While the presence of intervening variables (e.g. additional diversity trainings, familiarity with the test, history) likely influenced these
results, it appears that students in the 2014 graduating class perceived themselves as more culturally competent at the end of the
program than when they matriculated. The fact this trend had consistently been maintained over the life of the MSW program at WKU
is evidence that faculty are doing an adequate job of preparing these students for culturally sensitive practice.
PRACTICE READINESS EXAM (PRE)
Per departmental and WKU Graduate School requirements, all graduating MSW students must successfully pass a Practice Readiness
Exam (PRE) during their final semester in order to successfully complete the program. Per the 2013-2014 MSW Student Handbook,
the PRE exam gives students the opportunity to demonstrate basic competency in essential content areas of social work practice with
an emphasis in rural settings.
The exam contains 100 multiple choice questions in an objective format, similar to social work licensure exams. It is indeed an
excellent tool to prepare students to pass the Intermediate licensure exam post-graduation, one of the desired outcomes of the MSW
Program at WKU. While the MSW Program does not publish licensure pass-rate statistics, historical and anecdotal evidence points to
a considerable degree of success in this area, at both the CSW and LCSW levels.
Students are required to pass the PRE exam with a score of 70 or better in order to graduate from the program. Per the WKU Graduate
School Policy, a student only has two attempts to pass this comprehensive exam. Consequently, a student who fails the exam two
times is subject to dismissal from the MSW Program. However, it should also be noted that students who fail the exam are provided
with written information regarding their performance and advised on areas they need to improve on before they take it (in several
weeks or during the next Fall semester).
As the test assesses knowledge of specific content from readings, lectures, and field practicums/experiences, it is considered an
important evaluative measure of student learning and retention in the program. It measures not only concrete concepts gleaned but also
puts students in “practice situations” where they must respond based on their professional and ethical training. It has been continually
refined over the 10 years it has been administered and an item analysis is conducted after each administration to assess the efficacy of
specific questions (to decide which ones to keep and which to discard). New questions are continually developed to have fresh
questions on each administration, in addition to “battle tested” ones.
13
It should also be noted that, after evaluating the evaluation strategy employed in years past, a decision was made this year to drop the
self-rated foundation and concentration level objective tool in lieu of a more concentrated evaluation of the PRE, which is anchored in
conceptual knowledge and the actual ability of students to evaluate practice and ethical situations. In the future, even more complex
analyses will be conducted with PRE results and they will be correlated with other measures such as the ACAT and Lum’s Cultural
Competency Inventory, in order to assess program outcomes.
The 2014 exam included questions in the following subject areas (numbers of questions on each area exam are denoted):
 Cultural Competency and Diversity(4)
 Foundation (27) and Advanced Practice (16)
 Human Behavior in the Social Environment (7)
 Ethics (6)
 Foundation (10) and Advanced Social Welfare Policy (4)
 Groups (6)
 Diagnosis (5)
 Family Practice (6)
 Administration and Supervision (4)
 Community Organization (6)
 Rural Communities (2)
 Research (7)
Approximately 67 students took the PRE on April 7, 2014. Of these, 59 students achieved a passing score of 70 and 8 failed to achieve
this benchmark (following item analysis). Thus, approximately 88 percent of students taking the exam passed and 12 percent failed
during this administration. These numbers were similar to prior success rates on the exam over a ten-year period, which usually
average around 10 percent needing to do a retake.
The subject areas where the most students appeared to struggle (taking into account the number of questions in each area) were:
Foundation Policy, Diagnosis, and Community Organization. The areas students missed the fewest questions in were: Ethics,
Advanced Practice, Research, and Foundation Practice.
The first result is interesting given the fact traditional students in this graduation cohort tended to score only average on Foundation
Level Policy questions on the ACAT. Of course, as mentioned above, there was a marked improvement in this subject area with
students taking the ACAT this year (Class of 2015). While these two tests are not identical in terms of content, it is valuable to look at
trends across evaluative measures.
14
Some of the areas where students fared the best (Foundation Practice and Research Methods) were perhaps not coincidentally the
same two areas that foundation students usually perform the best in on the ACAT. For years, these have been the two domains where
students have scored the highest in on the ACAT and this pattern is usually also present on the PRE.
Students have also traditionally performed well on the Advanced Practice questions on the PRE. Interestingly, they have usually
scored low in the Ethics domain on the ACAT but they performed exceptionally well in this area on the PRE. Scores also improved
dramatically in this domain on this year’s ACAT results, indicating a shift in student performance.
Similar to the ACAT and Lum’s Inventory, the results of the PRE will be reviewed at the MSW Program Retreat in August 2014. An
additional factor that will be discussed is whether to offer the test on a computer in future administrations, which would even further
mirror the process with the CSW and LCSW exams. As well, the PRE committee will review the efficacy of the questions from this
year and create new ones for the 2015 test.
MSW FIELD DATA/EVALUATION
Just as with the domains assessed above (and in accordance with the accreditation standards established by the Council on Social
Work Education (CSWE)), the MSW Program annually collects field data, in order to assess the program’s compliance and success in
meeting established benchmarks for each standard. More specifically, the percentage of students achieving each CSWE competency is
calculated for both traditional and advanced standing graduates. This information, which is gathered by the Field Director and posted
on the MSW Program website, helps the program evaluate areas of strength and potential growth in field objectives. Below is a
summary from May 2014 summarizing the field assessment for academic year 2013-2014:
ASSESSMENT OF STUDENT LEARNING OUTCOMES
WESTERN KENTUCKY UNIVERSITY MSW PROGRAM
LAST COMPLETED ON MAY 2014
This form is used to assist the COA in the evaluation of the program’s compliance with Accreditation Standards stated below.
4.0.2 The program provides evidence of ongoing data collection and analysis and discusses how it uses assessment data to
affirm and/or make changes in the explicit and implicit curriculum to enhance student performance.
4.0.5 The program appends the summary data for each measure used to assess the attainment of each competency for at least
one academic year prior to the submission of the self-study.
15
All Council on Social Work Education programs measure and report student learning outcomes. Students are assessed on their
mastery of the competencies which comprise the accreditation standards of the Council on Social Work Education. These
competencies are dimensions of social work practice which all social workers are expected to master during their professional
training. A measurement benchmark is set by the social work programs for each competency. An assessment score at or
above that benchmark is considered by the program to represent mastery of that particular competency.
COMPETENCY
COMPETENCY
BENCHMARK
PERCENT OF STUDENTS ACHIEVING BENCHMARK
% OF STUDENTS
ACHIEVING BENCHMARK
(TRADITIONAL)
% OF STUDENTS ACHIEVING
BENCHMARK
(ADVANCED STANDING)
Identify as a
professional
social worker and
conduct oneself
accordingly.
Apply social work
ethical
Principles to
provide
professional
practice.
Apply critical
thinking to inform
and communicate
professional
judgments.
Engage diversity
and difference in
practice.
70% of students
will earn 4 or
higher
98%
98%
70% of students
will earn 4 or
higher
80%
95%
70% of students
will earn 4 or
higher
73%
93%
70% of students
will earn 4 or
higher
80%
86%
Advance human
rights and social
70% of students
will earn 4 or
80%
87%
16
and economic
justice.
Engage in researchinformed practice
and
practice informedresearch.
Apply knowledge
of human behavior
and the social
environment.
Engage in policy
practice to advance
social and
economic wellbeing and to deliver
effective social
work services.
Respond to
contexts and shape
practice.
Practice
Engagement
Practice
Assessment
Practice
Intervention
Practice
Evaluation
higher
70% of students
will earn 4 or
higher
80%
84%
70% of students
will earn 4 or
higher
86%
95%
86%
86%
86%
84%
86%
95%
86%
95%
86%
95%
86%
95%
70% of students
will earn 4 or
higher
70% of students
will earn 4 or
higher
70% of students
will earn 4 or
higher
70% of students
will earn 4 or
higher
70% of students
will earn 4 or
higher
70% of students
will earn 4 or
higher
Looking at the results above, it is apparent that the MSW Program was successful during the 2013-2014 year in meeting all of the
competency benchmarks established by the CSWE, across both program tracks. Traditional students rated particularly high on
identification and conduct as a professional social worker and practice engagement, assessment, intervention, and evaluation.
17
Advanced standing students similarly rated high on identification and conduct. They also significantly exceeded the benchmark on
applying ethical and HBSE principals in practice, as well as in practice considerations. One area that rated lowest for both groups was
the research-practice linkage, which is continually stressed in the Research Methods and Applied Research courses. While students
exceeded the benchmark in this area, perhaps new strategies can be devised to help students recognize and apply research concepts in
the field/practice arena. This topic will be discussed at the Fall 2014 MSW Program Retreat.
References
Gabbard, W. J., & Starks, S., Cappiccie, A., & Jaggers, J. (2011). An effective model of teaching cultural competency to MSW
18
students in a global society. Journal of Human and Social Sciences, 6(2), 204-208.
Lum, D. (2003). Culturally competent practice: A framework for understanding diverse groups and justice issues. Pacific Grove, CA:
Brooks/Cole.