1
MSW Evaluation Report: 2011-2012
Submitted by: W. Jay Gabbard, MSW, Ph.D.
To: Dr. Dean May, Department Head-Social Work-Western Kentucky University
Dr. Amy Cappiccie, MSW Program Director
Date: 7/7/12
Contained in this report is a summary of evaluation data gleaned from the MSW program for
the academic year 2011-2012. This information can be used to supplement departmentaland
college reports, as well as for CSWE Accreditation purposes.
ACAT
In order to assess the mastery of foundation level knowledge of MSW students at WKU, the
ACAT measures standard curricular areas within accredited graduate social work programs.
Curriculum A, the one administered to “traditional” WKU MSW students, assesses knowledge in
the following curricular areas:








Human Behavior in the Social Environment
Social Policy
Social Work Practice
Research Methods
Diversity
Populations at Risk
Social and Economic Justice
Values and Ethics
Results from the ACAT are provided to individual students in the form of standard scores for
each curricular area as well as for overall performance for the cohort taking the test. These
curricular areas and the standard scores associated with them can be linked directly to program
objectives.
The ACAT specifically assesses concepts from the following Foundation level objectives: 1.1.1f,
1.1.2f, 1.1.3f, 1.1.4f, 2.1.6f, 2.1.7f, 4.1.10f, 4.1.11f
Results-2 Year Traditional Cohort (Class of 2012)
The ACAT is normally administered at the end of traditional students first year of coursework.
However, due to a delay, 24 students in the 2 Year Traditional Program (Class of 2012) did not
take their ACAT exam until the beginning of Fall Semester 2011 (their second year of classes).
Here are the results of this administration:
2
Area
Standard Score
%ile
Reference Group Size
Diversity
508
53
7941
Populations at Risk
508
53
7941
Social/Economic Justice
503
51
7941
Values and Ethics
530
62
7941
Policy & Services
531
62
8240
Social Work Practice
547
68
8240
HBSE
516
56
8240
Research Methods
555
71
7941
OVERALL PERFORMANCE
536
64
7941
The results of this administration indicated that overall, this cohort of traditional MSW students
performed at the 64th percentile compared to similar MSW students nationwide. This was one
of the highest overall percentiles achieved during the time that the ACAT has been
administered to traditional MSW students at WKU.
Students scored highest in the areas of research and social work practice. They also performed
well in the areas of values and ethics and policy and services. The weakest scores were in the
domains of social/economic justice, diversity, and populations at risk. The results here were a
bit puzzling, as there is a flagship diversity course (SWRK 501) that all students take their first
semester and content on diversity and populations at risk is infused throughout the foundation
year. However, these results will be reviewed (see below) to see if additional diversity content
needs to be added to the curriculum.
At the next MSW program meeting in Fall 2012, the results of this administration will be
discussed with faculty and a strategy devised to maintain strengths and address deficiencies in
the aforementioned areas. As well, ACAT results will be shared at the next Departmental
Advisory Committee (DAC) meeting during the 2011-2012 academic year, in order to solicit
feedback from community members and stakeholders. Results were also provided to students,
in order that they might improve in areas where they were deficient.
3
Results-2 Year Traditional Cohort (Class of 2013)
The traditional MSW class of 2013 (24 students) took the ACAT at the end of their first year of
classes in Spring 2012. Here are the results of this administration (individual and aggregate
results have already been sent to students):
Area
Standard Score
%ile
Reference Group Size
Diversity
532
63
8293
Populations at Risk
513
55
8293
Social/Economic Justice
497
49
8293
Values and Ethics
503
51
8293
Policy & Services
505
52
8740
Social Work Practice
553
70
8740
HBSE
541
66
8740
Research Methods
545
67
8740
OVERALL PERFORMANCE
534
63
8293
The results of this administration indicated that overall, this cohort of traditional MSW students
performed at the 63th percentile compared to similar MSW students nationwide. Thus, the
overall scores from this cohort remained relatively stable when compared to the prior cohort,
who scored at the 64th percentile.
Similar to the prior cohort, students scored highest in the areas of social work practice and
research methods. The areas of greatest improvement (at least in terms of relative percentile
ranks) from the prior administration were in the areas of diversity and HBSE, with both areas up
10 percentile points.
The weakest scores were in the domains of social/economic justice (the lowest from the last
administration), values and ethics, and policy and services. It appears that social/economic
justice needs to be targeted and discussed during the next MSW program meeting in Fall 2012,
as it has been the lowest scoring percentile on the last two tests. As well, these ACAT results
will be shared at the next Departmental Advisory Committee (DAC) meeting during the 20112012 academic year, in order to solicit feedback from community members and stakeholders.
4
Cultural Awareness Inventory
One of the primary goals of the MSW program at WKU is that graduates emerge with the
knowledge, skills, and self-awareness required for culturally sensitive practice. Towards this
end, both advanced standing and traditional students take a flagship diversity course (SWRK
612 and 501 respectively) and diversity content is infused throughout the curriculum. In order
to assess student’s cultural awareness, the MSW program at WKU utilizes an instrument that
has been employed in numerous evaluative studies nationwide.
The Cultural Awareness Inventory is adapted from Lum’s Social Work Cultural Competencies
Self-Assessment (2003). The Cultural Awareness Inventory consists of a number of sections,
including an introduction, background information, 44 statements that are answered on a 4point scale, and two open-ended items at the conclusion of the instrument. The majority of the
instrument is made up of the 44 self-reported statements that encompass the cultural domains
of: cultural awareness, knowledge acquisition, skill development, and inductive learning.
Students respond to each statement based on the following directions:
Rate yourself on your level of competency on a scale of 1 – 4:
1 = Unlikely
2 = Not very likely
3 = Likely
4 = Definitely
Circle the appropriate number.
Again, given the MSW program’s mission “To educate and prepare students for professional social
work practice to meet the needs of increasingly diverse rural populations,” the Cultural Awareness
Inventory is used to assess a critical component of this aspect of the mission statement.
Cultural Awareness Inventory-How the Instrument and Data are Used:
The Cultural Awareness Inventory is employed as a pre-and post-test assessment procedure,
statistically evaluated with dependent t-tests. It is administered at the beginning and conclusion
of the student’s MSW program. The goal is to assess for significant changes (or lack thereof) in
students’ perceived level of cultural competency. While self-perceived change does have its
obvious limitations (diversity content is also measured more concretely on the ACAT), it does
allow for students to reflect back on the knowledge and skills they have acquired in courses and
in fieldwork, which is a valuable exercise for them to engage in at the conclusion of their study.
Results-Class of 2012 (Advanced Standing and Traditional)
Below are the t-test results of this inventory for both advanced standing and traditional
graduates in 2012. There were also a number of students who were not present for the posttest the day it was administered who are not reflected in the data below.
5
Results (2012 Graduates)
Paired Samples Statistics
Mean
Pair 1
N
Std. Deviation
Std. Error Mean
pre cc score
139.90
42
15.540
2.398
post cc score
158.79
42
13.530
2.088
Paired Samples Correlations
N
Pair 1
pre cc score & post cc score
Correlation
42
Sig.
.142
.368
Paired Samples Test
Paired Differences
95% Confidence Interval
Mean
Pair
pre cc score - post
1
cc score
-18.881
Std.
Std. Error
Deviation
Mean
19.097
2.947
of the Difference
Lower
-24.832
Upper
-12.930
Sig. (2t
-6.407
df
tailed)
41
.000
6
Here is a visual display of these results on a scatterplot (notice the upward significant trend):
Looking at the results of the dependent t-test for all 2012 graduates who completed both the pre
and post ratings, there was a significant (at the .001 level) increase in all MSW students’ cultural
competency scores from when they started the program to the end of their course of study. This
trend is further confirmed by the positive upward trend displayed on the scatterplot above.
The average increase of 19 points on the inventory provides evidence that students (from their
perspective) became more sensitive to culturally diverse groups and more skilled in providing
culturally sensitive interventions in practice during their time in the MSW program at WKU.
This is an important finding, as one of the primary goals of the social work profession and MSW
7
program at WKU is to produce graduates who are highly skilled in assessing and intervening
with diverse client populations.
Next, here are the results of the dependent t-test for the 29 advanced standing (e.g. one year)
students in the graduating 2012 cohort who completed the pre and post assessment:
Paired Samples Statistics
Mean
Pair 1
N
Std. Deviation
Std. Error Mean
pre cc score
141.72
29
15.659
2.908
post cc score
160.24
29
11.792
2.190
Paired Samples Correlations
N
Pair 1
pre cc score & post cc score
Correlation
29
Sig.
.411
.027
Paired Samples Test
Paired Differences
95% Confidence Interval
Mean
Pair
pre cc score - post
1
cc score
-18.517
Std.
Std. Error
Deviation
Mean
15.242
2.830
of the Difference
Lower
-24.315
Upper
-12.719
Sig. (2t
-6.542
df
tailed)
28
The results of the advanced standing cohort’s dependent t-test also indicated a significant (at the
.01 level) increase in self-reported cultural competency from orientation to graduation. The
average increase for this cohort (19 points) was similar to that of the overall gain for the
combined graduating class discussed above. Also, the correlation between the scores was
statistically significant at the .05 level (p = .03), unlike that of the overall graduating class.
In sum, the significant increase in cultural competency for advanced standing students supports
the program and professional goals of producing graduates who are sensitive to the needs of their
diverse clients and skilled in providing culturally sensitive interventions. The infusion of
diversity content throughout the curriculum and flagship diversity courses contribute to students’
continual growth in these areas while in the program.
.000
8
Finally, here are the dependent t-test results for the students in the 2 year traditional cohort that
graduated in 2012 (pretest was obtained at orientation in 2010). Please note that there were a
number of students who did not complete the post-test or who left the program (or who shifted to
part-time status), which contributed to the lack of post-assessment data with a number of
students in this cohort.
Paired Samples Statistics
Mean
Pair 1
N
Std. Deviation
Std. Error Mean
pre cc score
135.85
13
15.071
4.180
post cc score
155.54
13
16.865
4.678
Paired Samples Correlations
N
Pair 1
pre cc score & post cc score
Correlation
13
Sig.
-.376
.205
Paired Samples Test
Paired Differences
95% Confidence Interval
Mean
Pair
pre cc score - post
1
cc score
-19.692
Std.
Std. Error
Deviation
Mean
26.512
7.353
of the Difference
Lower
-35.713
Upper
-3.671
Sig. (2t
-2.678
df
tailed)
12
In terms of the traditional two year graduates, they manifested a slightly larger average increase
in cultural competency scores than their advanced standing colleagues and the combined class of
2012 with an average increase of 20 points in their two years in the program. This increase was
significant at the .05 level (p = .02).
Thus, students in both graduating cohorts and the overall group indicated a statistically
significant (at the .05 level) increase in cultural competency during their tenure at WKU. These
results indicated success in reaching a stated goal of the MSW program, with regards to
cultural competency and diversity:
“More specifically, dependent t-tests (students put the last few digits of their student id numbers
on the instruments so pre-post results can be paired) after graduation each year. An increase in
overall positive trend, confirmed by a p value of less than benchmark standard of .05 (traditional
in social science research) is the rejection level for statistically significant findings and an
indication of success in this area.”
.020
9
Foundation and Concentration Assessments
The MSW program faculty and administration are interested in knowing students’ assessments of
their educational outcomes before and after each program sequence. At the beginning and end of
the foundation curriculum for students in the traditional two-year program, and orientation and
end of the MSW program for advanced standing students, assessments are administered to
monitor students’ performance based on a self-efficacy report. On these assessments, students
rank their learning outcomes in relation to all 12 foundation performance indicator. An important
feature (and limitation) of this instrument is that it is self-reported.
Upon beginning and completing the concentration sequence, all students are administered a
concentration assessment to evaluate their performance, based on a self-efficacy report. Students
rank their learning outcomes in relation to all 13 concentration performance indicators (specific
indicators are at the beginning of the document and in the instrument section of Standard 8). An
important feature (and limitation) of this instrument is that it is self-reported.
Below is the pre-test data for the 2011-2012 traditional program cohort (e.g. graduating class of
2013), with regards to the 12 performance indicators identified above:
Descriptive Statistics-Foundation Year Pre
N
Range
Minimum
Maximum
Mean
Std. Deviation
Variance
utilizes critical thinking
13
3
2
5
3.15
.987
.974
applies generalist
13
4
1
5
2.85
.987
.974
13
4
1
5
2.92
1.188
1.410
utilizes communication skills
13
3
2
5
3.54
.776
.603
demonstrates
13
3
2
5
3.69
.855
.731
11
3
2
5
3.64
.924
.855
identifies value conflicts
11
3
2
5
3.45
1.128
1.273
critiques research
11
4
1
5
2.64
1.027
1.055
integrates research
11
4
1
5
2.91
1.044
1.091
interprets historical events
11
3
2
5
3.18
.982
.964
analyzes policies
11
4
1
5
3.00
1.095
1.200
applies strategies and
11
3
2
5
3.27
.905
.818
pre score
13
45
15
60
35.46
11.900
141.603
Valid N (listwise)
11
perspective
synthesizes theoretical
frameworks
professionalism
understands values and
ethics
techniques
interventions
10
Looking at the descriptive statistics above, the average score for all 12 indicators was a 36 (out
of 60 total points), with a standard deviation of 11.90 and variance of 141.60. Students in this
cohort rated themselves highest on the foundation pretest in the areas of demonstrating
professionalism, understanding values and ethics, and utilizing communication skills. They selfrated lowest in the areas of critiquing and integrating research. It is not surprising that entering 2
year MSW students would self-rate lower in these areas, as research is an area that often yields
the most apprehension for entering graduate students.
Here were the post-results on the 12 foundation indicators for this same group of students:
Descriptive Statistics-Foundation Year Post
N
Range
Minimum
Maximum
Mean
Std. Deviation
Variance
utilizes critical thinking
19
2
3
5
3.79
.535
.287
applies generalist
19
2
2
4
3.47
.612
.374
19
3
2
5
3.37
.684
.468
utilizes communication skills
19
2
3
5
4.00
.577
.333
demonstrates
19
3
2
5
4.21
.918
.842
19
1
4
5
4.53
.513
.263
identifies value conflicts
19
2
3
5
4.16
.688
.474
critiques research
19
3
2
5
3.63
.831
.690
integrates research
19
2
2
4
3.58
.692
.480
interprets historical events
19
3
1
4
3.32
.885
.784
analyzes policies
19
4
1
5
3.42
1.071
1.146
applies strategies and
19
2
3
5
3.68
.671
.450
pre score
19
13
39
52
45.16
4.004
16.029
Valid N (listwise)
19
perspective
synthesizes theoretical
frameworks
professionalism
understands values and
ethics
techniques
interventions
Note: A dependent t-test comparison was not possible with this data due to the lack of
coding/id numbers on the post-test. After the traditional MSW cohort’s first year of classes, the
average post score was 45 (out of 60 points), a 9 point overall mean increase from their pre-test
average score at orientation. This was indicative of increased perceptions of competence across a
broad spectrum of foundation indicators.
Once again, the domains of values/ethics and professionalism were the areas where students
reported being the most competent. The areas of critiquing and integrating research techniques
were no longer the lowest (possibly due to the completion of introductory research methods) and
11
the new lowest areas were interpreting historical events and understanding theoretical
frameworks.
The area of greatest perceived growth in the foundation year was in students’ ability to critique
research (plus .99) and that least perceived growth was in their ability to interpret historical
events (plus .14). Also, there was a notable increase (plus .89) in students’ ability to understand
values and ethics within the context of the social work profession.
Here is the standard for evaluating these foundation and concentration level objectives in the
MSW program:
Each objective is evaluated on a five-point continuum:
1
2
3
4
5
(59-below, F)
(60-69, D)
(70-79, C)
(80-89, B)
(90-100, A)
Severely Lacking Competency
Somewhat Inadequate Competency
Adequate Competency
More than Adequate Competency
Outstanding Competence
On all of the post-test objectives, students reported they achieved adequate competency (and in
four domains above average competency). This was indicative of a positive upward trend in
student’s perceptions of their competency in key foundation level areas. These findings were, of
course, tempered by the fact that the data was self-reported.
Shifting to concentration objectives, below are the pretest results across the 13 various
program indicators for the Advanced Standing Class of 2012 (only advanced standing is included
here as no pretest concentration objectives data for traditional students in their second year was
gathered):
12
Descriptive Statistics-Concentration Year Pre (Advanced Standing)
N
generalize foundation
Range
Minimum
Maximum
Mean
Std. Deviation
Variance
28
3
2
5
3.46
.693
.480
28
3
2
5
3.54
.693
.480
28
4
1
5
3.21
.833
.693
28
3
2
5
4.00
.720
.519
28
3
2
5
3.93
.766
.587
28
3
2
5
3.79
.833
.693
27
2
3
5
3.81
.681
.464
27
2
3
5
3.93
.675
.456
27
2
3
5
4.15
.602
.362
27
3
2
5
3.15
.818
.670
analyzes practice research
27
3
2
5
3.19
.786
.618
performs policy rural setting
27
3
2
5
3.04
.706
.499
applies admin skills
27
3
2
5
3.44
.892
.795
pre score
28
35
23
58
45.86
7.658
58.646
Valid N (listwise)
27
knowledge
integrates specialized
knowledge
implements community
organization
demonstrates effective
leadership
demonstrates professional
use of self
integrates understanding
rurality
manages value and ethical
conflicts
evaluates compliance cc
standards
evaluates compliance code
ethics
research competencies
rural
Looking at the results of the pretest scores for the 2011-2012 Advanced Standing cohort, they
rated out an average competency score of 45.86 (out of 65 points), with a standard deviation of
7.66 and a variance of 58.65. Similar to foundation year students, these students rated research
competencies and analysis (as well as policy) as the lowest areas of perceived competency. This
was not a surprise, as graduate social work students traditionally have the most apprehension in
these areas before beginning coursework. This is also after having had BSW courses in these
areas.
The highest ratings were in (similar to foundation year students) ethics, professionalism, and
leadership. In all of the areas surveyed, respondents rated themselves as having “adequate”
competency (e.g. greater than a score of 3) before commencing their MSW studies.
13
Below are the post scores on concentration level objectives for this same Advanced Standing
cohort:
Descriptive Statistics-Concentration Year Post (Advanced Standing)
N
generalize foundation
Range
Minimum
Maximum
Mean
Std. Deviation
Variance
30
2
3
5
4.33
.711
.506
30
3
2
5
4.30
.750
.562
30
3
2
5
3.77
.898
.806
29
2
3
5
4.34
.670
.448
30
2
3
5
4.57
.679
.461
30
2
3
5
4.57
.626
.392
29
2
3
5
4.62
.622
.387
29
2
3
5
4.72
.591
.350
29
2
3
5
4.72
.591
.350
29
3
2
5
4.24
.830
.690
analyzes practice research
29
3
2
5
4.21
.774
.599
performs policy rural setting
29
2
3
5
4.07
.799
.638
applies admin skills
29
3
2
5
4.24
.786
.618
pre score
30
41
24
65
55.53
9.358
87.568
Valid N (listwise)
28
knowledge
integrates specialized
knowledge
implements community
organization
demonstrates effective
leadership
demonstrates professional
use of self
integrates understanding
rurality
manages value and ethical
conflicts
evaluates compliance cc
standards
evaluates compliance code
ethics
research competencies
rural
Looking at the data, similar to foundation level students, there was a marked overall increase in
perceived competency on concentration level objectives, increasing from an average of 45.86 on
the pretest to 55.53 on the posttest, for a total increase of 9.67 points.
For advanced standing students, the domains of greatest perceived growth were in that of
research competency (plus 1.09), analyzing research (plus 1.02), and performing policy in a rural
setting (plus 1.03). These were areas where students rated fairly low on the pretest, so this
development indicated an increased perception of mastery of two areas that these students
perceived themselves as weak in at the beginning of their studies.
14
In terms of areas of least perceived growth, leadership was the first (plus .34) but this was also
the highest domain rated on the pretest. Again, notwithstanding the limitations of self-reported
data, students in this cohort reported being more skilled and competent in all 13 domains
assessed by the instrument and “more than adequate competency” in most domains. The highest
reported areas of competency were in ethics and the ability to evaluate cultural competency
standards, while the lowest in the advanced standing cohort was in the ability to implement
community organization strategies.
In terms of traditional students completing their second concentration year of the MSW program,
because no pre-test of concentration level objectives was obtained at the beginning of the 20112012 academic year, only posttest data was obtained for this cohort during their final year. This
data is displayed below
Descriptive Statistics-Concentration Year Post (Traditional)
N
generalize foundation
Range
Minimum
Maximum
Mean
Std. Deviation
Variance
15
2
3
5
4.20
.561
.314
15
2
3
5
4.20
.676
.457
15
2
3
5
4.00
.655
.429
15
2
3
5
4.27
.594
.352
15
2
3
5
4.60
.632
.400
15
1
4
5
4.60
.507
.257
15
2
3
5
4.20
.676
.457
15
1
4
5
4.73
.458
.210
15
1
4
5
4.67
.488
.238
15
2
3
5
4.13
.743
.552
analyzes practice research
15
2
3
5
4.13
.640
.410
performs policy rural setting
15
2
3
5
4.00
.655
.429
applies admin skills
15
2
3
5
4.33
.617
.381
pre score
15
21
44
65
55.07
5.587
31.210
Valid N (listwise)
15
knowledge
integrates specialized
knowledge
implements community
organization
demonstrates effective
leadership
demonstrates professional
use of self
integrates understanding
rurality
manages value and ethical
conflicts
evaluates compliance cc
standards
evaluates compliance code
ethics
research competencies
rural
15
Looking at the posttest results, the average perception of competency for traditional students
completing their two-year program was 55.07 (compared to an average of 55.53 for advanced
standing students). Their highest rated domains were (similar to other cohorts at both the
foundation and concentration levels) in the areas of ethics and compliance with cultural
competency standards. Their lowest perceived areas of competency were in the areas of research,
policy, and community organization, similar to the results of the advanced standing cohort. On
all domains, students self-rated as performing at “more than adequate competency.”