University of Missouri – Kansas City
General Education Core Assessment – 2013-14 and 2014-15: Executive Summary
September 2015
Ruth E. Cain, Ed.D., Director of Assessment
Concurrent with the introduction of the 2013 General Education Core, UMKC developed tow
direct methods to assess student achievement of the general education learning outcomes: 1) data
collection by Discourse faculty utilizing shared rubrics to evaluate student portfolios, and 2)
course-embedded assessment of student artifacts in the Anchor and Focus courses. A third direct
method, the ETS® Proficiency Profile, was used by UMKC prior to the introduction of the 2013
General Education Core and continues to be used. Details concerning these three assessment
methods and findings from 2013-14 and 2014-15 are contained in the attached reports 1.
The ultimate purpose of assessment is to make course and programmatic alterations that result in
enhanced student learning, and to evaluate the impact of those changes on student learning.
Therefore, this summary document provides an overview of the general education assessment
results for 2013-14 and 2014-15 and suggestions for using the data to inform improvements to
teaching and learning to enhance student achievement of the general education outcomes. It also
summarizes planned and recommended changes to increase the validity and reliability, as well as
efficiency, of assessment methods in future assessment cycles.
Assessment of the Discourse I, II, and III
In 2014-15, faculty teaching Discourse I, II, and III utilized rubrics to assess portfolios
containing student artifacts collected throughout the semester. The rubrics utilized a scale of 0 –
4, with 4 indicating the highest level of achievement. While the rubrics were designed to reflect
the specific student learning outcomes for each Discourse level, there are several criteria which
remain constant across the rubrics: Demonstrate Critical Discourse Analysis and Critical
Language Awareness; Use of Academic Discourse; Revise; Critique; and Research.
The assessment results for these criteria indicate a consistent increase in student achievement
across the three levels of Discourse, as detailed in Table 1 2. However, as indicated in Table 2,
the percentages of students achieving at the highest level (4 on a scale of 0 – 4) on the rubric
criteria in Discourse III ranged from slightly over 25% to approximately 40%, which suggests
that students are not achieving these outcomes at desired levels by the time they complete the
Discourse sequence. Although achievement targets were not established for the rubric criteria,
as Discourse III is the last course in the sequence, aggregate student achievement should be
expected at a much higher percentage than the 2013-14 results indicate (Table 2), and a goal of
75% - 80% of students achieving at the highest level may not be unreasonable.
ETS® Proficiency Profile
The ETS® Proficiency Profile (EPP) is a nationally-normed, multiple-choice test designed to
assess students’ abilities in four areas: critical thinking, reading, writing, and mathematics. The
EPP has two scoring conventions: norm-referenced (scaled) scores and criterion referenced
1
These three reports are: Pilot Assessment of the Final Portfolios in Discourse I, II, and III: 2014-15; Using the
ETS® Proficiency Profile to Assess General Education Learning Outcomes; and General Education Core
Assessment: 2013-14 Course Reports.
2
All data tables referenced in this report are located in the Appendix.
1
scores. Total scaled scores have a possible range of 400-500, and each of the four subscores has
a possible range of 100-130. Table 3 presents the total scaled scores and subcores for UMKC
seniors for 2010-11 through 2014-15. These data indicate that little variation in scores over the
past five years and that UMKC student consistently score lower on critical thinking than on the
other three skills. The five-year aggregate results in Table 4 indicate that UMKC students scored
slightly higher than students at peer-institutions 3 on the reading, but scored significantly lower
(p<0.001) than students at the peer institutions on the total scaled score and on critical thinking,
writing, and mathematics.
ETS® estimates the number of students who were proficient, not proficient, or marginal at each
identified proficiency level. (Students classified as marginal are those whose test results do not
provide enough evidence to classify the student as either proficient or not proficient and does not
ascribe any positive or negative value.) Table 5 provides the proficiency classification for
students who completed the exam in 2014-15 and Table 6 presents the percentages of students
classified as proficient over the past five years. As with the scaled scores, the proficiency data
indicate a consistent decline in student achievement in critical thinking over five years and
declines in proficiency in reading at Level 2 and mathematics at Levels 2 and 3.
Focus and Anchor Course Assessment Reports
Sixty 2013-14 General Education Course Assessment Reports were analyzed to evaluate student
achievement of four General Education learning outcomes. As detailed in Table 7, results for
each of the four outcomes for which reports were received suggest the following:
• for Scientific Reasoning and Quantitative Analysis, of the 18 reports, 11 reported that the
outcomes were met; 4 reported mixed results (i.e., some of the outcomes were met, others
were not met); and 3 reported that none of the outcomes were met;
• for Arts and Humanities, all 18 reports indicated that the outcomes were met;
• for Culture and Diversity, of the 6 reports received, 5 indicated that the outcomes were
met; and 1 reported mixed results; and
• for Human Actions, Values and Ethics, of the 18 reports received, 10 reported the
outcomes were met; 5 reported mixed results; and 3 reported the outcomes were not met.
While these results are highly positive, over half the reports relied on summative grades and
twenty percent did not provide substantiating evidence. Additionally, a substantial portion of the
reports did not link the course SLOs assessed to the general education outcome(s) designated in
the report. Therefore, these results may not be valid or reliable indicators of student achievement
of the General Education outcomes identified for the Anchor and Focus courses.
Using the General Education Core Assessment Results
The results of the three General Education Core assessment methods, together with the
evaluation of the effectiveness of the methods, indicate the need to enhance achievement of the
general education learning outcomes, as well as to revise assessment methods to ensure the
validity and reliability of assessment methods.
3
For 2014-15 analysis, the institutions included in the comparative analysis were: Arizona State University –
Tempe, Florida International University, Temple University, The University of Akron, University of Alabama at
Birmingham, University of Cincinnati, University of Memphis, University of Nevada, University of South Florida –
Tampa, and Wayne State University.
2
For example, Discourse faculty, in collaboration with the English and Communication
departments might select the learning outcomes for which student achievement is the lowest or
other essential learning outcomes and develop strategies to enhance development of the
identified skills. For example, assignments could be reviewed and revised to ensure they elicit
the identified learning outcomes and that student experiences are appropriately scaffolded, both
within and across Discourse levels, as well as in conjunction with the paired Anchor courses, to
provide opportunities for skill development. Faculty may also consider opportunities to model
specific abilities, such as revising one’s own or critiquing others’ work.
To further evaluate the achievement of UMKC students on the mathematics subcomponent of the
EPP the results of students majoring math-intensive fields (e.g., STEM disciplines, statistics,
natural and social sciences) could be disaggregated from those in disciplines such as arts and
humanities. The institution should identify desired levels of achievement for both groups and
evaluate student achievement against those benchmarks. Such analysis would inform
discussions concerning needed supports (e.g., course and curriculum revisions, additional
tutoring opportunities) to enhance student achievement in mathematics in the general education
program and in major fields. EPP results from 2014-15 and prior should be used as benchmarks
against which to evaluate the results of students who entered UMKC in Fall 2013 and later to
assist in assessing the impact of the 2013 General Education Core on student learning.
Strategies to enhance development of these essential learning outcomes such as information
literacy, critical thinking, and writing might include development of additional writing-intensive
(WI) courses; requirements for WI courses in all academic units; development of assignments
that elicit critical thinking; development of collaborative rather than individual research
assignments; or scaffolding of assignments within and across general education courses.
Institutional supports for faculty and curriculum development offered through FaCET could
include workshops facilitated by nationally-recognized experts in teaching and evaluating
general education outcomes, reading circles, and opportunities for faculty to share good
practices. These and other professional development opportunities could focus on topics such as
course design/redesign to ensure educational experiences support achievement of the general
education outcomes; development of writing prompts and assignments to elicit demonstration of
the desired outcomes; scaffolding assignment and sequential curriculum design to support
development of the abilities; and methods to evaluate critical thinking and writing. Institutional
support could also include funding for collaborative groups to attending regional and national
conferences on teaching and assessing general education outcomes.
The evaluation of the initial assessment of the General Education outcomes revealed the need to
improve the effectiveness and efficiency of data collection. As a first step, the GECC developed
a three-year cycle for data collection to replace the initial effort to assess every course every
year. The Discourse Coordinator and faculty have modified assessment methods. Beginning in
2015-16, shared speech and essay rubrics will be used to assess student achievement across the
three levels of Discourse. The GECC is considering alternatives to the Anchor and Focus course
report forms, such as developing shared rubrics and review panels to evaluate representative
samples of student artifacts. While the use of assessment panels would increase the validity and
reliability of assessment of the General Education Core, it would also necessitate institutional
support, such as funding for summer stipends for rubric norming and artifact evaluation sessions.
3
DATA TABLES
Table 1: Discourse I, II, III 2014-15: Percentage of Students Achieving a 3 or Higher on Each
Criteria
Discourse
Discourse I
Discourse III
II
Demonstrate CDA and CLA
55%
68%
84%
Use of Academic Discourse
60%
66%
76%
Revise
52%
67%
75%
Critique
68%
72%
77%
Research
60%
68%
75%
Table 2: Discourse III, Spring 2015, Percentage of Students Achieving 4 on Each Criterion
% Achieving a 4
Community Engagement
41%
Apply CDA and CLA
31%
Use of Academic Discourse
33%
Communicate in Multiple Genres
33%
Revise
27%
Critique
28%
Research
40%
Presentation Style
41%
Table 3
EPP: Scaled Total Scores and Skill Subscores for UMKC Seniors AY 2010-11 through AY 201415
2010-11
2011-12
2012-13
2013-14
2014-15
N=897
N=1,031
N=1,395
N=1,357
N=1,297
M
SD
M
SD
M
SD
M
SD
M
SD
Total Score 452.57 19.60 451.56 19.58 453.73 20.14 452.83 19.40 452.73 19.54
Critical
114.45 6.37 113.92 6.35 114.23 6.55 113.77 6.48 113.48 6.32
Thinking
120.70 6.13 120.49 6.14 120.74 6.74 120.43 6.71 120.55 6.82
Reading
115.25 4.50 115.11 4.57 116.04 4.95 115.99 4.76 115.89 4.89
Writing
Mathematics 115.02 6.18 115.03 6.29 115.16 6.03 115.08 6.05 115.25 6.00
4
Table 4
EPP: Comparative Scaled Scores for Senior Students at UMKC and Peer-Institutions –
2010-2015
UMKC Senior Students
Aggregate Peer Senior Students
N=6,305
N=5,132
Mean
SD
Mean
SD
Total Score*
452.73
19.54
455.30
6.00
Critical Thinking*
113.48
6.32
114.30
1.40
Reading
120.55
6.82
120.30
1.50
Writing*
115.89
4.89
116.20
1.20
Mathematics*
115.25
6.00
117.10
2.00
*p < 0.001
Table 4
EPP: Criterion-Referenced Scores of UMKC Senior Students - 2014-15
Proficient
Marginal
Not
Proficient
Critical Thinking
8%
27%
66%
Reading, Level 1
80%
10%
10%
Reading, Level 2
49%
23%
28%
Writing, Level 1
72%
23%
5%
Writing, Level 2
26%
40%
34%
Writing, Level 3
15%
27%
58%
Mathematics, Level 1
67%
19%
14%
Mathematics, Level 2
38%
28%
34%
Mathematics, Level 3
9%
21%
70%
Table 6
EPP: Percentage of UMKC Seniors Classified as Proficient on Criterion-Referenced Scores –
2010-11 – 2014-15
2010-11
2011-12
2012-13
2013-14
2014-15
Critical Thinking
14%
12%
9%
8%
8%
Reading, Level 1
80%
78%
80%
78%
80%
Reading, Level 2
55%
55%
53%
48%
49%
Writing, Level 1
72%
71%
75%
75%
72%
Writing, Level 2
26%
26%
28%
26%
26%
Writing, Level 3
11%
9%
18%
14%
15%
Math, Level 1
68%
69%
69%
66%
67%
Math, Level 2
44%
42%
39%
38%
38%
Math, Level 3
14%
14%
11%
9%
9%
5
Table 7: General Education Anchor and Focus Course Assessment Reports – 2013-14
(N=60)
Met:
Summative course grade
Summative artifact grade
SLOs
No data (faculty statement)
Mixed:
Summative course grade
Summative artifact grade
SLOs
No data (faculty statement)
Not Met:
Summative course grade
Summative artifact grade
SLOs
No data (faculty statement)
Scientific
Reasoning
and
Quantitative
Analysis
(N = 18)
11
8
1
2
4
Arts and
Humanities
(N = 18)
18
6
1
8
3
0
Culture
and
Diversity
(N = 6)
5
4
1
1
3
1
3
1
0
3
0
Human
Actions,
Values and
Ethics
(N = 18)
10
3
1
3
3
5
2
2
1
3
1
2
6