Nathan Lindsay & Dan Stroud
8-1-2012
ETS Proficiency Profile Test Report
The Education Testing Service Proficiency Profile (ETS-PP) is a standardized assessment that
focuses on student learning outcomes for undergraduates as they enter major field course work.
The instrument is designed to provide evidence regarding the institution’s effectiveness in
helping students achieve the General Education learning outcomes. All UMKC undergraduates
are required to take the test before they graduate. During this past academic year, there were a
total of 1,448 students who took the exam from August 2011 through July 2012.
Two sets of analyses were run on the UMKC data from the ETS-PP: the Custom Comparative
Data Report and the Item Analysis Report. The Custom Comparative Data Report indicates how
UMKC’s students' skills and knowledge compare with the skills and knowledge of students at
similar institutions. The report generates descriptive statistics (over a three year period) based on
a reference group of 10 or more comparable institutions or programs of interest which were
selected based on similar institutional characteristics. The anonymity of each institution's
performance is maintained by reporting only the aggregate performance of the selected reference
group. The number of students tested and sampling procedures vary from one institution to
another. Therefore, it is impossible to verify that the students tested at each institution are
representative of all the institution's students in that program.
Table 1: List of Benchmarking Institutions
Number
of
School Name
Students
Ball State University, IN
307
Indiana State University
477
Florida International University, FL
2,479
Lamar University, TX
672
Missouri State University, MO
14,994
Southeast Missouri State University, MO
11,226
University of Akron, The, OH
515
University of Alabama at Birmingham, AL
1,700
University of Cincinnati, OH
321
University of South Florida – St. Petersburg, FL
27
Total
33,359
Table 1 outlines the ten institutions
chosen for the comparative review
of peer institutions for the ETS-PP.
Among this group are several
institutions with which UMKC
regularly benchmarks its data. If
there are other institutions of higher
learning that would be beneficial to
exchange for those in this
comparison, the report can easily be
adapted. However, please note that
all of our regular peer institutions do
not participate in this testing process,
and thus some information may not
be available to us for review.
The numbers presented in Table 2 (below) include the mean scores, first for UMKC and then for
the peer institutions listed above. These sub-scores and assessment indicators are both composite
scores that are derived from the scores of many items. The numbers indicate, at least from this
1
Nathan Lindsay & Dan Stroud
8-1-2012
comparative sample of similar institutions, that our students are learning and ultimately
performing ahead of the curve.
Skill Dimension
Reading, Level 1
Reading, Level 2
Table 2 - Custom Comparative Data Report
Proficient
Marginal
Comparative
Comparative
UMKC
Data
UMKC
Data
77%
59%
15%
19%
53%
34%
16%
16%
Not Proficient
Comparative
UMKC
Data
9%
23%
31%
50%
Critical Thinking
11%
6%
21%
16%
68%
77%
Writing, Level 1
Writing, Level 2
Writing, Level 3
70%
25%
9%
57%
18%
7%
21%
38%
34%
26%
33%
24%
9%
37%
58%
17%
49%
69%
Mathematics, Level 1
Mathematics, Level 2
Mathematics, Level 3
65%
38%
12%
56%
29%
8%
18%
25%
21%
23%
24%
16%
17%
37%
68%
21%
48%
75%
The total test consists of 108 assessable items that are split over three abbreviated tests. As
outlined below in the Item Information Report, the items are prioritized first by skill area, and
subsequently by proficiency level to make comparisons easier for grouping. The proficiency
level in Table 3 refers to the difficulty level of the test questions. Scores compared in this report
are referenced to the overall national percentage for each item. Items that are lower than the
national average are highlighted.
In looking over the Table 3 data below, the overall performance at the institutional level is very
positive. Reading and critical thinking were especially impressive, as every question and
indicator tracked above the national average. The writing and mathematics areas proved nearly
as strong, with only one indicator that fell below the national average in each area.
We recommend that these data be used to stimulate dialogue across campus about the curriculum
and pedagogy surrounding these topics. Faculty should reflect on whether these scores are
congruent with the skills and knowledge that students demonstrate in the classroom, as well as
the skills that they struggle to demonstrate. Faculty might also ask whether they are teaching the
“type of content” listed in the first column of Table 3. We plan to meet with faculty from the
General Education Committee, the University Assessment Committee, and the Higher Learning
Commission Assessment Academy team to discuss these scores. Based on these conversations,
we encourage the development of an action plan to respond to the findings in this report.
2
Nathan Lindsay & Dan Stroud
8-1-2012
Table 3 - Item Information Report (n = 1,448)
Skill Area
Average
Proficiency Level
Percent Correct
Institution
Percent
Correct
National
Reading
I
75.57
68.74
2
Reading
I
77.65
68.45
Recognize a valid inference
5
Reading
II
49.76
44
Discern purpose of a reference
2
Reading
II
70.95
67.65
Synthesize material
2
Reading
II
66.75
56.65
Discern main idea
2
Reading
II
68.6
58.5
Discern primary purpose
3
II
68.07
58.8
Recognize flaws in argument
2
III
52.3
47.15
Extrapolate from known facts
1
III
48.5
42.9
Recognize an assumption
5
III
69.94
62.02
Evaluate hypotheses
3
III
61.33
52.73
Determine relevance of information
10
III
52.01
46.09
Evaluate an argument
1
III
23.1
21.2
Evaluate data for consistency
2
III
50.2
41.25
Evaluate explanations
1
III
57.3
47.8
Recognize salient features
2
Reading
Critical
Thinking
Critical
Thinking
Critical
Thinking
Critical
Thinking
Critical
Thinking
Critical
Thinking
Critical
Thinking
Critical
Thinking
Critical
Thinking
III
67
63.75
Recognize agreement
6
Writing
I, II^
66.6
64.32
Order sentences
2
Writing
I
71.8
64.2
Recognize appropriate transitions
1
Writing
I
72.3
74.3
Recognize incorrect punctuation
1
Writing
I
91.5
87.2
Recognize incorrect construction
1
Writing
III
47.7
46.2
Type of Content
Number of
Questions
Discern facts from a passage
11
Meaning in context
3
Nathan Lindsay & Dan Stroud
8-1-2012
Incorrect word choice
1
Writing
I
61.5
54
Recast an existing sentence
5
Writing
II
80.98
77.74
Combine simple clauses
1
Writing
II
67.3
62.2
Recognize correct construction
7
Writing
III
56.79
51.4
Incorporate new material in paragraph
1
Writing
II
67.8
55.5
Recognize most effective revision
1
Writing
III
51.9
52
Arithmetic word problem - profit
1
Mathematics
I
77.9
77.3
Algebraic expression
1
Mathematics
I
75.6
72.1
Word problem - algebraic expression
1
Mathematics
III
11.5
7.6
Arithmetic word problem
1
Mathematics
I
82.5
73.9
Arithmetic word problem - rates
1
Mathematics
I
83.3
77.2
Arithmetic word problem - work units
1
Mathematics
I
94.2
93.8
Geometric word problem - lengths
1
Mathematics
I
76.7
68.4
Data Interpretation - percents
1
Mathematics
I
53.1
47.2
Translation to algebraic expression
2
Mathematics
II
69.65
62.85
Simplify algebraic expression
1
Mathematics
II
85.5
81
Arithmetic word problem - embedded ratios
1
Mathematics
II
59.4
53.8
Arithmetic word problem - complex wording
1
Mathematics
II
80.8
76.4
Arithmetic word problem - weighted mean
1
Mathematics
II
58.7
57.4
Draw conclusion from algebraic expression
1
Mathematics
II
31
29.5
Solution is algebraic expression
1
Mathematics
III
15
10.5
Percent change - ratio and proportion
1
Mathematics
III
31.6
21.2
Data interpretation - percent increase
1
Mathematics
III
51.4
36.5
Arithmetic word problem - ratios,
approximation
1
Mathematics
II
54.8
54.3
Exponential growth
1
Mathematics
III
70.2
64.3
Data Interpretation - range
1
Mathematics
I
70
66.7
4
Nathan Lindsay & Dan Stroud
8-1-2012
Data Interpretation - compare two charts
1
Mathematics
I
35
26.8
Data Interpretation - percent change
1
Mathematics
III
42.7
41.4
Data Interpretation - rate of change
1
Mathematics
III
78.3
73
Percent decrease
1
Mathematics
III
64
50.6
Understanding of properties of integers
1
Mathematics
III
60.6
54.9
Measurement conversion
1
Mathematics
III
68.8
66.8
^ Questions split between levels I (4) and II (2)
5