SIZING UP STANDARDIZED ASSESSMENTS:
BREAKING DOWN THE ETS PROFICIENCY
PROFILE, MAJOR FIELD TESTS, AND THE
ROOWRITER
By Nathan Lindsay
and Dan Stroud
January 30, 2013
ASSESSMENT AT UMKC
Consists of a multifaceted approach:
 Formative
 Direct
and Summative Assessments
and Indirect Assessments
Direct assessments include embedded
assessments (in the classroom), and
standardized assessments

UMKC STANDARDIZED TESTS
OVERVIEW
For many years, undergraduate students at UMKC
have participated in the following standardized
tests:

The ETS-Proficiency Profile test that measures
General Education learning outcomes

Major Field Tests (MFTs) that measure learning
outcomes for students in their disciplines.

The WEPT (soon to be the RooWriter) that measures
students’ writing ability
For the ETS tests, only this past year did we
purchase a reporting license that allows us to closely
analyze the data
QUESTION
 What
are some of the pros and cons
of standardized tests?
WHY STANDARDIZED TESTS MIGHT BE
USEFUL ALONG SIDE IN-HOUSE ASSESSMENTS
o
o
o
Standardized tests allow comparisons to
both national and peer data (which are
requested by external constituents).
There are benefits in the use of
longitudinal data.
ETS test design includes strong:
Validity
2) Reliability
3) Fairness/Equity
4) Representation
1)
HOW ETS DEVELOPS TEST QUESTIONS
Below is the process by which tests and test
questions are developed for future ETS Proficiency
Profile tests and MFTs:
http://www.ets.org/s/understanding_testing/flash/h
ow_ets_creates_test_questions.html
TEST SCORING
Two Types of Test Scores
Norm-referenced (scaled) scores
- These scores pull their meaning from comparing
the scores between and individual student or group of
students with those of either the same individual or
group or another at different points of time. The
majority of test results are of this nature.

Criterion-referenced (proficiency classified) scores
- These pertain to the level of proficiency students
obtained within a particular skill set. This type of score
stands on its own – with no need to compare to another
group, though they can still use specific comparisons as
well.

Sampling the Proficiency Profile
Sampling the Proficiency Profile
Sampling the Proficiency Profile
ETS – PROFICIENCY PROFILE
(IN 2011-2012, N = 1448 STUDENTS)


Test given to all students after they have completed
70 credit hours. Completion is a requirement of
graduation.
Scores were above benchmark peers for all four areas:
Reading
 Critical Thinking
 Writing
 Mathematics



Scores on individual items exceeded the national
average 51 of 53 times.
Higher scores may be due to the fact that most
UMKC students take the ETS-PP right before they
graduate (which is not always true at other
institutions)
MAJOR FIELD TESTS (MFTS)

UMKC students take the MFTs in the following
areas:








Biology
Business
Psychology
Political Science
Chemistry
Computer Science
Mathematics
Physics
20 students have to take the test to have sufficient
comparative data, so results are only available for some
of these disciplines.
TWO REPORTS AVAILABLE FOR
MAJOR FIELD TESTS


The Custom Comparative Data Report indicates
how UMKC’s students' skills and knowledge
compare with the skills and knowledge of
students at similar institutions.
In the Item Information Report, the items are
prioritized first by content, and subsequently by
sub-content to make comparisons easier for
grouping. Scores compared in this report are
referenced to the overall national percentage for
each item.
MAJOR FIELD TESTS RESULTS AT UMKC




Biology (N = 87 Students) – UMKC scored above
average on 8 of 9 Assessment Indicators.
Business (N = 192 Students) – UMKC scored
above average on 4 of 9 Assessment Indicators.
Psychology (N = 81 Students) – UMKC scored
above average on 2 of 6 Assessment Indicators.
Political Science (N = 20 Students) – UMKC
scored above average on 1 of 3 Assessment
Indicators.
DISSEMINATING THE ETS RESULTS




The General Education scores have been reviewed
by the General Education Committee, the
University Assessment Committee, and the
Assessment Academy Team.
These MFT reports have been sent to all relevant
departments, which have been encouraged to
discuss how they can enhance their curriculum to
address the areas of weakness within each test.
A summary of our standardized tests will be shared
with the Higher Learning Commission.
We will collect data during the 2012-2013 academic
year to determine any changes over time.
DEPARTMENTAL DIALOGUE:
CONVERTING THE DATA TO ACTION

Use the data to stimulate conversation in the department
about the curriculum and pedagogy surrounding these
topics. Address the following questions:
1.
2.
3.
4.
5.
6.
7.
How do the content areas of the test relate to our degree’s
learning outcomes? Should test items be added?
Are there any areas of the curriculum that could/should be
emphasized more strongly? [curriculum mapping]
Do the results have face validity? How do the results
compare to our findings from course-based assessments and
our program/degree assessments?
How did different student groups/demographics score on the
test? Should more sub-groups for analysis be created?
What are other ways that we could address/assess the areas
that were lower? Should we develop our own test?
What trends do the scores show longitudinally?
How could we increase student motivation for the test?
HOW TO CUSTOMIZE MFT
REPORTING GROUPS



Cohorts
- Determine which students to put in what
groups as you design report emphasis
Subgroups
- You can create up to 16 subgroups
- For each subgroup, you can design two questions
with up to 8 possible responses.
There must be a minimum of 5 students in each group,
cohort, or subgroup. Minimums for reporting
cohorts or subgroups must add up together to 20
as a minimum to report.
DISCUSSION
Are there other ways that your unit/department could
develop test questions that would evaluate the
learning outcomes for your degree(s)?
Questions to consider:
o
What’s the point/purpose of the test?
o
What are the intended uses of this test?
o
What aspects of each unit or department do we wish to
measure?
o
What scores will be reported?
o
Is the test in question statistically reliable, valid, and
fair?
STUDENT MOTIVATION FOR
STANDARDIZED ASSESSMENTS



Key Issue
- High stakes for institutions
- Low stakes for students
Incentivizing is two tiered
- Get students to take the test
- Get students to do their best
Best Practice—Connect to Academic Outcomes
- Participation grade for a class
- Students who attain a score of _______
or higher receives _______?
DISCUSSION
What incentives could be used to help students to
take standardized testing more seriously?
OTHER WAYS TO ENCOURAGE
MEANINGFUL PARTICIPATION

Inform students well in advance

Include the MFT in the syllabus

Send letter or email from Dean/
Department Chair

Develop faculty understanding of the purposes of the test

Create interdepartmental competition

Develop an Assessment Day (Long range goal)
WEPT


This is a Pass/Fail test, and approximately 5% of
students have failed the test the last few semesters
Each test is rated on 4 areas:
Point/Thesis/Idea – Evaluated on the clarity of purpose
throughout the essay.
 Development/Support/Evidence – Evaluated on the
relevance and convincingness of the proof offered in
support of the thesis.
 Organization/Logic/Coherence – Evaluated on the ease of
understanding and logical arrangement of the essay.
 Usage/Mechanics/Documentation – Evaluated on the ease
of reading and grammatical and style correctness


The test is graded on a four-point scale

The average scores for area one (i.e., “Point/Thesis/Idea”)
were the highest at 3.10, up from the 3.07 in the previous
semester. The remaining three scores average fell near
2.82 (2.82, 2.83, and 2.80, respectively).
THE ROOWRITER
To replace the WEPT for incoming freshmen in
Fall 2013
 An online diagnostic/formative test with many
benefits for students, faculty, and the institution
(see attached handout)
 The University Writing and Reading Board is
currently working on finalizing all of the logistics
for the test
 Please refer to the RooWriter website for an
overview:


http://dev.umkc.edu/roowriter/index.cfm
QUESTIONS
 Are
there other standardized tests
that UMKC should be considering?
 What
other questions about
standardized tests do you still have?