Implementing a Standardized Performance Test of Writing
The Swimmers
Raymond Moy – University Director of
Assessment. 2 ½ years at CUNY, 25 years at
the United Nations in language proficiency
testing and organizational development.
Melissa Uber – University Director of
Testing. 23 years at CUNY – 12 at Hunter
College and 5 at New York City College of
Technology.
CUNY: Who are we?
The University is comprised of 23 institutions:
11 Senior Colleges
6 Community Colleges (a 7th opens Fall 2013)
An Honors College, the Graduate School and University
Center, the Graduate School of Journalism, the School of
Law, the School of Professional Studies, and the School of
Public Health.
Fall 2010 undergraduate enrollment 228,484:
58.6 % are female
28.4 % are 25 years or older
43.6 % have a native language other than English
57.3 % are Pell Grant Recipients
44.2 % are First Generation in College
College Readiness: Points of View
Open admissions
High schools want more liberal standards
College faculty want tighter standards
Tying budgets to enrollments
Administration wants more liberal standards
Governmental funding bodies want tighter
standards
Raising the quality of academic programs
Requires clear understandable standards
CUNY Assessment Testing: A History
CUNY’s mission to make quality higher education
accessible to all New Yorkers
Admission standards liberalized to admit all
students with a NY state high school diploma in
1970’s
CUNY Board of Trustees mandates evidence of
college proficiency by 60th credit beginning in 1976
Political concern about declining academic
standards and ineffective use of public funds.
A Framework Emerges:
Annual audits to ensure college faculty
were grading to the rubric
Certification training to qualify faculty
as scorers
Establishment of borough grading sites
to ensure standardization of grading
University Testing Database and
Scheduling System deployed
Placement vs. Exit
Single score did not provide adequate
feedback to developmental students or
faculty
Test task was artificial and did not
resemble the writing expected in college
Students wanted instructors to teach to
the test and test-taking techniques
Test Development Phases
From the Standards for Educational and Psychological
Testing, AERA, APA, & NCME (1999):
1.
2.
3.
4.
Delineation of the purposes(s) of the test and the scope
of the construct or the extent of the domain to be
measured;
Development and evaluation of the test specifications;
Development, field testing, evaluation, and selection of
the items and scoring guides and procedures;
Assembly and evaluation of the test for operational use.
Requisite Skills for Freshman Composition
Reading
Read with understanding at minimum level of the class
Summarizing
Summarize text
Summarize and use a text
Summarizing: recognizing viewpoint, thesis
Critical thinking
Responding to content
Ability to compose a thesis statement that answers the question
State ideas, describe experience and reflect on it.
Critical thinking: reading, understanding, responding
Distinguishing fact from opinion
Supporting ideas
Support with evidence from a text
Organization
Overall organization (beginning, middle, end)
Organizing: beginning, middle, end
Move from general to specific and reverse
Verbal accuracy
Edited English
Reflect on one’s own writing: revise and edit
Measurement Construct
Understand and critically evaluate the ideas in a
text
Use ideas in the text to support and develop
student’s own ideas
Organize a focused response that demonstrates
a clear and coherent progression of ideas
Choose sentence structures and vocabulary for
clarity and precision
Adhere to the conventions of English grammar,
usage, and mechanics
Performance Rubric
Performance Rubric
CATW Score Scale
Test task design
Assignment
Begin by reading the passage below.
Hype
Advertisements are the most prevalent and toxic of the mental pollutants. From the moment your alarm
sounds in the morning to the wee hours of late-night TV, commercial pollution floods your brain at the
rate of about three thousand marketing messages per day. Every day an estimated 12 billion display ads, 3
million radio commercials, and more than 200,000 TV commercials are dumped into North America’s
collective unconscious. The increase in commercial advertising has happened so steadily and relentlessly
that we haven’t quite woken up to the absurdity of it all. No longer are ads confined to the usual places:
buses, billboards, stadiums. Anywhere your eyes can possibly come to rest is now a place that, in
corporate America’s view, can and ought to be filled with a logo or product message.
You fill your car with gas, and there’s an ad on the nozzle. You wait for your bank machine to spit out
money and an ad scrolls by in the little window. You drive through the countryside and the view of the
wheat fields is broken at intervals by enormous billboards. Your kids watch Pepsi and Snickers ads in the
classroom. A company called VideoCarte installs interactive screens on supermarket carts so that you can
see ads while you shop. (A company executive calls the little monitors “the most powerful
micromarketing medium available today.”) There is nowhere to run. No one is exempt and no one will be
spared. In the silent moments of my life, I often used to hear Beethoven’s Ninth Symphony play in my
head. Now I hear that kid singing the Oscar Meyer wiener song.
Excerpted from Kalle Lasn. “Hype,” Signs of Life in the USA: Readings on Popular Culture for Writers, 4th
ed. (Sonia Maasik & Jack Solomon, eds.) Boston: Bedford/St. Martin’s, 2003. 217-220.
Writing prompt
Writing Directions
Read the passage above and write an essay responding to the
ideas it presents. In your essay, be sure to summarize the
passage in your own words, stating the author’s most
important ideas. Develop your essay by identifying one idea
in the passage that you feel is especially significant, and
explain its significance. Support your claims with evidence or
examples drawn from what you have read, learned in school,
and/or personally experienced.
Remember to review your essay and make any changes or
corrections that are needed to help your reader follow your
thinking. You will have 90 minutes to complete your essay.
Pilot Test Questions
Do forms make a difference?
Does text length make a difference?
Are ESL students disadvantaged?
Can readers score an analytic rubric as
fast as a holistic rubric?
Can readers score an analytic rubric
reliably?
Results of the Pilot Test
We had a 90% perfect or adjacent scoring agreement between raters
on all test forms.
For passage selection, length and readability are not as important as
well-written passages with clearly stated main ideas.
Students were comfortable with topic selections and the amount of
time allotted
Most important dimensions in determining level of writing are
ability to organize a focused response and to choose vocabulary and
sentence structures effectively
Analytic scoring only takes slightly longer than holistic and can be
just as accurate
Field Testing
Development of parallel forms
Text selection specifications
Editor to manage the process
Faculty contributions/review
Testing with CUNY developmental students,
ESL, and Freshman comp
Standardizing grading
Reader training and certification
Anchor papers
Practice papers
Text summary and anchor paper articulations
Plot of Test Form Means
Distribution of Rater Difference Scores
Setting the Proficiency Standard
Rubric performance level – 56 (which
represents a 7 on each dimention)
Score that maximizes the distinction
between developmental and Freshman
Comp students – 55
Panel of expert judges classifying borderline
papers - 57
Implementation: the Logistics
Data systems modifications
Copyright clearance
Certification training for faculty
Informational sessions for all stake-
holders
Test prep materials
Test administration manual
Student score reports/rosters
Student Score Report
Class Roster Scores
Impact of the CATW
Faculty buy in for the test
Established a definition of college level writing – what
is to be learned and what is to be taught
High schools can be more focused on preparing their
students
Faculty have a unified goal for their writing courses
Transferability of the proficiency requirement
Clearer placement of students in developmental
courses
Identification of effective teaching practice
CATW conference
Better preparation for academic performance
CUNY Testing Website:
More information regarding the CUNY
Testing Program and the CATW can be
found at:
www.cuny.edu/testing
0
