Department of Humanities
College of Sciences and Liberal Arts
Writing Program Assessment at
New Jersey Institute of
Technology
Carol Siri Johnson
Associate Professor and Chair
Outline of Presentation
• Personal Introduction
• New Jersey Institute of Technology
• ABET
• Writing / Technical Writing Assessment
• Using Online Portfolios
• Using Analytic Scoring
• Examples of Data Analysis Results
Personal Introduction
• Mount Holyoke College
• English Ph.D., but few jobs
• New – Technical Writing Field
• Tenure-track Position at NJIT
• Chair, Research EA and TC History
My Assessment History
• Knew nothing about it
• Disliked standardized tests
• Was unconvinced of its value
• Then I saw a best-paper reading
• The Power of Assessment
Assessment at NJIT
• Department of Humanities active in
assessment for ~20 years
• Previous Chair, NJCBSPT
• Norbert Elliot and ETS
• Course / Program Assessment
• Analytic Online Portfolio Assessment
On a Scale – Norbert Elliot
What is Assessment?
● Giving a student a grade?
● End-of-semester course evaluations?
● Faculty review of portfolios?
● Student surveys?
● The chair’s opinion?
What is Assessment?
● For Teachers, GRADES
● For Departments, COMMON EXAMS
● For Administration, COURSE
EVALUATIONS
● For Learning, OUTCOMES – stating and
reaching curricular goals in Courses and
Programs
Assessment Can Be:
• Program assessment
• Curricular assessment
• Method of automatic updating
• Method of teaching collaboration
• Gathering statistics for multiple
purposes.
ABET 2000- Accreditation Board
for Engineering and Technology
ABET’s 8 General Criteria:
#4 Continuous Improvement
“The results of these evaluations must be
systematically utilized as input for the continuous
improvement of the program.”
Outcomes Assessment Cycle
NJIT Core Curriculum – GURs
(General University Requirements)
● English Composition – 6 credits
● Cultural History – 6 credits
● Humanities Electives – 6 credits
● Senior Seminar – 3 credits
GURs Outcomes Assessment
● English Composition – every semester
● Cultural History – every other spring
● Humanities Electives – every other fall
● Senior Seminar – every semester
Assessment Basics – Validity
• The issue of “validity” means we are
measuring what we want to measure;
the test is appropriate to the subject
• The problem with writing assessment
has been using multiple choice tests only
– they alone are not valid as a measure
of writing
• Multiple measures are often the best
Assessment Basics –
“Associative” Validity
Correlating Multiple Inputs:
• Machine-scored essay scores
• Portfolio scores
• Grade in course
• Student GPA
Assessment Basics –
Reliability
Can we be sure that the measurement, if taken
again, would yield the same results?
• Inter-reader Reliability (scored by two
separate readers, discrepancies resolved by a
third)
• Content Reliability (portfolio contains work
from entire semester)
Two Important Steps:
• Decide on and define goals or
criteria
• Develop method to assess
outcomes (goals/criteria)
Large Scale Writing Assessment
Initiatives
● 1950s ETS – Holistic Scoring
● 1970s California (Edward White)
● 1980s New York (CUNY)
● 1990s New Jersey (NJCBSPT)
● 2000s NJIT – Program Assessment
Example: Technical Communication
Analytic Online Portfolio Assessment
• Established Criteria by “Online Delphi” –
email exchanges by faculty
• 2004 – Failed effort (network down)
• 2005-2009 – Successful Analytic Online
Portfolio Assessment, every semester
• 2009-Present – once per year
Technical Communication Cycle
Using Online Portfolios
Procedure
1. Have Students Make Online Portfolios with
Examples of the Main Assignments
2. Collect URLs
3. Select Random Sampling (~60-100)
4. All Instructors Meet
5. ~1 Hour Calibration with Sample Portfolios where
instructors discuss scoring processes
6. ~3 Hours Scoring
7. Assessment Director Conducts Necessary
Adjudications
8. Enter Scores into Excel or SPSS Dataset
Old
Humanities
Rubric
Old Score Sheets
Revised Score Sheet
Sample Data Collection
A Community of Assessment
• Shared Instructional Content
• Self-Evaluation and Norming
• Curricular Goals Evaluation
• Continual Improvement (ABET)
• Interesting Statistics