COAS: Drexel University
waypoint
online assessment and structured peer review
Andrew J. McCann
visiting professor of english, drexel university
founder and president, subjective metrics, inc.
www.gowaypoint.com
ajmccann@subjectivemetrics.com
215.713.9393
05.11.2005
©2003-2005 Subjective Metrics, Inc.
1
Agenda
• Background
• Introduce waypoint
– Brief overview
• Drexel applications
• Live Demonstration
– Evaluate
– Manage
– Libraries
05.11.2005
©2003-2005 Subjective Metrics, Inc.
2
Problem: Feedback
• Handwritten and/or
manually typed
• “Lost”
• Little accountability
• Consistency issues
• No data
• Needed: not AI – but a
technological tool
05.11.2005
©2003-2005 Subjective Metrics, Inc.
3
Waypoint: Evaluation Tool
• Encourages pre-written feedback:
clearer, more detailed explanations
of key concepts
• Facilitates sharing of assessments
amongst instructors
• Quantifies evaluations by skill
• Archives all feedback
Web-based:
•
cross-platform
•
backed up
• zero maintenance
05.11.2005
©2003-2005 Subjective Metrics, Inc.
4
Waypoint Process
05.11.2005
©2003-2005 Subjective Metrics, Inc.
5
Develop Assessment
05.11.2005
©2003-2005 Subjective Metrics, Inc.
6
Develop Assessment
05.11.2005
©2003-2005 Subjective Metrics, Inc.
7
Receive Paper or Exam
(Hard copy shown here)
05.11.2005
©2003-2005 Subjective Metrics, Inc.
8
Receive Paper or Exam
(Hard copy shown here)
05.11.2005
©2003-2005 Subjective Metrics, Inc.
9
Evaluate Against Skills
05.11.2005
©2003-2005 Subjective Metrics, Inc.
10
Evaluate Against Skills
05.11.2005
©2003-2005 Subjective Metrics, Inc.
11
Respond to Student
05.11.2005
©2003-2005 Subjective Metrics, Inc.
12
Respond to Student
05.11.2005
©2003-2005 Subjective Metrics, Inc.
13
Snapshot Analysis
05.11.2005
©2003-2005 Subjective Metrics, Inc.
14
Snapshot Analysis
05.11.2005
©2003-2005 Subjective Metrics, Inc.
15
Longitudinal Cohort Analysis
Organizational Behavior: Research Competency
1 to 4 scale; 110 students evaluated
70
60
# Teams
50
Feb '04
Mar '04
Jun '04
40
30
20
10
0
1
05.11.2005
2
3
©2003-2005 Subjective Metrics, Inc.
4
16
Process Summary
Organizational Behavior: Research Competency
1 to 4 scale; 110 students evaluated
70
60
# Teams
50
Feb '04
Mar '04
Jun '04
40
30
20
10
0
1
05.11.2005
2
3
4
©2003-2005 Subjective Metrics, Inc.
17
Drexel Applications
• tDec Humanities (Dr. Valarie Arms)
– Evaluation of all major writing assignments (600 students)
• Writing Center (Harriet Millan)
– Customized assessments for WITs
• College of Engineering (Kevin Scoles & Adam Fontecchio)
– Evaluation of lab reports with WITs and TAs
• College of Business (Frank Linnehan)
– Structured response to student writing & accreditation data
generation
• Engineering Management (Mike Scheuerman)
– Peer review
05.11.2005
©2003-2005 Subjective Metrics, Inc.
18
Demonstration
• www.gowaypoint.com
• Evaluate
– “Final Report”
• Peer Review
– “Final Report”
• Manage
– Quantification
– Sorting
– Data analysis
05.11.2005
©2003-2005 Subjective Metrics, Inc.
19
Additional Features
• Self-assessment
• Collaborative assessment
– Multiple instructors can contribute to
evaluation
05.11.2005
©2003-2005 Subjective Metrics, Inc.
20
Questions and Discussion
05.11.2005
©2003-2005 Subjective Metrics, Inc.
21
Testimonials
• “I've been doing this for twenty years and have my own system
of grading and commentary…and found little need to improve.
And then Waypoint came along. Somehow--and I really can't
even explain it--but my grading time has been cut down by more
than half and my students are actually thanking me for the indepth commentary.”
– Professor Ken Bingham
• “Usually I have trouble criticizing a peer’s paper if I’m not given
certain criteria to judge. During this peer review, I was actually
focused and excited about judging a peer’s paper.”
– Erin Williams, COE 2008
• “If this peer review program was available in high school, I would
have probably done a lot better in English.”
– Ed Itaas, COE 2008
05.11.2005
©2003-2005 Subjective Metrics, Inc.
22
Testimonials
• “An innovative teacher-friendly, student-friendly,
efficient approach to grading writing—the most
creative and time-saving method for evaluating
writing I have ever found or used.”
– Gayle, a high school English teacher with 35
years’ experience
• “I don’t know how I ran a writing program for three
years without it.”
– Harriet Millan, Director of the University Writing
Program
05.11.2005
©2003-2005 Subjective Metrics, Inc.
23
Quantitative Data
• Anonymous survey of Peer Review process
– 114 freshman engineers: 51 use waypoint, 63 the
‘old’ method
1-5 Scale (1 – strongly disagree, 5 – strongly
agree)
wp (N=51)
Mean
‘old’
(N=63)
Mean
P value
(t-test, 95%
CL)
1. I received clear and helpful criticism of
my draft.
3.90
3.52
p<0.025
2. Evaluating other students’ papers
helped me better understand the
assignment and the play.
3.86
3.48
p<0.02
3. I made significant changes to my first
draft based on peer review feedback.
3.81
3.25
p<0.005
4. I made significant changes to my first
draft independent of peer review
feedback.
3.49
3.43
p=0.38
05.11.2005
©2003-2005 Subjective Metrics, Inc.
24