Eurocall 2012 Gothenburg, Sweden Jooyoung Lee & Volker Hegelheimer Iowa State University Automated Writing Evaluation tools provide both numerical scores and formative feedback ¡  Positive findings: ¡
§  Motivation (Grimes & Warshauer, 2006) §  Grammar (Chodorow et al., 2010) §  Rhetorical development (Cotos, 2011) ¡
Negative findings: §  Great focus on grammatical and mechanical aspects §  Losing sense of audience (CCCC, 2004) ¡  AWE is perceived more favorably when the machine feedback is complemented by human feedback (Chen & Cheng, 2008) ¡  My observation: Few classes entirely depending on AWE ¡  Then, one naturally arising question is? ¡  However, no study has explored a hybrid use of AWE and teacher feedback. ¡  This would be most interesting and practical question to ask (Ebyary & Windeatt, 2010). 1.
2.
3.
How are Criterion and teacher feedback similar or different from each other in terms of kinds of problems addressed, explicitness, and wording? How do students revise their paper based on each feedback? To what extent are revisions based on each feedback successful? ¡  Students’ reactions > Teachers’ actions ¡  Students’ actual revision behavior, not their perception or report ¡  Participants: 20 students from four academic writing classes offered at Iowa State University ¡  Student writing samples: Multiple drafts of one essay from each student (45 drafts in total) ¡  Collection: Accessed Criterion website or received a Word file from the instructors Paper submission policy Class A D1-­‐> Criterion D2-­‐> Peer review D3-­‐> Teacher D4-­‐> Teacher Class B & C D1-­‐> Criterion & Teacher D2-­‐> Criterion & Teacher Class D D1-­‐> Criterion D2-­‐> Criterion ¡  867 feedback/revisions = 294 (Criterion) + 418 (Teacher) + 155 (Self-­‐revision) ¡  Error categories: Criterion and teachers’ labeling ¡  Characteristics of feedback: Ferris (1997) ¡  Revision types and success: Ferris (1997) ¡  Based on the preliminary analysis of several writing samples, I refined the coding schemes. Criterion Teacher Self-­‐revision Article 22.4 Tense 12.9 Content/
development 18.7 Fragment S 13.9 Content/
development 12.2 Structure 17.4 Run-­‐on S 12.9 Expression 9.8 Word choice 15.5 Spelling 12.6 Word choice 9.3 Expression 14.8 Preposition 6.5 Structure 7.4 Conjunction 5.2 Repetition of W 5.4 Conjunction 7.2 Punctuation 4.5 Too many short S 5.4 Organization 6.0 Singular/plural 4.5 Ill-­‐formed V 4.4 Part of speech 5.0 Tense 4.5 S beginning with conjunction 2.7 Meaning 4.1 Organization 2.6 Proofread 2.0 Singular/plural 3.1 Article 1.9 Direct Indirect Indirect with Suggestions Criterion 14.3 44.2 41.5 Teacher 37.4 55.8 6.8 Direct: Instead of sending their children go abroad (“delete”) Indirect: I just finished the college entrance examination on the past July (“wrong preposition”) Indirect with suggestions: “Proofread the sentence to be sure that it has correct punctuation and that it has an independent clause with a complete subject and predicate.” •  Criterion ▪  Always in a complete sentence form ▪  Remember to capitalize the first word of each sentence. ▪  A lot of hedges ▪  This sentence may be a fragment or may have incorrect punctuation. •  Teacher ▪  Differences across teachers ▪  Very short ▪  Tense ▪  Practicing? ▪  Introduction: ok ▪  Both marginal and end comment Criterion Teacher Self No Change 50.5 16.3 0 Add Word 9.1 6.6 12.7 0 5.9 10.9 Delete Word 5.4 8.5 3.0 Delete Sentence 2.0 1.2 6.7 Change Word 14.5 43.6 42.4 Change Sentence 10.1 9.7 18.8 Transpose Word 0 0.9 1.8 Transpose Sentence 0 0.7 3.6 7.4 1.4 0 Add Sentence Disappear ¡  Successful revision (self-­‐revision) §  But my mother disagreed with my suggestion about go abroad. The only reason my mother disagreed with me was my mother didn’t want me to escape any difficult no matter in that time or future. §  I really envied them at that time, but my mother disagreed with my suggestion about going abroad. The only reason was she did not want me to escape from any difficults no matter in that time or future. ¡
Mixed effect §  Every person has him/her own difficult should to undertake. §  Criterion says “This verb may be incorrect” §  Every person has him/her own difficult should be undertaken. ¡
No change (self-­‐revision) §  work out a question §  work out a problem RQ 3. Success of Revision ¡  Unsuccessful revision §  I did very worse of them. §  Teacher says: “Something missing” §  Deleted the sentence Successful Mixed No discernable change Unsuccessful Criterion 69.0 2.4 9.9 18.7 Teacher 63.1 4.0 1.3 31.6 Self 69.7 1.3 9.7 19.4 ¡  Criterion/Teacher distinction is not the only variable influencing revision types and success of revision ¡  Other possible variables: Error type, explicitness of the feedback, individual different, instructional context… ¡  Inter-­‐rater reliability