Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis Dr Richard Parsons Centre for Learning and Teaching University of Dundee r.parsons@dundee.ac.uk Self, Peer and Tutor Assessment of Text Online. Design, Delivery and Analysis • Overall position of CAA at University of Dundee • Why self and peer assessment? • Development story • Key features – Academic author sets exercise online – Students use web, email and text messaging – Robust reliable, secure – Checks for copying • Demonstration – create an exercise, live • Using the system – key points – Good question design – Involve the students – Teach the students • Student activities and responses • Marking moderation and plagiarism • The future University of Dundee Learning Technology Systems • Blackboard VLE (5.5 Unix, upgrading to Enterprise 6.0 Unix) • QuestionMark Perception 3.4, dual server system – 1. Low stakes/formative server coupled to Blackboard (Oracle Unix database) – 2. High stakes/summative server runs stand alone (Oracle Unix database) • Provide full support for CAA including a “CAA Online” staff education course, initial authoring service • Have Policy & Procedure documents for CAA (based on BS 7988), (also for VLE and eLearning strategies) • Provide support for a range of additional services including online evaluation forms, plagiarism detection service, online staff eLearning tuition, publicise innovative solutions for supporting and enhancing learning (e.g Self and Peer Assessment Program) University of Dundee MLE P3 User exists 3 weeks before appointment Completed daily e-Directory Create user name and email process LDAP authentication Remove user process Student Management System Student exists Staff details also stored in SMS Module enrolments of students Module assessment mark storage LDAP client? LDAP Novell Fileservers Student Desktop VLE Integrated Systems GroupWise Web Email QMP formative server CAA-01 Module enrolment Bb client Mark return Blackboard VLE – ELS 6.0 Hardware 2 * 480s Version 6.0.8 Users created from SITS SMS system Module enrolment Staff assignments from web system LDAP client Module assessment mark storage Consultancy visit 12 May – 16 May Training teaching staff Staff Assignment Record Secure web access QMP Exam server CAA-02 Special exam enrolment Mark return Library Catalogue access Resource management system LDAP client? Richard Parsons, CLT LEU Why Self and Peer Assessment? • Assessment is one of the keys to learning • Being able to assess your own work is a vital skill • Developing peer assessment skills is important for future careers • By carrying out assessments, students learn about the assessment process • Shifts tutor work from post to pre-assignment • Web-based and innovative, so can interest the students Development Story • First a system to replace some lectures when I was overseas • First self-assessment only, but peer assessment seen as important, results emailed and texted to students • Developed into a system that allowed online authoring • Added the ability to tutor mark as well as self & peer assessment • Made more robust based on usage – Remove the back button – Disable right click – Provide more guidance to staff and student users • Added a check for copying and plagiarism • Tidied up code to W3C compliance Online Self & Peer Assessment of Text (Outline of the process) Instructor designs question, sets word limits and details criteria and scoring ranges All files created, URLs provided to instructor Question provided, answer researched and composed 1. Student submits their text answer (may copy and paste) 2. Self assessment completed by student using criteria offered Peer 1 assessment emailed anonymously to student author Peer 1 assessment emailed anonymously to student author 3. Student completes an assessment of one of their peers 4. Student completes an assessment of the second of their peers 5. Summary of marks presented. Student thanked for completing exercise, brief feedback completed Tutor can mark a selection of answers and moderate and collate results Final marks are checked and transferred to register Technical Details HTML pages for input Author completes webform Sets question & marking criteria Perl program creates And sets file protections All operates on a Unix web server, with hyperlinks as required to Blackboard Perl files to process input User interaction HTML pages for output Perl file to process output Author access to results Demonstration of the Program Create a NEW exercise, live and interactive? www.dundee.ac.uk/learning/ilt/ Using the System – Key Points • Good question design • Involve the students • Teach the students Student Instructions – Online Self and Peer Assessment Exercise Week containing Tuesday 11 March Receive the first assignment, question and instructions Research the first topic, craft your first text answer Mark your first answer using the criteria provided and mark two peer’s answers Receive the second assignment and instructions Tutor marking of exercise 1 Marks sent to student author Week containing Tuesday 15 April Research the second topic, craft your second text answer Tutor marking of exercise 2 Mark your second answer using the criteria provided and mark two peer’s answers Complete a short evaluation form where your opinion of the exercise will be sought. Summary marks returned via VLE – Self, peer and tutor marks compared. Statistics provided overall to judge the success of the exercise Dr Richard Parsons r.parsons@dundee.ac.uk Student Performance Environmental science exercise - Write a short piece of text that describes the major scientific reasons why global warming is almost certainly occurring at present. The total length of the text must be between 200 to 300 words. • Students performed well, range of marks awarded and obtained. Reasonable correlation between self, peer and tutor marking • Students recognised that they were learning about assessment. Positive feedback received • One of those exercises where you see the students learning Criterion vs Norm-based Marking 100 Self / Peer / Tutor mean 80 60 40 20 0 0 20 40 60 80 Expert (Norm-based reference) 100 Student comments demonstrate increased understanding “A good computer program that more classes should use” “A great learning tool for the students. Enjoyable too, better than the labs” “Useful for understanding what markers look for in a piece of work” “It was interesting to see how the marking process takes place, and how you can have an essay that in its own right is well written but does not contain the points needed by the marker” “Assessing others gives me comparison to my own” “Good to see how we can improve our work by our mistakes” “Useful for the ability to receive results via text message” “It was good to be able to compare your own work with others and offered an insight into the basic processes staff must use” “Makes you realise that you need more depth in your answer and to read the question” “It is quite funky” Demonstration of plagiarism/copying/ collusion detection A check for copying is now embedded within the program http://www.personal.dundee.ac.uk/~rparsons/leu/zz02087process.htm Operates in the following manner: Recursive check for 6 word sets, ignoring most punctuation Tutor can insert text from web pages, textbooks and lecture notes as required No service is used to store the student’s work, so no complicated legal issues Will not pick up copying from year to year, or institution to institution Online Self and Peer Assessment Exercise Formative to Summative Assessments: Considerations Formative Summative Low stakes High stakes Checks No tutor marking, copying detection Experience Initial exercise limited training Moderation Little moderation of marking Random tutor marking, copying detection Training in assessment Anomalous marks adjusted Comparative tutor marking, plagiarism detection Students experienced at self and peer assessment Students gain marks for marking correctly Further development Should be able to soon offer • to include images • check submission step • present criteria to students before exercise • students receive marks for marking And perhaps eventually include • feedback on feedback • computer marking of free text Self and Peer Assessment What I leave you with • Online system is available • Relatively easy to use with supporting documentation • It has some features that can engage students (text messaging) and staff (plagiarism detection) • Academic issues remain important – – – – Question design Students trained in assessment techniques Student understanding of what constitutes plagiarism Policy for dealing with plagiarism (formative work) • Further developments are possible, suggestions welcomed