LPP KA3-ICT Project 2011-13
511485-LLP-1-2010-1-NO-KA3-KA3MP
Done-IT
Develop of open systems services for smartphones that facilitates new evaluation methods, and
enhances use of immediate feedback on evaluation results obtained in tests as a creative learning
tool.
WP 3: Develop new evaluation models
D3.1: New Peer Learning assessment methods
Author and editor:
Co-Authors:
Version:
Date:
Start month
End month
Package leader
Language of the report
Regine Ringdal, Gabrielle Hansen, Trond
Morten Thorseth, John B. Stav and Liviu
Moldovan
Bela Gayer, Miro Uran, Gabor Markus, Per
Bergegard
Final
31.03.2013
1
27
HiST Contract Research
English
This project has been funded with support from the European Commission. This publication
reflects the views only of the authors, and the Commission cannot be held responsible for any use,
which may be made of the information contained therein.
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
This document may not be copied, reproduced, or modified in whole or in part for any purpose
without written permission from the Done-IT consortium. In addition to such written permission
to copy, reproduce, or modify this document in whole or part, an acknowledgement of the
authors of the document and all applicable portions of the copyright notice must be clearly
referenced. All rights reserved.
This document may change without notice, but consortium members should be informed, and
number of version, stage and date should be given.
Project consortium






Sør-Trøndelag Unievrsity College, Faculty of Technology, Trondheim, Norway
Centrum for Flexible Learning, Söderhamn, Sweden
Petru Maior University of Targu-Mures, Tirgu Mures, Romania
Magyar Hegesztéstechnikai és Anyagvizsgálati Egyesülés (MHtE), Budapest,Hungaria
Institut za Varilstvo (IzV), Ljubljana, Slovenia
HiST Contract Research, Trondheim, Norway
The summary is written in English, Norwegian, Swedish, Hungarian, Slovenian and Rumanian.
2
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Table of Contents
Done-IT ............................................................................................................... 1
1. Summary in EN, NO, SE, HU, SL and RO...................................................... 4
2. Introduction .................................................................................................. 14
2.1 Assessment - the situation today ........................................................................... 14
3. The DoneIT-approach: Peer Learning Assessment System ..................... 17
4. Requirements for use .................................................................................. 19
5. Initiating ........................................................................................................ 20
6. The questions............................................................................................... 21
6.1 The nature of Multiple Choice questions ............................................................... 21
7. Rules settings .............................................................................................. 23
8. Identifying problems .................................................................................... 24
9. Learning activities........................................................................................ 25
9.1 Feedback ................................................................................................................ 25
10. Practical possibilities with use of the new Peer Learning Assessment System
(PELE). .............................................................................................................. 28
10.1 Group activity ....................................................................................................... 28
10.2 Immediate feedback............................................................................................. 29
10.3 Active learning ...................................................................................................... 30
11. Learning mechanisms involved ................................................................ 31
11.1 Group discussion .................................................................................................. 31
11.2 Verification of a group discussion ........................................................................ 32
12. Feedback to the teacher ............................................................................ 33
13. Big versus small assessments ................................................................. 34
14. The second chance .................................................................................... 35
15. Experiences with new Peer Learning Assessment methods in Romania36
15.1 Teaching in a new way ......................................................................................... 36
15.2 Evaluating in a new way ....................................................................................... 37
15.3 New type of evaluation supported by infrastructure .......................................... 41
15.4 Conclusions........................................................................................................... 42
16. Literature references ................................................................................. 47
3
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
1. Summary in EN, NO, SE, HU, SL and RO
How may the teacher or instructor turn PELE into a new evaluation method?
PELE in a nutshell online video is found at http://histproject.no/node/856.
PELE may be used in assignments, exercises, laboratory work, tests and exams that are structured
as multiple-choice questions with a certain number of alternatives. When your teacher account
for PELE has been created (D8.1 in WP 8), you may do this:
1. Develope a number of assignments, exercises, and laboratory related quizzez, tests of
exams containing closed multiple choice questions. The tests should include the whole
curriculumn, and utilize a number of conceptual questinos. The later is important for use
in assessment for learning processes. Upload them into PELE
2. In each course, organize a number of short tests. It is better to provide many, but short
assessment activities. This could for instance involve one test every 3-4 weeks. Each
assessment activitity may contribute with a certain percentage to the final score/grade
that each student obtain at the end of a course.
3. Distribute the assignments, exercises, laboratory related quizzez, tests or exam to the
class on paper. With only a few questions, just project them on the wall.
4. Use PELE to collect the result from each test.. Students may use hand held devices like
laptops, tablets or modern smartphones to answer the test.
5. Provide immediate feedback after closure of each test. It is recommended to do that after
having a short break in class. The teahcer may for some of the questions, give the
students a second chance by starting up (from PeLe) the online Student Response
System.
6. PELE may be used to collect resutls from a final exam. This requires that the
questionnaries are using closed multiple choice questions. The analytical part of an exam
must still be organized by using traditional hand written solutions.
With PELE, the traditional "large" final exam at the end of the course may be replaced with a set
of smaller tests during the semester, and a reduced/smaller final exam that addresses the
analytical part. The analytcal part usually requires reflection processes, whereby multiple
choice questions cannot be used.
It is a key issue that the students them self register their data (inclusive the student name),
making it is easy for the teacher or instructor to handle several courses, each containing several
tests in large classes.
4
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
It is not necessary to reconstruct rooms or buildings in order to start using PeLe. Access to a good
wireless network is enough.
Consider to use the Mobile PeLe Service Unit (MPSU) if the WI-FI network is overloaded (has too
limited capacity), security rules limit the access from students hand held devices in the class room,
if you don't have any WI-FI network infrastructure, or no access to Internet. MPSU is a server and
built-in Access Point (AP), setting up a local WI-FI network. Additional AP and/or Wireles Routers
may be connected in order to increase the capacity of the local wireless network.
During the assignments, exercises, tests or exams the students may use their hand held device to
connect to Internet based resources. PeLe may not be used to collect analytical descriptions from
assignments, exercises, tests or exams, since PeLe dosent yet support free text input.
PeLe gives the teacher a new learning tool, allowing him/her to either give verification or
elaborative feedback to individual students or groups of students immediately after a
test. Students will, when they still remember the test questions, learn why the correct answer is
correct and why the other ones are incorrect. Thus, mobile technology provides new assessment
and testing criteria for education and training.
Translation of summary into Norwegian:
Oppsummering
Hvordan kan læreren eller instruktøren bruke PELE som en ny evalueringsmetode?
PELE i et nøtteskal-videoen kan finnes på nettsiden: http://histproject.no/node/856.
PELE kan brukes til oppgaver, øvelser, laboratoriumsarbeid, prøver og eksamener som er
strukturert som flervalgsspørsmål med et visst antall alternativer. Når din lærerkonto for PELE er
opprettet (D8.1 i WP 8), kan du gjøre følgende:
1. Utvikle en rekke oppgaver, øvelser, laboratorium-quizer og eksamenstester som består av
lukkede flervalgsspørsmål. Testene skal omfatte hele pensumet og bruke en rekke
konseptuelle spørsmål som det er viktig å bruke i vurdering av læringsprosesser. Du kan
laste dem inn i PELE.
2. Organisere en del korte tester på hvert kurs. Det er best å ha flere korte
evalueringeringsrunder. Dette kan for eksempel innebære en prøve hver 3-4 uker. Hver
evalueringsrunde kan bidra med en viss prosent til sluttresultatet/ sluttkarakteren som
hvert elev oppnår i slutten av kurset.
5
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
3. Dele ut oppdragene, øvelsene, laboratorium-testene, prøvene eller eksamensoppgavene
til klassen på papir. Når de består av noen få spørsmål, kan du bare projisere dem på
veggen.
4. Bruke PELE til å samle resultater fra enhver test. Studentene kan bruke håndholdte
enheter som bærbare PC-er, tablet-PC-er eller moderne smarttelefoner til å svare på
prøven.
5. Gi umiddelbar tilbakemelding til studenetene etter slutten av hver test. Det anbefales å
gjøre det etter å ha en kort pause i klassen. Læreren kan i tillegg gi elevene en ny sjanse til
å svare noen av spørsmålene ved å starte opp online Student Response System som
finnes i PELE.
6. PELE kan brukes til å samle resultater fra en slutteksamen. Dette krever at
spørreskjemaene bruker lukkede flervalgsspørsmål. Den analytiske delen av eksamen må
fortsatt være organisert ved hjelp av tradisjonelle håndskrevne løsninger.
Med PELE kan den tradisjonelle "store" avslutningseksamen i slutten av kurset bli erstattet med
et sett av mindre tester i løpet av semesteret og en redusert/mindre avslutningseksamen som
inneholder den analytiske delen. Den analytiske delen krever vanligvis refleksjonsprosesser der
flervalgsspørsmål ikke kan brukes.
Det som står sentralt er at studentene selv registrerer sine data (også navnet på studenten) slik
at det blir lett for læreren eller instruktøren å håndtere flere kurs som alle inneholder flere
tester i store klasser.
Det er ikke nødvendig å rekonstruere rom eller bygninger for å begynne å bruke PELE. Det er nok
med tilgang til et godt trådløst nettverk.
Du kan vurdere å bruke Mobile PELE service Unit (MPSU) dersom WI-FI-nettverket er overbelastet
(har for dårlig kapasitet), om sikkerhetsregler begrenser tilgang fra studenters håndholdte
enheter i klasserommet eller hvis du ikke har noen WI-FI-nettverkinfrastruktur eller tilgang til
Internett. MPSU er en server med innebygd aksesspunkt (AP) som setter opp et lokalt trådløst
nettverk. Ytterligere AP-routere og/eller trådløse routere kan bli forbundet for å øke kapasiteten
til det lokale trådløse nettverket.
I løpet av oppdrag, øvelser, prøver eller eksamener kan studentene bruke sine håndholdte
enheter til å koble seg til Internett-baserte ressurser. PELE kan ikke brukes til å samle analytiske
beskrivelser fra oppdrag, øvelser, tester eller eksamener siden PELE ennå ikke støtter fri skriving.
PELE gir læreren et nytt læringsverktøy som det gjør det mulig for ham/henne å gi både
verifiserende og elaborerende tilbakemelding til enkeltelever eller elevgrupper rett etter prøven.
Studentene vil lære hvorfor det riktige svaret er riktig og hvorfor de andre er gale mens de fortsatt
husker testspørsmålene. Dermed gir mobilteknologi ny vurderingskriterier og prøvekriterier for
utdanning og opplæring.
6
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Translation of summary into Swedish:
Sammanfattning
Hur kan läraren eller instruktören göra PELE till en ny bedömningsmetod?
Internetvideon ”PELE i ett nötskal” är tillgänglig på http://histproject.no/node/856.
PELE kan användas i olika uppdrag, övningar, laboratorieaktiviteter, test och prov som består
av flervalsfrågor med ett visst antal alternativ. När ditt PELE-lärarkonto är skapat kan du
göra följande:
1. Skapa uppdrag, övningar och laboratoriefrågesporter, -test och -prov som består av slutna
flervalsfrågor. Test borde innehålla den hela läroplanen och ett antal konceptuella frågor. Det
viktigt att använda konceptuella frågor i ett uppdrag för inlärningens skull. Överför dem till
PELE.
2. Ordna flera korta test i varje kurs. Det är bättre att skapa fler korta uppdrag och aktiviteter,
t. ex. ett test varje 3-4 veckor. Varje aktivitet kan med ett visst antal procent bidra till
slutpoängvärdena/-betyget varje student uppnår vid kursslutet.
3. Dela ut uppdrag, övningar, laboratoriefrågesporter, test eller prov i pappersform. De ska
bestå av bara några frågor och visas på skärm.
4. Använd PELE för att samla in svaren. Elever kan använda handhållna enheter, som t. ex.
bärbara datorer, surfplattor eller moderna smartphones för att svara.
5. Ge direkt feedback efter slutet på test, helst efter en kort rast. För några frågor kan läraren
ge elever andra chans genom att sätta igång (från PELE) online Elevresponssystemet
(Student Response System).
6. PELE kan användas för att samla in resultat på slutprovet. Det kräver att frågeformulär
som består av slutna flervalsfrågor. Den analytiska delen av provet måste fortfarande
anordnas med hjälp av traditionella handskrivna lösningar.
Med hjälp av PELE kan traditionella ”stora” prov på kursslut ersättas av en uppsättning
små test under terminens lopp och en begränsat/mindre slutprov som riktar sig åt den
analytiska delen. Den analytiska delen kräver ofta reflektionsprocesser, som inte kan
dra nytta av flervalsfrågor.
Det är mycket viktigt att elever själva registrerar sina uppgifter (inklusive elevens
namn), vilket gör det lätt för läraren eller instruktören att samtidigt ta hand om flera
kurser, med flera test i stora grupper.
7
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Det är inte nödvändigt att inreda om rum eller byggnader för att börja använda PELE. Det
räcker till med tillgång till ett bra trådlöst nätverk.
Tänk på att använda Mobila PELE Serviceenheten (MPSU) om WI-FI nätverket är överbelastat
(har för begränsad kapacitet), om säkerhetsregler begränsar elevernas tillgång från
handhållna enheter i klassrummet, om du inte har någon WI-FI nätverksutrustning eller
Internettillgång. MPSU är en server med en inbyggd anslutningspunkt (AP) som skapar ett
lokalt WI-FI nätverk. Ytterligare AP och/eller en trådlös router kan också kopplas till för att
förbättra det lokala trådlösa nätverkets kapacitet.
Elever kan under uppdrag, övningar, test eller prov använda sina handhållna enheter för att
koppla till Internetbaserade resurser. PELE kan inte användas för att samla in analytiska
beskrivningar i uppdrag, övningar, test eller prov eftersom PELE inte stöder fria textinsatser
än.
PELE ger läraren ett nytt undervisningsverktyg och gör det möjligt för honom/henne att
antingen ge verifikation eller utvecklad feedback till enskilda elever eller elevgrupper direkt
efter ett test. Elever ska lära sig varför det rätta svaret är rätt och alla andra är fel medan de
fortfarande kommer ihåg testfrågorna. På så sätt skapar den mobila teknologin nya
uppdrags- och testkriterier inom utbildning och yrkesträning.
Translation of summary into Hungarian:
Összefoglalás
Hogyan lehet a tanárnak, vagy az oktatónak a PELE rendszert egy új értékelési módszerként
alkalmazni a munkája során?
PELE-ről dióhéjban egy online videó található http://histproject.no/node/856. web lapon.
PELE rendszer felhasználható több területen, például feladatok, gyakorlatok, laboratóriumi
munka során, tesztek és vizsgák alkalmával, amelyek felépítése a feleletválasztós kérdésekre
épül egy bizonyos számú alternatíva megadásával.
Ha az oktatónak érvényes belépési kódja van a PELE rendszerhez, (D8.1 a WP 8), akkor
ezeket a feladatokat végezheti el:
1.. Fejleszthet számos feladatot, gyakorlatot és laboratóriumi munkával kapcsolatos
vetélkedőket, teszteket vizsgákhoz, zárt feleletválasztós kérdésekkel. A kérdéseknek célszerű
kiterjednie az egész tananyagra, így használhat számos koncepcionális kérdést is. A
későbbiekben fontos olyan értékelés elkészítése is, ami a tanulási folyamatokra világít rá.
8
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Ezek feltölthetőek a PELE rendszerbe is.
2.. Minden kurzushoz, készíthet számos rövid tesztet. Jobb ha sok, de rövid értékelési
tevékenységet végez az oktató. Például adott testet 3-4 hetente ismételnek. Az egyes
értékelési tevékenységek hozzájárulhatnak egy bizonyos százalékban, hogy a végeredmény /
rangsor, pontos legyen és minden diák megkaphassa azt végén.
3.. Osszuk ki a feladatokat, gyakorlatokat, laboratóriumi munkával kapcsolatos feladatokat,
tesztek és vizsga anyagokat az osztálynak papíron. Csak néhány kérdést, amiket kivetíthetünk
a falra is.
4.. Használja PELE rendszert begyűjteni az eredményeket minden teszt végén. A hallgatók
használhatnak kézi eszközöket persze, mint a laptopok, Tablet vagy modern okos telefonok a
válaszok megadásához.
5. Azonnali visszajelzést ad a lezárása után minden PELE teszt. Javasoljuk, hogy miután egy
rövid szünetet tartottunk az osztályban. A tanár néhány kérdést a hibásan megválaszoltak
közül vissza ad szavazásra, hogy a diákok egy második esélyt kapjanak és használják a
válaszokhoz az SRS Hallgatói Válaszadó Rendszert.
6. PELE-t lehet egy záróvizsga eredményének gyűjtésére is használni. Ez megköveteli, hogy a
kérdőívek segítségével zárt feleletválasztós kérdésekre adjanak választ a hallgatók. A vizsga
analitikai részéhez kell még a hagyományos kézzel írott megoldásokat is alkalmazni persze.
A Pelé rendszer használatával a hagyományos "nagy" vizsgák a kurzus végén
helyettesíthetőek egy sor kisebb teszt használatával a félév során, és egy redukált / kisebb
záróvizsgával, amely foglalkozik az elemző részekkel. Az analitikai rész általában csak
gondolkodás folyamatait méri, amely feleletválasztós kérdések esetében nem használhatók.
Az egyik legfontosabb kérdés, hogy a tanulók saját adataikat regisztrálják (beleértve a
hallgató nevét is), így könnyen megtalálhatja a tanár, vagy az oktató, és könnyen tudja kezelni
több tanfolyam esetében is. Nem baj ha mindenki több vizsgát akar tenni vagy nagy
osztálylétszámok vannaka vizsgázó csorportoknál.
Nem szükséges átépíteni semmit a helyiségekben vagy épületekben a PELE használatának
megkezdése érdekében. A használathoz egy jó vezeték nélküli hálózat is elég.
Fontoljuk meg, hogy a Mobil PELE Service Unit (MPSU)használható-e, ha a Wi-Fi hálózat
túlterhelt (túl korlátozott kapacitással rendelkezik), a biztonsági szabályok korlátozzák a
hozzáférést a diákok kézi eszközeihez a tanteremben, vagy ha nincs WI-FI hálózati
infrastruktúra, illetve nincs internet hozzáférés. MPSU egy szerver és a beépített hozzáférési
pont (AP), amely létrehozta a helyi Wi-Fi hálózatot. Kiegészítő AP-vel és / vagy vezeték
9
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
nélküli routerel lehet csatlakoztatni a készülékeket annak érdekében, hogy növeljék a
kapacitását a helyi vezeték nélküli hálózat terhelése nélkül.
A feladatok, gyakorlatok, tesztek vagy vizsgák alkalmával a diákok használhatják a kézi
eszközöket, csatlakozhatnak az internetes forrásokhoz. PELE nem alkalmas gyűjteni az
analitikus leírásokat feladatokat, gyakorlatokat, teszteket és az ilyen vizsgákhoz sem jó, mert
PELE még nem támogatja a szabad szövegbevitelt.
PELE ad a tanárnak egy új tanulási eszközt, amely lehetővé teszi számára / neki, hogy vagy
ellenőrzéshez használja, vagy elaborative visszajelzést ad az egyes tanulóknak vagy
csoportoknak a teszt után. A hallgatók, ha még emlékeznek a vizsgálati kérdésekre, hogy
miért a helyes válasz helyes, és helytelen helytelen, akkor újra tudják gondolni a
megoldásaikat. Így a mobil technológia új értékelési kritériumokat és vizsgálati módszert ad
az oktatáshoz és a képzéshez.
Translation of summary into Slovenian:
Povzetek
Kako lahko učitelj ali inštruktor uvede metodo PeLe (Peer Learning Assessment Service) kot novo
metodo ocenjevanja?
Spletni prikaz PeLe metode je na voljo na http://histproject.no/node/856
Metodo PeLe lahko uporabljamo pri vajah, delu v laboratoriju, reševanju vprašalnikov, testov in
izpitov, ki so sestavljeni na način reševanja z več možnimi odgovori in z določenim številom
alternativ. Ko ustvariš svoj PeLe učiteljski račun (D8.1 v DP 8), ti program zagotavlja dostopanje do
sledečih možnosti:
1. Razvoj številnih nalog, vaj in laboratorijsko povezanih vprašalnikov, testnih pol in izpitov z več
možnimi odgovori. Testi morajo vključevati celoten učni načrt, vključno s številnimi
konceptualnimi vprašanji. Zlasti slednje je pomembno za uporabo v procesih ocenjevanja za
učenje. Naložite jih v v PeLe sistem.
2. V vsakem ciklu izobraževanja organizirajte več kratkih testov. Dosedanja testiranja so namreč
pokazala, da so dobljeni rezultati boljši, če zagotovimo pogostejše a hkrati krajše ocenjevanje. To
lahko na primer vključuje en preizkus vsake 3-4 tedne. To na končni ravni lahko rezultira z višjim
odstotkom uspešnosti.
3. Razdelite naloge, vaje, vprašalnike, teste ali izpite za razred v papirni obliki. Z le nekaj vprašanji
jih projicirajte na steno.
10
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
4. Uporabite metodo PeLe za analizo posameznega testa. Slušatelji lahko za reševanje uporabljajo
ročne naprave, kot so prenosni računalniki, tablica ali sodobni pametni telefon.
5. Po zaključku posameznega testa ali izpita zagotovite takojšnjo povratno informacijo
Priporočljivo je, da to naredite po kratkem odmoru. Učitelj lahko da študentom pri reševanju
nekaterih vprašanj možnost ponovnega odgovarjanja, tako da zažene v programu PeLe aplikacijo
online Student Response System (“second chance” - druga možnost).
6. Sistem PeLe lahko uporabimo za zbiranje rezultatov zaključnih izpitov. Vprašalniki morajo biti
sestavljeni na način z zaprtimi večmožnostnimi odgovori. Analitični del moramo še vedno narediti
ročno.
S sistemom PeLe lahko običajni zaključni izpit nadomestimo z nizom manjših testov med
semestrom. Tako zmanjšamo analitični del zaključnega izpita. Ta običajno zahteva procese
refleksije, pri čemer uporaba sistema več možnih odgovorov ni mogoča
Ključno je, da študenti sami vnesejo svoje podatke (vključujoč ime študenta), zaradi česar je
postopek za učitelja enostavnejši, še posebno kadar učijo več različnih predmetov z več
predvidenimi testiranji in v velikih razredih.
Za uspešen začetek dela po PeLe metodi ni potrebno preurejati dosedanjih prostorov. Zagotoviti
je potrebno le dostop do dobrega brezžičnega omrežja.
V primeru, da je brezžično omrežje preobremenjeno (ima tudi omejene zmogljivosti), ali
varnostna pravila omejujejo dostop študentom z ročnimi napravami v učilnicah, če nimate
vzpostavljenega brezžičnega omrežja, ali nimate sploh dostopa do interneta, lahko uporabite
Mobile PeLe storitev (MPSU). MPSU je v tem primeru strežnik in vgrajen Access Point (AP) je
nastavljen kot nosilec lokalnega brezžičnega omrežja. Dodatne AP in /ali brezžični Router-ji so
lahko povezani, da povečamo zmožnost lokalnega brezžičnega omrežja.
Med reševanjem nalog, vaj, testov ali izpitov lahko slušatelji uporabljajo svoje ročne elektronske
naprave, da se brezžično povežejo z osnovnimi bazami v PeLe sistemu. PeLe ne moremo uporabiti
za zbiranje analitičnih tolmačenj nalog, vaj, testov ali izpitov, ker sistem PeLe še ne podpira
brezplačnega vnosa besedil.
Sistem PeLe daje učitelju na voljo novo učno orodje. Omogoča mu sprotno preverjanje in mu daje
povratne informacije za posameznike ali skupine študentov takoj po opravljenem testu. Za
študente je pomembno, da takoj po zaključku testa izvedo, zakaj je nek odgovor pravilen in zakaj
so ostali napačni.
11
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Translation of summary into Romanian:
Cum poate profesorul sau instructorul transforma PELE într-o nouă metodă de evaluare?
Videoclip-ul despre PELE se găsește la adresa http://histproject.no/node/856.
Pele pot fi folosit în evaluări, exerciții, lucrări de laborator, teste și examene, care sunt structurate
sub formă de întrebări cu răspunsuri multiple, cu un anumit număr de alternative. Dacă vă creați
contul de profesor (D8.1 în WP 8), cu ajutorul PeLe puteți face următoarele:
1. Dezvolta o serie de evaluări, exerciții, chestionare legate de laborator, teste de examene care
conțin întrebări cu răspunsuri multiple închise. Testele trebuie să cuprindă întregul curriculum și
să utilizeze o serie de întrebări conceptuale. Sub această formă pot fi utilizate în evaluarea
procesului de învățare. Încărcați-le în PELE
2. In fiecare curs, se pot efectua o serie de teste scurte. Activitățile de evaluare este mai bine să le
efectuați mai multe și de scurtă durată. De exemplu, se poate efectua un test la fiecare 3-4
săptămâni. Fiecare activitate de evaluare poate contribui cu un anumit procent la nota pe care
studentul o obține la finalizarea cursului.
3. Se distribuie cursanților evaluările, exercițiile, chestionarele de laborator, testele sau
examenele pe suport de hârtie. Dacă acestea constau din câteva întrebări, se pot proiecta pe
perete.
4. Se utilizează PELE pentru colectarea rezultatelor testelor. Studenții pot folosi dispozitive
portabile cum ar fi: laptop-uri, tablete sau telefoane inteligente moderne, pentru a răspunde la
test.
5. Se comunică rezultatele imediat după finalizarea testului. Acest lucru este recomandat după o
scurtă pauză. Pentru anumite întrebări profesorul poate să le ofere studenților încă o șansă prin
activarea (din Pele) Sistemului de Răspuns al Studenților.
6. PELE pot fi folosit pentru colectarea rezultatelor de la un examen final. Acest lucru necesită
utilizarea de chestionare cu întrebări cu răspunsuri multiple închise. În continuare, partea analitică
a examenului trebuie să fie susținută sub forma clasică, scrisă de mână.
12
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Cu Pele, examenul tradițional "amplu" de la finalizarea cursului, poate fi înlocuit cu un set de teste
mai mici susținute pe parcursul semestrului și un examen final restrâns / mai mic, care se referă la
partea analitică a cursului. Aceasta necesită, de obicei, reflecție, în cursul căreia nu se pot utiliza
întrebări cu mai multe variante de răspuns.
Este o problemă-cheie ca studenții să-și înregistreze datele de autentificare (inclusiv numele),
ceea ce permite profesorului / instructorului să susțină mai multe cursuri in clase mari, fiecare
dintre acestea conținând mai multe teste.
Nu este necesar că construiți laboratoare sau clădiri, pentru a utiliza Pele. Accesul la o rețea
wireless este suficient.
Puteți să folosiți unitatea mobilă Pele Service (MPSU) dacă rețeaua Wi-Fi este supraîncărcată (are
o capacitate limitată), normele de securitate limitează accesul dispozitivelor mobile ale
studenților în sala de clasă, dacă nu aveți nicio infrastructură WI-FI de rețea FI, sau dacă nu aveți
acces la Internet. MPSU este un server construit în Access Point (AP) pentru crearea unei rețele
locale Wi-Fi. AP suplimentare și / sau rutere wireless pot fi conectate în vederea extinderii
capacității rețelei locale wireless.
În timpul evaluării, exercițiilor, chestionarelor de laborator, testelor sau examenelor studenții pot
folosi dispozitivele mobile proprii pentru conectare la resursele de pe Internet. PeLe nu poate fi
folosit pentru colectarea de descrieri analitice de evaluări, exerciții, teste sau examene deoarece
PeLe nu permite introducerea textului liber.
PeLe oferă profesorului un nou instrument de învățare, care îi permite să examineze și să
colecteze impresii de la studenți sau grupuri de studenți, imediat după un test.
Studenții, când încă își mai amintesc întrebările din test, au posibilitatea să învețe de ce anumite
răspunsuri sunt corecte, iar celelalte sunt incorecte. În acest fel, tehnologia mobilă oferă noi
metodologii de evaluare și criterii de testare care se pot folosi în educația și formarea
profesională.
13
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
2. Introduction
The aim of this report is to give some pedagogical arguments for why teaching and assessments
with the assessment system developed through the DoneIT project, might work and to some
extent how. We have gathered some pedagogical argumentation for why the Done-IT approach
might be beneficial in students learning processes, in addition to the traditional assessment of
student’s level of understanding.
This picture illustrates how a written exam is done in higher education in Norway.
2.1 Assessment - the situation today
Within the research field on assessment it is normal to distinguish between two types of
assessment, summative and formative. The first of these measures and controls that defined
learning objectives are achieved. Feedback information is provided after students work is
completed, often in form of a final core or a grade. The purpose of the assessment is to rank,
endorse or verify students' competence (Sadler, 1998). Assessments can also generate feedback
information provided during a learning process, as information students can use to improve their
own learning and further performances. This feedback information may also be a tool for
teachers. Teachers can adjust their own teaching practice in relation to students' needs. When
assessment serves these purposes it is called a formative assessment (Sadler, 1998).
Within the research literature on assessment, it is promoted a need to explore how higher
education institutions, and others, can use assessment more effective in promoting student
learning. Feedback is barely conceptualized in the theoretical literature on higher education, and
14
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
other places, making it difficult to design effective feedback practices, and to evaluate their
effectiveness in relation to students' learning (Sadler, 1998; Yorke, 2003).
In higher education, formative assessment and feedback is often a teacher-led process, and is
often perceived as the teacher's responsibility. Feedback is still considered a mediation process,
where teachers communicate feedback information to students about their academic strengths
and weaknesses, which students are supposed "to decode" and convert to concrete actions to
improve their understanding and academic progress (Nicol and Macfarlane-Dick, 2006). The
traditional and most common feedback practice generally involves correction of errors.
Traditional classroom assessments/tests are a widely used feedback practice for this. Great
emphasis is placed on the adequacy of scores, and less emphasis on the importance of them
(Hattie & Timperley, 2007). There is little research evidence that such traditional classroom tests /
tests assist students in their learning process. A meta-study by Black & William (1998), examined
578 studies that related classroom tests to the students' learning, and concluded the following:
Classroom assessment typically encourages superficial and rote learning, concentrating on recall of isolated details,
usually items of knowledge which pupils soon forget . . . teachers do not generally review the assessment questions that
they use and do not discuss them critically with peers, so there is little reflection on what is being assessed. (p. 17)
Assessments are in a way the main focus in a student’s life. It is the judgment from the final exam
or test that decides their future and possibilities for further studies and a possible job. A school
diploma, with an evaluation of their professional qualification is the only proof they can use after
the education or course. This makes it natural for many students to focus on the performance on
the assessment rather than learning focusing on learning the subject.
The most common assessment activities students participate in throughout a semester are
academic tests and/or a final examination. Students perform through answering a number of
academic questions one day, and get a verifying feedback, often in form of a grade or pass/fail
response, days or perhaps weeks after their performance. Elaborative feedback information, like
in-depth focus on what is right or wrong and why, is rarely included, reducing the learning effect
of the feedback information significantly. For a professional feedback to influence students'
learning process in a positive way, it needs to include both verifying and elaborative information
(Kulhavy & Stock 1989).
With the New Peer Learning Assessment System (PeLe), developed through the done IT project,
each student can respond to test questions using a smartphone. An automatic grading system,
integrated in the assessment system, assess their performance. This assessment technology gives
teachers immediate access to student’s performances, and makes it possible for him/her to
localize student’s problem areas in a fast, simple and efficient manner. This further enable
teacher`s to give students verifying and/or elaborative feedback information immediately after
the test is finished (post assessment activity). Students can get feedback on their performance,
15
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
while their performance and thought processes behind, is fresh in their memory. Through various
elaborative feedback activities, such as in-depth teacher explanation or peer learning through
student collaboration, students can get an opportunity to achieve a greater understanding of, and
learn from their mistakes.
16
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
3. The DoneIT-approach: Peer Learning
Assessment System
The following figures in this report are taken at various points in the project and therefore do not
represent necessarily the finished product. The DoneIT-approach to an assessment is based on
the following way of performing an assessment. Details about how the system can be used are
found in the user manual. This is a general approach to what is done:
1. Preparation. The teacher prepares questions for the assessment and creates a printout
that can be handed out to the class. The assessment has to be prepared as a multiple
choice question assessment.
2. Registration. When the assessment is finished and ready, the setup has to be prepared
for the DoneIT system. The teacher has to start DoneITC and create a new assessment
setup document. Here the system is fed information about the assessment, the number
of questions, number of alternatives for each question and what alternative(s) is correct.
3. Rules. When the assessment is set up, the rules have to be set. The rules tell the system
how the assessment is to be evaluated. As a default value, a correct alternative give 1p.
while a wrong alternative gives 0p. Not responding to an alternative gives 0p.
4. End preparation. When the assessment is set up correctly, the assessment can be
uploaded to the DoneIT server and gets an assessment code. The assessment can also be
stored to file and brought to the classroom upon the assessment day.
5. Start. You start the assessment by uploading the setup document, and get an assessment
code.
6. Login. Students login with their username and password to the given assessment
identified by the code.
7. Handout. When everybody has logged in, something the teacher can
monitor in DoneITC, the paper version is handed out and the
students can start working with the assignment.
8. Monitor. While the students respond, the situation is monitored and
the teacher can prepare for what to do now after the assessment is
over.
9. Submission. Students submit their responses, and take a break.
10. Identification. When everybody has submitted, the teacher can start
to identify their problem areas, based on what the students have
responded. The interface gives a picture of how the students have
responded.
17
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
11. Post assessment activity (PAA). When the questions causing problems are selected and
the class is gathered again, the main interactivity can begin. The approach here is
individually and will be variable from teacher to teacher or based upon what the students
have responded.
The main purpose of the DoneIT project is to open up for the PAA and to give
the teacher freedom to choose from a set of different approaches.
18
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
4. Requirements for use
In order to run the system you need a good WiFi coverage over the area with sufficient access for
all students to participate with their response unit. A student can use a smartphone, an iPad or
iPod or a computer that is WebKit1 compatible. With the first prototype the system requires
webKit browsers and works only on iPhone, iPad or iPod, but a student interface that is more
generic html+javascript is under construction and we are planning for a solution where most kind
of browsers can be used.
You also need a computer with the DoneIT Controller software DoneITC installed on a computer
where the screen can be projected on the wall for the class. More information about the system
and how can be used is found in the DoneIT User manual.
1
http://en.wikipedia.org/wiki/WebKit
19
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
5. Initiating
When you start to use the system, it’s important to give the students a thoroughly introduction;
what is a Peer Learning Assessment System, why should they use it and how does it work? Based
on initial testing of the assessment system with three students groups, we learned that for
students, use of technology in an assessment situation represents something new and slightly
frightening. They are used to answering on paper, as this is what they have done throughout their
academic careers. Initially, they therefore felt a bit insecure about the technology, and the first
test we had with them introduced considerable stress because of this. The uncertainty was rooted
in fear of the iPod suddenly turning off, or breaking down, or that their responses would be lost.
When we enter a classroom with the assessment system, one of our primary focus areas will
therefore be on making the students feel confident and comfortable with the technology. The
system should be presented with a thorough introduction followed by a sample test, so that the
students get used to the technology as soon as possible.
Creating a simple assessment where you force the students to do certain activities just to get to
know the system might be one way of getting students used to navigation and responding. You
should explain why you want to use it and how things will work, and how the students should use
their interface, what the buttons do and so on. Do not expect that everyone understands how
they should use the system. We have spent hours with students testing out and trying to make
the student interface user friendly and intuitive and made corrections based on feedback from
the students. We have also followed up with interviews of the students trying to identify parts of
the interface that is not intuitive. The best way will be to give an introduction and let the students
try before any kind of assessment is performed. Some people might fear technology and don’t
trust the technology when participating.
20
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
6. The questions
In order to use the DoneIT system you have to prepare you assessment as a Multiple Choice
Question assessment. When you have a set of multiple choice questions, you want to measure
something or create something in the post assessment activity. You have to make sure that your
questions target what you want to measure and meets to some extent what you want to in the
PAA. However Multiple Choice Questions (MCQ) can be limited, and there is a lot of critics
towards the use of such assessments. The results of your use depends on the context you use the
questions (Nicol 2007). The DoneIT approach, with the Peer Learning Assessment System, tries to
utilize a twist in the way that assessments are done. We want to change the assessment context.
Academic assessments, like test, students typically perceive as a performance situation, their
focus is on achievement. We want an assessment to be perceived as a learning situation, where
students can learn from their mistakes. We want to turn an assessment into an arena for learning.
6.1 The nature of Multiple Choice questions
In a world where the access to computers have improved, resources to education reduced, and
the number of students is increasing, assessments in the form of a Multiple Choice Question
(MCQ) assessment has increased. If the assessment is done electronically online, the results can
be provided immediately, and since this approach saves time for the teacher removing the time
spend to correct and give feedback, and giving time to teach even more, the trend is that the use
of MCQ has increased. However as stated by Nicol, the nature of MCQ provide some pedagogical
limitations by nature (Nicol, 2007).
In the literature it is argued that MCQ promote memorization and factual recall (Airasian, 1994;
Scouller,1998) and does not encourage students to think and understand the big picture. The
situation can be improved if the questions are made better, requiring students to really think
before they submit an answer. This however requires a lot of work for the teacher to construct
assessments that has good questions.
Secondly, the feedback provided through MCQs is usually quite limited as it is predetermined
during test construction. Hence there is little scope for personalization of feedback based on
different student needs. For subjects like mathematics and physics, where a significant target of
the course is to teach the students practical calculation procedures and reasoning skills, feedback
from a MCQ does not provide information about where an error was made in the reasoning or
calculation. The students get a feedback that the calculation was wrong or correct. If an activity
should enhance learning, the feedback must be in such a way that the student can change the
reasoning where the logical brake is, understand what went wrong and also understand what is
21
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
the correct way to reason or calculate. At HiST we often see students fail in their reasoning at
different levels in the same course, and feedback is needed on an individual level. With the use of
MCQ this is not possible using online MCQ assessments.
Thirdly MCQs are often used, not for pedagogical purposes, but to improve teacher efficiency,
freeing time and for more teaching. The aim is not effective learning but effective teaching. The
focus of learning might be lost in the search for cost effective teaching. For the students, the
outcome of the course is the results from the assessment and the grade from their performance.
The MCQ type of assessment, encourage students to optimize the assessment procedure. They
learn to recognize alternatives, exclude unlikely alternatives and to memorize facts, rather than
constructing a response based on their own understanding of the subject. Often students don’t
have a possibility to unravel the problem given and to get an understanding of the problem,
something that often lead to guessing and optimizing results by excluding the most unlikely
alternatives. Recent publications states that students should play a more active and participating
part in the assessment procedures (Boud 2000; Yorke 2003).
The way that we try to compensate for these drawbacks is to change the context. We apply MCQ
with an activity after the assessment where the teacher can work with the group and possibly
create different activities that benefit learning and give the students a chance to learn from their
own mistakes.
22
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
7. Rules settings
Grading MCQs has to be done by a set of grading rules. The system allows the teacher to set rules
within certain limits. You can give the same rules for all questions, or you can override the rules
on individual questions. If a teacher experiments too much with the rules, the outcome might
become unpredictable for the students, and this might have negative effect on the students. If
your assessment contains few questions, the possible outcome of the assessment is limited. If you
run an assessment with ten questions, using standard rules, you will get results ranging from 0p to
10p. By changing the rules you might alter the outcome. You can introduce multiple cardinality
(more than one correct answer) with the questions, and you can give different score to the
alternatives. This will generate a larger span in the outcome. You can also introduce negative
score on wrong alternatives, and even allow the questions to bleed (allowing questions to give a
negative score that is subtracted from the total). If you use punishment on wrong alternatives and
does not allow the questions to bleed, you might end up with a minimum of zero points in score.
If you allow bleeding, it would be an idea to use an initial score, such that the outcome will be
positive.
This will give the chance to extend the outcome space and give a better ranging of the results, if
used carefully. However, an understanding of how the outcome space of the assessment looks
like might be challenging when the rules are complex. To help the teacher in designing the
assessment we have introduced the “monkey cage” that will simulate a random response
statistically looks like. By letting the some thousand simulated “monkeys” respond, you can get a
feeling of the probability to get a score at pure random.
If these simulations can be used pedagogically with the class is difficult to predict, but it will give
you as a teacher the possibility to see how well a student should respond to be better than just
guessing.
23
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
8. Identifying problems
When the assessment is finished, you will be able to see what questions caused problems. You
then may select what questions you want to introduce to the post assessment activity phase.
If you have a high spread in the responses, this means that the class has a big problem
understanding the question, and might have guessed their response. The spread factor is a factor
ranging from 0% to 100% where 0% means that all students have responded to 1 question. A
spread factor of 100% means that there is equal number of responses to each alternative. If you
have more than one correct alternative, you will not get a spread factor of 0%, even if all students
respond correctly to your alternatives.
The distribution might help you to select what kind of activity you want to choose. If the group is
divided, it might be an indication that those who know can teach the others that don’t know. We
lack the experience to predict what you should do. You simply have to gain experience and listen
to what you students can tell you!
24
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
9. Learning activities
For a student, learning activities can consist of several types of activities.





Lectures, seminars, courses
Practical learning, problem based activities
Reading, tutorials, podcasts
Discussions, peer learning and peer support
Reflection, engagement and interaction
Often lectures only serves as an information source for learning objectives. “This is what you
should learn this week!” Fitting the assessment as a natural part in this set if activities, is difficult,
since it traditionally only provides the teacher with some feedback on his/her teaching. It is up to
the student to find ways to learn the subject, with the aid of some guided exercises and practice.
Upon how well a student has learned some material, Bloom’s taxonomies may serve as a guide
for arranging the level of knowledge. Bloom divided this domain into six levels of understanding
as given in the table below, revised by (Anderson et al. 2001). Acquisition of facts or having
knowledge is only the initial stat of understanding. The facts must be understood
(comprehension) before they can be applied to new situations (application). Knowledge must be
organized and patterns recognized (analysis) before it can be used to create new ideas
(synthesis). At the end, being able to judge and consider competing or evidence, the student
needs to be able to assess (evaluation) the relative value and validity of information or ideas.
Level of thinking
Knowlegde
Comprehension
Application
Analysis
Synthesis
Evaluation
What it represents
Facts
Understand meanings
Apply to new situations
See organization and pattern
Generalize and create new ideas
Assess value of evidence
9.1 Feedback
Feedback is important for learning. In any learning situation, feedback in one form or the other is
the mechanism that regulates the learning process. Some use the model of building a brick wall,
where feedback is the cement holding the wall together. If no cement is used, the wall can reach
an instability and collapse. Feedback serves as the verifying glue that the knowledge build in a
student’s head holds.
25
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Today, traditional assessments give a the students a chance to respond to a set of questions and
get a verifying feedback, either in the form of a degree or a passed/ not passed, often days and
weeks later. This feedback gives little or no useful feedback to the students when they don’t
know exactly what caused the degree.
The problems that were solved are out of the head and the assessment is done. If the students
could get an immediate feedback and a possible input on how to think, reason, or argue to solve
the problems, the feedback could act as a tool for the student to adjust or correct thinking
patterns. With the correct / wrong judgment, there is no such feedback to the student. If the
results appear days later, the judgment is not connected to the problem solving thoughts and
hence no corrections are made. By giving immediate feedback, the students can have a chance
to connect this minimal feedback to the actual performance done in the assessment. According to
(Race 2005) feedback is most effective when:






Timing: Feedback is given as soon as possible.
Personalizing: Feedback fit each student's achievement, individual nature, and
personality.
Empowering: the feedback intended to strengthen and consolidate learning, and not the
opposite.
Analytical: Explain carefully what was good or excellent in detail, so that the message is
clear. If the only feedback is excellent, the student does not know what was excellent.
Constructive: the feedback gives guidance on how to improve.
Managable: the amount of information has to be manageable for both parts.
One last important point is that feedback has to be manageable for both for students and the
teacher. It is impossible to provide the perfect feedback, since there just is no time. For students
the amount of feedback has to be manageable and the feedback should be focused on the points
that have the highest impact on possible improvement.
With electronic MCQ assessments this kind of feedback is a challenge. Timing can be met with
immediate feedback, but the other points are difficult to find practical ways to solve. But Phil Race
(Race 2005) also suggests some payoff feedback methods that can be provided. Some of these
are:





students comparing work
peer‐marking with feedback
constructive questioning within groups
verbal feedback to whole class or groups
group peer review
26
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Common for all these is that the activity can be done in the class, and using the group as a whole,
either in groups or in class.
In the review of Nicol 2007, he looks closer at different contexts where MCQ has been applied. He
also presents seven principles for good feedback practice that could support the development of
learner self-regulation (Nicol & MacFarlane-Dick 2006).
1
2
3
4
5
6
7
Good feedback practice
helps clearify what good performace is (gols, criteria, standards);
facititates the development of self-assessment and reflection in learning;
delivers high quality information to students about their learning;
encourages teacher and peer dialogue around learning;
encourages positive motivational beliefs and self-esteem;
Provides opportunities to close the gap between current and desired
performance;
Provides information to the teachers that can be used to shape teaching.
27
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
10. Practical possibilities with use of the
new Peer Learning Assessment System
(PELE).
On an academic test students use their former knowledge to answer questions. By using the new
assessment system, the teacher can in a simple way use students test results to identify their
problem areas, and make a methodological choice about where he/she should focus further
teaching in the post assessment face. The purpose is to give students more detailed feedback in
relation to their own achievements. By linking feedback to other classroom activities, the goal is
to improve the quality of feedback students receive about their own performance.
The way students perform, will place large constraints on the methodological choices teacher
makes after the test is completed. It`s their performances that will decide whether the teacher
encourages students to participate in group discussions, or others activities. Teacher's choice of
method will be related to the response distribution on the various test questions. Teachers can,
for example:

Give students a verifying feedback by showing them the response distribution from
different questions and emphasize which alternative is correct.

Give students an elaborative feedback, by giving them a detailed explanation about what
is right, less right, or wrong and why.

Give students the opportunity to learn from their mistakes by cooperating with their
fellow students.

Initiate a larger class discussion / debate

Give students a hint and ask them to work on a specific question one again, either
individually or in groups, and answer the question again using the response system.
10.1 Group activity
The Peer Learning Assessment System has been designed in such a way that teachers can
immediately use students’ results from an assessment and initiate numerous learning activities in
the classroom using several methodological approaches. One of these approaches is peer learning
through small group discussions and student collaboration. An interesting question is how a group
activity can benefit students learning?
28
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Collaboration between students is linked to both better understanding of the learning material
and better performances (Collier 1980; Ames 1984, 1992; Slavin 1983, 1985; Graham & Weiner
1996). According to Collier (1980), this can be explained by the fact that the learning materials
often appear clearer to students when it is "translated" by, and illustrated using examples from
fellow students rather than teacher led explanations. Students use a language (words and
concepts) between themselves that they understand, rather than more advanced terminology.
Discussions within the student groups can also help create a more relaxed and calm atmosphere
which in turn can lead to a greater willingness to admit uncertainty (Collier 1980)
Group discussion also has the advantage of involving students more active in the learning process.
Further, it can be used effectively with younger students for short periods of time. However,
unless you require participation and use a system to mark each student who speaks, it can be
challenging for the teacher to get everyone involved. There has been evidence for an increased
learning and a more favorable attitude towards learning when undergraduate courses in science
have been taught in small groups (Springer, Stanne, and Donovan 1999).
Group learning does not necessary work with every assignment. Cohen (1994) found that small
group learning can be productive for conceptual learning, under certain conditions. The most
important of these conditions is that the task is a real group task. Problems that are context-rich,
many meet these requirements. Heller et al. (1992) also found that in well-functioning
cooperative groups a better problem solution emerged than was achieved by individuals working
alone and the instructional approach improved the problem-solving performance of students at
all ability levels.
The success for creating an activity and engagement in the class is the nature of the question. A
simple factual question might have little effect on creating a discussion. There is one answer, if
the student does not know it or remember any argument for it, there is nothing to discuss. The
challenge is to find questions and problems that are within students reach, but at the same time it
is not crystal clear what might be the correct answer. If several alternatives are partly correct but
one is most correct, this might cause discussion and lead to a deeper understanding. The goal is to
get students to go more in depth on topics.
10.2 Immediate feedback
Feedback is an important contribution in relation to students' learning (Mory 1992; Laurillard
1993; Kluger & DeNisi 1996; Nelson & Pearson 1999), and it is one of the most important
principles for ensuring good and effective teaching (Rose Shine & First 1971; Chick Call & Gamson
1987; Book 1999). The message is really very simple: students learn best when they receive
feedback on their understanding and performance (Book 1999).
29
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Through use of new assessment system students can gain immediate feedback on their own
performances. Immediate feedback is important for the students to be able to connect the
feedback information they receive to the assessment and their own performance done.
10.3 Active learning
Learning cannot be compared with some sort of spectator sport (Chick Ring & Gamson 1987;
Nelson & Pearson, 1999), although perhaps misconceptions about the term "active learning" has
suggested otherwise. According to Bonwell & Eison (1991), the term "active learning" has been
used a little too random and perhaps most of intuitive understanding rather than fixed
definitions, which has led to a perception that all learning is active and that students are also
actively involved when they just sit quietly and listen (Bonwell & Eison 1991). Analysis of research
(Chick Ring & Gamson 1987), suggest that students must do more than just sit quietly and listen
to be actively involved in their own learning process, they must read, write, discuss, or be
engaged in some form of problem solving. In addition, they should also engage in deeper thought
processes as analysis, synthesis and evaluation (Bonwell & Eison 1991). Such thoughts about
learning are in line with a constructivist approach, which emphasizes the active role of students in
relation to the development of personal knowledge.
Active learning involves students doing things and thinking about the things they are doing.
Typical activities where:







Students are involved in more than passive listening
Students are engaged in activities (e.g., reading, discussing, writing)
There is less emphasis placed on information transmission and greater emphasis placed
on developing student skills
There is greater emphasis placed on the exploration of attitudes and values
Student motivation is increased (especially for adult learners)
Students can receive immediate feedback from their instructor
Students are involved in higher order thinking (analysis, synthesis, evaluation)
Here the task might range from simple to complex tasks, and still be within the active learning
part.
The Done-IT project aims to develop a new assessment model in which students' academic
performance can be used as a tool to promote active learning. Immediate access to students'
academic performance allows for quick receipt of verifying feedback, and use of more elaborate
feedback methods involving active participation from students.
30
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
11. Learning mechanisms involved
11.1 Group discussion
“Students learn what they care about and remember what they understand” Ericksen, 1984, p. 51.
This citation covers a lot of what good teaching is about. If you as a teacher just continues to talk
and a student do not understand, it’s impossible to remember. You have to make sure that they
understand and that the content of what you are teaching makes sense. One way of doing that is
to allow students to rephrase and organize their own structure of understanding. When they have
got the understanding, remembering is no longer an impossible task. Group activity and group
discussion serves as a way to restructure and formulate orally an understanding of a concept. It
will also immediately give feedback to the students wither they understand or not.
In order to trigger a group discussion, several approaches can be done from the teacher with the
Done-IT system. He/she might display the question and talk about the question, identifying the
target of the problem. The application is made transparent such that the questions can appear
through the application and results can be displayed on top. It is possible to display the
percentage right or wrong chart to the class, giving no leading information to the correct
alternative.
Figure 1 Percentage right or wrong bar chart from the DoneIT prototype.
It is also possible to show the distribution of the responses, but without showing what alternative
is correct.
31
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Figure 2 Histogram showing the distribution of responses from the DoneIT prototype.
11.2 Verification of a group discussion
Any form of group discussion should always end up with a teachers led explanation afterwards.
Smith et al., (2011) shows that for peer instruction, a method applied when using student
response systems, where students first answer individually, and then again later after a group
discussion, the final teacher explanation is important. The combination of peer discussion
followed by instructor explanation improved average student performance substantially when
compared with either alone. Research from HiST regarding use of student response system in
classes also highlights the importance of thoroughly teacher explanation after a session of student
discussions and voting. For the students, the teacher's explanation was perceived as a
constructive feedback, and was highlighted as critically important for their own experience of
learning. Through the teacher's explanation, the students got an understanding of why the
various options were correct or incorrect. One thing is to cast a vote that turns out to be right or
wrong; another matter entirely is to be able to understand why it is right or wrong. If they achieve
such an understanding, they feel that they really learn something through the questions. The
students are keen to point out that the teacher should give adequate explanations for why the
wrong options are incorrect. For the students, this is a way for those who answered incorrectly to
understand why they got it wrong. The same methodological approach will be important using the
new Peer Learning Assessment System in the classroom. Any student led activity should be
followed by a teacher led explanation at the end.
32
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
12. Feedback to the teacher
The DoneIT system has a built inn functionality to allow students to “mark” a question. This
marking will appear to the teacher and he/she will be able to get statistics of how many marks are
given on a question. Students have argued for making this anonymous. The teacher should not
see who marks a question. The idea is that the teacher can define what the mark represents. The
students can also use it to keep track of questions that are difficult and needs more attention.
Possible approaches can be used:



Teacher doesn’t use the marking function: Free to use at will for the students.
Signal for “Please Explain” from the students. The teacher encourages students to mark
those questions they want the teacher to explain carefully after the assessment.
I am sure flagging of the questions. Students can mark the question they feel certain on
they know. This to make them think more deeply about the questions being asked.
Figure 3 The students interface when question 1 has been “marked”.
The “Please Explain” will give the students some influence of what the teacher should spend time
on after the assessment, and may trigger positive motivation towards wanting to understand a
problem.
The system now will not allow students to benefit from an “I am sure” definition in such a way
that the score can be increased when the students have responded correctly and flagged that
response. The suggestion is here to choose an approach and stick to it. If you change the rules,
the use might cause confusion.
33
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
13. Big versus small assessments
One question that is not answered here is the size of the assessment. What way of using the New
Peer Learning Assessment System works best for the students? Should the assessments be given
with relative high frequency, and with a few questions, or should the assessments be less
frequent but bigger in size? What is the optimal size of the assessments, when trying to trigger
learning activities? One suggestion here is to make the assessments large enough such that some
problems can be identified. An assessment with questions ranging from simple fact questions to
more analytical problems such that some everyone reaches a problem they can’t solve.
34
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
14. The second chance
The Peer Learning Assessment System also introduce the possibility to answer a question a
second time, and therefor get a second chance on questions that are difficult to understand or by
any reason was not answered correctly. How this will be used is not easy to predict. If you allow
the students to get a pure second chance you might get unwanted effects. The Peer Learning
Assessment System allows teachers to send the question out to the students again after some
information, student discussion or a hint, and opens the possibility to respond to the same
question after a teacher- student interaction or in a student response system approach where the
results are stored. When the response is given as a part of an assessment, Dickinson and Flick
1998 observed how grading might undermine the pedagogical goals of the instruction. James
(2006) observed, working with student response systems, that when students responses were
graded, this cause some unwanted effects on the students. James also observed that when the
score for correct answers were given weak students started focusing on stronger student and
their response rather than arguing for their own meaning. The score focus turned the students
from a learning mode towards performance mode. So finding a way to use the second chance
data is essential. Pure performance mode might be leading the group activity away from what you
are trying to create in the class.
(James 2006) An analysis of conversation transcripts revealed that conversation partners with a
large disparity in student knowledge in the high stakes classroom focused most of their discussion
around the dominant student’s answer preference. Such was not the case in the low stakes
classroom where conversations were more balanced, including ideas put forth evenly from both
partners.
In future versions we have to focus on what to do with the score from a second chance data. It
can be that we introduce a second set of questions, at the same level and let the second question
count as a standard question. It might also be that the second chance score counts 50% or 20% of
the assessment score. Another possibility is to let the collective response control the weight of
the second chance data. If everybody answers correctly or participates, then the second chance
data counts 100%. The system has no way to handle this approach now but it might be done in a
later stage of the project.
35
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
15. Experiences with new Peer Learning
Assessment methods in Romania
15.1. Teaching in a new way
Due to the involvement in European projects UPM is currently using new pedagogical methods,
the blended learning solutions, video solutions, mobile learning, etc.
In the Done-IT project UPM has investigated new features for training, by using adequate
equipment bought for this purpose. UPM extended the features of the traditional teaching, and
investigated a new innovative dimension of teaching based up on using 2D or 3D multimedia
material for illustration of defects within material testing, but also 3D reconstruction of defects,
using non-destructive techniques.
This has come as a consequence of the teaching innovation on which, during material testing
courses delivered by UPM on which students organised in collaborative learning groups, solved
practical tasks, specially designed by the teacher with the aid of various video, 2D and 3D
resources.
The practical sessions are structured in two parts: first part under instructor’s guidance and
second part is self directed by students. In the last part of the practical session, selected groups
have performed a peer teaching by demonstrating key concepts to the whole group with available
material testing resources: pictures, videos, documents, web resources, samples with defects, but
also 3D images with defects of materials and process that generating it.
In the material testing courses and practical applications at UPM it is used:



2D images with defects projected on a high resolution monitor, in order students to be
able to have a good appreciation of the defect (nature, shape, characteristics);
3D images, where the defects can be illustrated from different orientations;
non-destructive testing of materials with the standard x-ray analysis, by using a 3D
reconstruction algorithm, regions of interest around the defects is possible to be
reconstructed.
36
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
15.2 Evalauting in a new way
After the theoretical/practical presentations a peer learning assessment is performed by engaging
large student groups in experiences of practical material testing skills in combination with
appropriate real time feedback in order to improve their deep understanding and learning of the
engineering aspects related to material testing.
Most of the higher education institutions in Europe have a few computer laboratories equipped
most of them with maximum 24 computers, on which is possible to run Learning Management
Systems with digital multiple-choice tests/exams for students. Currently is used LMS Moodle.
From another perspective, Smartphones with high resolution become more popular to the
students, the access being conditioned by their price, which becomes cheaper.
The characteristics of the various devices are so different that the issues related to delivering
information and services on the web involve not only presentational aspects, but also structural
and navigational aspects.
It is estimated that mobile device could replace in the very near future the access to evaluation
tests instead of computer interface, with the advantage that it is not necessary to have a lecture
in a room equipped with computers.
The traditional way of teaching face to face does not allow participants in the teaching process:
teachers and students to verify their learning achievements during partial or final exams in a short
period of time and the results are published after a few days, usually 10 days.
Students are interested to be evaluated during semester in order to retract a part of the
curriculum from the final exam.
In this context UPM has participated as higher education institution in the Done-IT with the main
deliverable Peer Learning Assessment System (PELE) for Smartphones. It is an assessment
system, which is designed to enhance students learning.
It is an advanced educational method based on Peer Learning processes, where students learn
from their peers. The efficiency of such processes is difficult to measure by use of existing
Information and Communication Technologies. Also a challenging problem is to consider using
of assessment for learning as a training method, because students are unable to immediately
verify their learning during tests/exams. Universities have access to computer science
laboratories, but these cannot be used during evaluation period for all the campus students. On
the other hand, within a few years ahead a lot of students will have access to cheap mobile
phones with high resolution pressure sensitive screens.
37
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Since 2010, UPM has used in higher education training delivery and evaluation of tests/exams the
mobile devices like iPodTouch in quality assurance training processes.
Usually, assessment is a process that measures student’s performance of learning by reproduction
content. The assessment provides teacher information about:
-
The successes of the teaching;
-
The measure of learning the thought subject by the students;
-
The degree of understanding the thought subject by the students.
This information is collected by the teachers in order to improve instructional methods of
teaching from one generation of students to another, for the benefit of the educational process in
the school/university. The collected information, affects the subsequent classes, helping them to
be where the teacher want to be at the end of the course.
Usually in education, courses are finished with a final exam for assessment of knowledge, as
displayed in figure 1. The dotted line indicates the skills obtained by the students. Such courses
are usually terminated with a traditional (handwritten) exam.
Fig. 1. Acquisition of knowledge in a traditional education course
Evaluation is a process through which instructors/teachers appreciate the quality of teaching by
achievements of the students. Considering the moment of evaluation can be: formative
evaluation by monitoring continuously students learning through ongoing feedback and
summative evaluation, conducted at the end of the course.
Usually, summative evaluations have a high contribution to the final degree, while formative
assessments have low or no contribution to the final degree. Summative evaluation may consist
in: final exam, project, essay, report. Results from summative evaluations may be used to improve
teaching and learning efforts in the next generation of students.
38
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Done-IT implements peer learning assessment solutions through verification or elaborative
learning processes that utilize immediate feedback after tests and/or exams. The traditional exam
(Fig. 1) is replaced with a number of tests (Fig. 2) that provide feedback from the student to the
teacher and from the teacher to student.
Fig. 2. Acquisition of knowledge in a course using several smaller tests in combination with PeLe
The course may be completed with a smaller final exam since the tests contribute to the final
score from the course. PeLe is a new evaluation model where test results for a class are turned
into an active, creative and collaborative learning process by the use of immediate feedback:



The verification feedback leaded by the teacher allows demonstrate the students why a
particular answer is correct and why the others are incorrect;
An elaborate feedback discussion run by students: the answers are displayed but they
don’t know which are the (in)correct ones;
An elaborative feedback discussion led by one student: the deviation from the correct
answer without addressing why this is correct and the other ones are incorrect.
The mobile student evaluation system for Smartphones gives the teachers a new tool to provide
verification or elaborative feedback to students immediately after a test or exam. Students still
remember questions after testing, and in this way they have the opportunity to learn the reason
an answer is correct or not. This method is a collaborative supported learning that helps students
to improve their knowledge in the study subject. Thus, mobile technology provides new
evaluation and testing criteria for education and training.
There are two arenas where one demands special attention regarding the design of the controller
interface:
39
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP


In classroom,
Out of classroom.
The in classroom arena is the most important since this is where one should start and the teacher
should always be in control. Everything is simple and easy to use, just single click events from the
teacher’s finger on the blackboard. The out of classroom comes later when the teacher wants to
administrate several assessments and design course, model for the course and so on.The
traditional reason to introduce electronic assessment is to save time for correcting answers. In the
Done-IT project this aspect has been changed slightly by focusing on the learning perspective:
● establishing a connection between the assessment and learning;
● using an assessment as a part of a learning activity performed by the teacher in such
a way that student learn from the assessments and the faults that are made.
When an assessment has been performed, the students has been working on a problem for a time
and given to the teacher a digitally delivery of the results. The teacher can see the results
immediately and can use these results to create an arena for learning. In figure 3 is depicted the
assessment arena: to the left, a traditional assessment, with the learning arena to the right where
information from the assessment, information from the teacher or peer information can be used
to work out an understanding of the problems assessed.
Figure 3: Relation between assessment arena and learning arena
If the information from an assessment system to the teacher is correct, dependent upon the
results, the teacher can organise teaching in many different ways. With the Done-IT assessment
solution the target is to provide the teacher with instant feedback on the status of the
assessment. Teacher analyses what questions did the students solve correct and what questions
caused more problems. Than he can apply the following scenarios:
40
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
a) Continue as usual and only give the results;
b) Give the students verifying feedback and explain what has been misunderstood;
c) Give the students a hint of what might be the problem for this question, but not the actual
solution;
d) Give the students the results, “this is what you voted” and allow the students the possibility to
discuss the problem;
e) Or to find other ways.
In case of c) and d) the students could be allowed to take part of the test again, or could be
after a group / peer discussion be able to renegotiate their response.
There are several possibilities that teacher is allowed to do:
a) Pick one question, as set up in the assessment, and send it out prepared for an SRS session on
the question;
b) Collect several of the questions after a pedagogical treatment, and allow the students to get a
second chance their answer to parts of the assessment.
The main advantage of entering the test subject just after the test is the problem is fresh in mind
since the student has just been working with it. The student might have spent time on parts of the
assessment but have just not found or understand the right answer. Dependent upon the nature
of the subject being thought, there might be just a hint from a peer student or a teacher that
might solve the problem. The entire idea here is to make the assessment an arena for learning.
There are many methodologically questions rising in such an arena, regarding: how the teachers
will use the system; how the students will like to respond electronically; how the immediate
feedback will change the view on assessment; the frequency and dimension of the assessments;
the focus of the learning culture more on learning than on assessment; the relevance of
continuous evaluation during a course in comparison with the final exam result; the category of
students that benefit from the change in the assessment, etc. The key issue in this approach is to
provide a set of tools that the teacher can use and feel free to apply different educational path in
the learning activity.
15.3
New
type
of
evaluation
supported
by
infrastructure
UPM extended the features of the Peer Learning Assessment Services, and investigated a new
innovative dimension of a test or an exam based up on using 2D or 3D multimedia material for
illustration of defects within material testing, but also 3D reconstruction of defects, using nondestructive techniques.
41
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Questionars designed by the teacher are not only plain text, they require analysis of a real world
situation related to materials defects, that is projected on the 2D or 3D monitors
In the assessment phase picture of the material with the defects is projected by the teacher on a
dedicated 2D high resolution monitor. Differences between some defects consist in minor details,
that have to be evaluated carefully, on high 2D quality images that are projected on the dedicated
2D monitors. But, there are situations when students cannot appreciate the shape of the defect
from the 2D view.
In order to overcome this inconvenieint, in the assessment phase, on the 3D monitor are
presented graphical pictures which illustrate defects in materials that can be watched from
different orientations, in order students to be able to fully understand the shape of the defect. In
a few seconds, rotating the object in different 3D positions they are able to understand the
material defects. The material testing laboratory at UPM is equipped with x-ray equipment that
can be used for non-destructive testing of materials. The 2D images from the equipment are used
in a 3D reconstruction algorithm that reconstructs regions of interest around the defects inside
materials. Then, the defects are evaluated on the 3D monitor on views from different
orientations.
15.4 Conclusions
The first employment of PELE for mobile devices was done over a period of 4 weeks in June 2012
in engineering courses. In the introductory phase of PeLe teachers have read the user manual to
understand how to use the system, and how to log at the system, than have projected the Online
Instructional videos that demonstrate how to: a) Create teacher and student accounts on the
PeLe assessment system (3 minutes long); b) Log on to the assessment services from students'
mobile devices (4 minutes long); c) Use the PeLe (Peer Learning) Assessment Services for
Smartphones, Pad`s, iPod, PC and Mac (the video contains 13 sections and is 11,5 minutes long).
–
These
videos
are
available
on
the
webpage
of
the
project
at
http://www.histproject.no/node/197. These activity lead to a very good satisfaction for the
introductory phase. Students’ feedbacks on the system were collected from a survey given at the
end of the test period. A selection of the results obtained from 97 students appreciates that:
42
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Q9) To what extent do you think the system, as it has been used, may integrate as a natural part
of an exam?: Excellent (28); Very good (38); Satisfactory (22); Not very good (7); Bad (2);
Q9) Integrate system in
exam
40
30
20
10
0
Q9) Integrate
system in exam
Q11) By using PeLe I had a chance to learn from the mistakes: I totally agree (37); I agree (45);
Neutral (9); I slightly disagree (6); I totally disagree (0);
Q11) Learn from the mistakes
50
40
30
20
10
0
Q11) Learn from the
mistakes
43
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Q12) By using PeLe I had a chance to learn from peers: I totally agree (33); I agree (48); Neutral
(15); I slightly disagree (1); I totally disagree (0);
Q12) Learn from peers
60
50
40
30
20
10
0
Q12) Learn from
peers
Q14) To what extent do you feel that the PeLe engages and activate you during evaluation?: Very
large (31); Large (43); Neutral (14); Small (7); Poor (2);
Q14) PeLe activate during
evaluation
50
40
30
20
10
0
Q14) PeLe activate
during evaluation
Very
large
Large Neutral Small
44
Poor
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Q15) To what extent do you think the PeLe can aid students learning of the course curriculum?:
Very large (32); Large (41); Neutral (15); Small (9); Poor (0);
Q15) PeLe aid students learn
50
40
30
Q15) PeLe aid
students learn
20
10
0
Very
large
Large
Neutral
Small
Poor
Q16) PeLe should be used in all classes/lectures: I totally agree (39); I agree (42); Neutral (11); I
slightly disagree (5); I totally disagree (0).
Q16) PeLe should be used in
classes
50
40
30
20
10
0
Q16) PeLe should be
used in classes
45
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Q17) To what extent you consider the 2D and 3D ilustrations help understanding the theoretical
problem/assessment question: I totally agree (54); I agree (42); Neutral (1); I slightly
disagree (5); I totally disagree (0).
Q17) 2D and 3D ilustrations help
understanding
60
50
40
30
Q17) 2D and 3D
ilustrations help
understanding
20
10
0
Totally I agree Neutral Slightly Totally
agree
agree disagree
Q18) The 2D and 3D illustrations facilitate peer learning/assessment processes: I totally agree
(59); I agree (38); Neutral (0); I slightly disagree (0); I totally disagree (0).
Q18) 2D and 3D illustrations facilitate
peer learning/assessment
70
60
50
40
Q18) 2D and 3D
illustrations facilitate peer
learning/assessment
30
20
10
0
Totally I agree
agree
Neutral Slightly Totally
agree disagree
46
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
As a general conclusion we appreciate that:
-
-
the impression for using Peer Learning Assessment Software (PeLe) is very positive and
students agree to use it as an advanced educational tool;
the new model for course delivery and assessment, supported by the ICT equipment
presenting materials characteristics and defects in 2D and 3D views, is largely accepted by
students.
The new innovative dimension of a test or an exam based up on using 2D or 3D
multimedia material for illustration of defects within material testing, but also 3D
reconstruction of defects, using non-destructive techniques, on which UPM extended the
features of the Peer Learning Assessment Services, are largely accepted by students.
16. Literature references
Anderson, L.W. (2001), Krathwohl, D.R., Airasia, P.W., Cruikshank, K.A., Mayer, R.E.,
Pintrich, P.R., Raths, J., Wittrock, M.C., (eds), 2001, A taxonomy for learning, teaching, and
assessing: A revision of Bloom’s taxonomy of educational objectives. Longman, New York.
Nicol, David (2007). E-assessment by design: using multiple-choice tests to good effect.
Journal of Further and Higher Education, Vol. 31. No. 1, Feb. 2007, pp 53-64
Nicol, D. J. & Macfarlane-Dick, D. (2006) Formative assessment and self-regulated
learning: a model and seven principles of good feedback practice, Studies in Higher
Education, 31(2), 198–218.
Boud, D. (2000) Sustainable assessment: rethinking assessment for the learning society,
Studies in Continuing Education, 22(2), 151–167.
Airasian, P. W. (1994) Classroom assessment (2nd edn) (New York, McGraw-Hill).
Scouller, K. (1998) The influence of assessment method on students’ learning approaches:
multiple choice question examination versus assignment essay, Higher Education, 35,453–
472.
Yorke, M. (2003) Formative assessment in higher education: moves towards theory and
the enhancement of pedagogic practice, Higher Education, 45(4), 477–501.
Race, P. (2005). Making learning happen: a guide for post‐compulsory education. London:
Sage Publications
Springer, L., Stanne, M. E. & Donovan, S. S. (1999). Effects of small-group learning on
undergraduates in science, mathematics, engineering, and technology: a meta-analysis.
Review of Educational Research 69 (1), 21-51.
Barnes, D. & Todd, F. (1995). Communication and Learning Revisited. Portsmouth:
Boynton/Cook Publishers.
47
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
Lemke, J. L. (1990). Talking Sience: Language, Learning and Values. Norwood, New Jersey:
Ablex Publishing Corporation.
Mortimer, E. F. & Scott, P. H. (2003). Meaning Making in Secondary Science Classrooms.
Maidenhead: Open University Press.
Cohen, E.G. (1994). Restructuring the classroom: Conditions for productive small groups.
Review of Educational Research, 64(1), 1-35.
M.K. Smith, W.B.Wood, K. Krauter, and J.K. Knight, Combining Peer Discussion with
Instructor Explanation Increases Student Learning from In-Class Concept Questions,
CBE—Life Sciences Education Vol. 10, 55–63, 2011
Andreas Gegenfurtner, Motivation and transfer in professional training: A meta-analysis
of the moderating effects of knowledge type, instruction, and assessment conditions,
Educational Research Review 6 (2011) 153–168
Ericksen, S. (1984). The essence of good teaching. San Francisco: Jossey-Bass.
James, Mark C., The effect of grading incentive on student discourse in Peer Instruction,
Am. J. Phys. 74 (8), August 2006
Dickinson, V. L. and L. B. Flick, Beating the system: Course structure and student strategies
in a traditional introductory undergraduate physics course for nonmajors, Sch. Sci. Math.
98(5), 238–246, 1998.
48