The Monte Carlo Quiz: Encouraging Punctual Completion and Deep Processing of Assigned Readings Author(s): Peter S. Fernald Source: College Teaching, Vol. 52, No. 3 (Summer, 2004), pp. 95-99 Published by: Taylor & Francis, Ltd. Stable URL: http://www.jstor.org/stable/27559190 . Accessed: 10/10/2014 16:17 Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp . JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org. . Taylor & Francis, Ltd. is collaborating with JSTOR to digitize, preserve and extend access to College Teaching. http://www.jstor.org This content downloaded from 132.178.2.65 on Fri, 10 Oct 2014 16:17:39 PM All use subject to JSTOR Terms and Conditions THE CARLO MONTE QUIZ OF AND DEEPPROCESSING PUNCTUAL COMPLETION ENCOURAGING ASSIGNEDREADINGS Peter S. Fernald The Monte Carlo Quiz (MCQ), a single Abstract. item quiz, is so named because chance, with the roll of a die, determines the quiz is administered; (a) whether article, chapter, or section of the (b) the specific assigned reading that the quiz covers; and (c) the par that makes ticular question up the quiz. The MCQ and deep pro both punctual completion encourages cessing of assigned readings and is easy to implement. to address a wide variety of It is readily designed Students' quiz scores learning objectives. is effective. tions suggest that the MCQ and evalua Monte Carlo Quiz With these thoughts in mind, I devel oped theMonte Carlo Quiz (MCQ). The city of Monte Carlo is noted for its gam bling casinos, and the MCQ is so named because chance or one two is Common important. practice When is to postpone assigned readings until the day or night before an exam (Burchfield and Sappington 2000). This loaf-and cram which pattern, wishes instructors' at an alarming increased a passing counter to most expectations, rate. At one compliance, reading versity, by runs and one-time has uni as measured surprise quiz, decreased from more than 80 percent in 1981 to approximately 20 percent in 1997 (Burchfield and Sappington 2000). How ever, ly the reading compliance when enhanced quizzes rate are is great adminis How University Vol. is a professor S. Fernald 52/No. of New Hampshire of psychology in Durham. at the student reads and studies also is important. Ideally, the student studies the and processes deeply carefully and Such studying reading. assigned processing enhances both understanding retention. and shallow or nificant studying on the Superficial rote processing, hand, typically produce learning. and other little, if any, sig Organizing informa et al. 1983) are memory processes of subject known to matter. These findings suggest that instructors should encourage students to engage in such processing. clearly such. two or concepts in the chapter or presented when first you and underline each, or ancillary theses, Identify and, define mention it. Then, show how the concepts or principles in some way(s) are both similar to and from one another. If you different wish, one of the concepts or principles may be selected 3. Application: ple from or discussion lecture, tion hierarchically (Bower 1970), ana (Craik and Tulving lyzing meanings and 1975), applying concepts (Palmere enhance Peter a as them principles article or ideas identifying through related closely secondary tered randomly and periodically out the semester (Ruscio 2001). a of rolling 1.Knowledge: Describe the major thesis, the central idea or set of ideas, in the reading. Make certain that the thesis you identify is primary. Also include 2. Comparison: a student reads and studies the is, (that die) is an important feature of the quiz. Before the semester begins, I prepare several quiz items such as the following: in chapter define or describe how it applies in this reading, course. a concept or princi or article, clearly Select the another it, and then indicate to you or someone you know. Provide sufficient details to justi that the concept or fy convincingly indeed applies as you suggest. principle 4. Critique: Write on some aspect a critical perspective of the chapter or arti 95 3 This content downloaded from 132.178.2.65 on Fri, 10 Oct 2014 16:17:39 PM All use subject to JSTOR Terms and Conditions that prompts you to ele, citing evidence or disagree agree with per that a critique may spective. Note or negative, positive, the author's some be combina tion of both. Your evidence may be based on (1) personal experience, (2) observations of others, (3) reports of others, (4) scientific findings, or (5) logic. When citing evidence, identify the type(s) of evidence you are using. 5. Passion: Citing page number(s), a verbatim tion, confusion, fear, the frustra sadness, anger, pleasure, of quote passage excitement, response: nation brief in you some type of emo that elicits tional or statement surprise, some delight, combi or what aforementioned, ever. Then identify your emotional that response, describe the meaning(s) the statement and provide for your has passage or possible actual Choice: you, reasons five any of the Answer questions. Items 1 to 4 are based on learning (1956). objectives described by Bloom Item 5 is included to enhance students' in the assigned emotional engagement each class a student meeting nomi nated by the class rolls the die up to three times. category system: E (excellent): Demonstrating sound under standing of the selected concept(s), the answer stated is clearly cent. In only one instance did the graders disagree by more than one grade level. accurate. and Method G (good): The answer ismostly or essen or correct but tially incomplete as illustrated the by unclearly stated, following examples: a secondary thesis is not identified (question 1); a concept is not clearly defined or a similarity is or not identified either inaccurate During the fall 2001 and spring 2002 in two semesters, I evaluated the MCQ and Externship. The courses, Counseling (question 2); an example does not pre cisely fit the concept (question 3); type of evidence is not clearly identified an emotional response is (question 4); not but implicit clearly specified or the is not page reference or verbatim hours. (question 5). S (satisfactory): The answer is incorrect, some familiarity yet it demonstrates with and understanding of some por tion of the assigned reading. deep On indicates the "no first roll, quiz" and an an odd number even number dictates that a quiz will be administered. The second roll of the die determines the The grades of E, C, S, and U are equiv to respective percentile TABLE 1. Means of grades and Standard particular question Instructions as follows: included to be in the syllabus are or may not be administered at quiz may or not a quiz is each class meeting. Whether on or article the chapter administered, which and the particular the quiz is based answered will be typically (that is, roll of the die) and occasionally by decree (my deci sion). The quiz is not open book. Therefore, not refer directly to the assigned you may to However, you may reading. bring notes class and Limit quiz. or no more tion 5, be by chance to refer while taking to one the your paragraph sentences. than eight For ques sure to include a in your notes quotation with page I spend one or commenting them answers on, and minutes grading assessed Deviations of Students' Ratings Externships 1. The MCQ 2. My ability was readings 3. If I had reading, each me motivated on to complete 5.35 time. to contribute stu using of the MCQ Counseling SD M .85 4.76 .84 1.28 4.19 .86 .76 4.82 1.47 SD the assigned to discussions enhanced to choose by between of the MCQ. the MCQ the assigned 4.69 a quiz and taking at every class meeting, I would choose the MCQ. in this questionnaire, 4. As employed the acronym MCQ 2.02 refers to multiple-choice questionnaire. on 5. I took notes 6. I recommend of the assigned that the MCQ readings. be used the course. times am disappointed there will be no quiz. of being asked five possibility encouraged Note: Ratings 1.67 5.37 in future me to read carefully chapters/papers. were made 1.33 .77 .93 4.87 .94 .84 4.26 1.20 1.43 2.82 1.77 3.96 1.23 offerings to take a quiz, I some prepared the roll of the die when to class 2.11 indicates 8. The 5.72 5.48 I come 7. When assigned numbers. two were processing ItemM readings to be count the quizzes answered. A question determined on Performance particular assigned reading (for example, chapter or article). The third roll indicates the psy upper-level dent quiz grades. The statistical index for punctual completion was the percentage of quiz answers receiving grades of E, G, or S; the index for deep processing was the percentage receiving the grade of E. The questionnaire included eight rating items (table 1). For each item, students circled one of six numbers (ratings) rang ing from 1 (very strongly disagree) to 6 U (unsatisfactory): The answer is incorrect and demonstrates no familiarity with or understanding of the assigned reading. alent included ed for 25 percent of the final grade in Counseling and 10 percent in Externship. I At the end of the spring semester, a questionnaire in both administered courses (Counseling, N = \6; Externship, N = 11). of reading and Punctual completion no has courses two chology majors (70 percent women and 30 percent men) at the University of New for three and met weekly Hampshire A (absent): Student was absent from class. readings. At experimenting with several grading systems, I settled on the following four quotation for response. 6. Student's above or 95, 85, 75, and 65. Intergrader reliability (agreement between two graders), based on fifty quiz answers (ten answers for each of the five questions), was 78 per After on 6-point different and questions take notes on the 4.62 scales 1.03 = ( 1 very strongly disagree, 6 = very strongly agree). answer. 96 This content downloaded from 132.178.2.65 on Fri, 10 Oct 2014 16:17:39 PM All use subject to JSTOR Terms and Conditions COLLEGETEACHING (very strongly agree). Students indicated their reasons for their ratings in a space provided immediately below each item. Item 4 was included to detect possible bias. response positive Table 2 contains answers quiz from answers meeting for the frequencies each receiving absence class. The I consider here five topics relevant to the MCQ: learning empirical findings, implementation, objectives and memory, adjustments in teaching and learning, and assigned of for punctual were 95.8 and assigned respectively. Table 1 shows means and standard devi ations for students' ratings of the MCQ. (3.5) Ratings higher than the midpoint suggest favorable attitudes. Except for item 4, the control item, and item 7, which concerns disappointment when no quiz was given, the average was above 3.5.1 for each rating examined ratings are quizzes administered item no MCQ, ing and are there no relevant G = good, grades G or rote than one However, cessing. would some foster The cessing. reasonably information 8 211 19 A = absent from class. with in my head. one all of the assigned though only quiz, I completed notes on them, and was always for a quiz. prepared sure everyone to make is a great way does the assigned The MCQ I liked the MCQ. It forced me to do the reading, yet occasionally from taking a quiz. The of having reading with process required a quiz or not was understanding. unique and a for and quiz 3 was there none is shown in table 3. and Memory smart. readings, reading. it provided It ensured that we to known types of processing are retention information enhances to rele two from one to grapple students Information meanings. are concepts pro of the analysis that involves and enhances memory (Craik Tulving 1975). The application question that the use of examples promotes et al. enhance memory 1983). (Palmere cessing meanings to students Requiring elaborate condi tions prompting them to agree or disagree with the author, the critique question took involves some already mentioned: a break did requires question to and different it encourages precise (Bower memory how show similar the of meanings, ration improves Vol. 52/No. yet encourages processing. prepared both in the reading. I would the MCQ have fallen behind The MCQ and critical encouraged reading, note-taking, thinking. nature of the MCQ to do the reading. The random motivation provided always There for taking a quiz, which insured that I did the reading. always was a chance Yet sometimes there was no quiz, which, it was random, gave me a feeling although of control. had pro vant for each quiz question included in the study. The knowledge question students to identify primary encourages and secondary ideas. Such hierarchical Without we deep suggestive, Students preferred theMCQ to quizzes administered at every class meeting (item 3); they were not disappointed when they another; Even might of are that the MCQ inconclusive, enhance the MCQ stuck processing. measure data Different U = unsatisfactory, The MCQ into the class. put excitement I liked the MCQ. It insured that what I read something expect that the task of preparing notes that adequately address the five quiz questions students about a perfor E, of superficial 1970). The comparison Statements of grade The students indicated that they took notes on (item 5) and carefully read (item 9) the assigned readings. By themselves, these activities do not constitute deep pro organization TABLE 3. Selected than quiz suggestive Learning Objectives 45 70 17 11 10 38 68 12 4 51 22 2 805 51 34 35 S = satisfactory, a level more of percent) received mance more students' However, processing. (59.8 comments ATotalU S deep (item 7); and they strongly recommended that I include the MCQ in future course offerings (item 6). Nearly all of the stu dents' written comments about the MCQ were positive. A selected sample of their of Quiz Grades 33 35 11 36 Total 115 E = excellent, most compares (Burchfield and Sappington 2000). It also is consistent with the observation that stu dents motivated by randomly adminis tered quizzes complete assigned readings at impressive rates (Ruscio 2001). I did not include a control group receiv Quiz Note: no when provid E Course Counseling'01 Counseling'02 Externship'01 Externship'02 the prescribed outcome favorably with the 20 percent compliance rate for completing readings assigned ed by students who did not give a rating of 1 for item 4, and I did not observe positive response bias for any of these students. TABLE 2. Frequencies for readings This dates. on half answers Empirical Findings As indicated by the high percentage (95.8 percent) of quiz answers receiving a grade of S or higher, as well as the ratings on item 1, students completed most of the and grade percentage the criteria ity and deep processing 59.8, baseline design and flexibility. Results of data for deep processing reading in college courses. It is the therefore, to determine impossible, absolute level of the effects of the MCQ Discussion measure of with examples. both comprehension This content downloaded from 132.178.2.65 on Fri, 10 Oct 2014 16:17:39 PM All use subject to JSTOR Terms and Conditions principle analysis organization, self-referencing, 97 each and elabo Elaboration and memo and Anderson ry (Bradshaw question passion and requires involves enhance memory 1977). the retention versus on massed Future assess directly processing. at the second class I meeting announce that I will administer a quiz, that the quiz will consist of question 5 (passion), and that the die will be rolled only to determine the reading on which the quiz is based. Once the students have completed their answers, I inform them that the quiz was only for practice. I then invite them class. Most to read answers their students seems so. do to enhance their to the Sharing ease with both theMCQ and one another. At the third class meeting, I repeat the procedure, I have that except students answer question 2 (comparison), which I is the most challenging of the five believe are students some With questions. assistance to grade able from me, their accurately quiz performance and both identify and understand any difficulties they experi enced in answering the the MCQ menting off to a favorable Even ever, with one students the MCQ. student wrote, gets practice most "I students theMCQ. her was rote. When an answer it to the not having I encour time. enough toward note-taking and toward concepts notes (for example, verbatim quotations) that would assist her in answering not only question 5 (passion) but also one or to my the other and Her questions. relief. The difficulty her quiz answer, spent a lot of time reading and taking notes. I still do poorly. I believe, taking. gestions I imag surprise, the sug she case, Thereafter, suggestions. fol spend ing less time and effort reading and taking two only of pages she notes, well on the quizzes. Coaching is helpful for many students. Adjustments performed of this sort stated from With ing 1997; Zachry to However, and activities problem-solving students take they engage in more fewer replac notes. discussions and coll?gial has atmosphere. a more to interactive Students two and cognitive analysis From psychology. para learning major digms, behavioral behavioral analysis, the MCQ borrows the principle of rein forcement (Skinner 1953). A high quiz on is contingent grade careful reading, students' studying, and note-taking, punctual question-driven other activities. learning in the section on memory and learning objectives, principles from cog are psychology The cessing. but one five set of for useful types of ing particular promot information questions pro are presented Other questions. learning and information processing objectives can be addressed through other questions. An instructor willing to take the time and can effort a create question designed a particular form of reflective to each assigned read relevant thinking of my One course. moves colleagues Students infor answers type for all five questions. On the day of the quiz, the roll of the die determines which answer stu will for submit er form the MCQ an with dents ing on Whatev evaluation. takes, it provides structure ongoing stu and for thinking about and reflect guidelines the assigned readings. The advan tages of such guidelines are well docu mented (Ausubel 1960; Corkill 1992). also can be used with The MCQ asks struggle with the complexity of themateri classroom and Flexibility are more engaged with both the subject matter an MCQ includes week, ask more thinking, questions, a greater demonstrate willingness reflective al. The that activities to reading. Anoth unrelated assignments er of my who his stu colleagues, requires a public to attend dents lecture every 1985). lectures, classroom lecture. The MCQ is not a singular technique. Rather, it is a set of principles taken from dents process lecturing and implementing classroom designing activities that promote reflective thinking and problem such solving. Designing activities requires teaching ingenuity unlike that required to prepare and deliver a lecture (Giordano and Hammer 1999; Meyers the Design ing. information otherwise, the various mation processing beyond note-taking to formal writing in her child development in Teaching and Learning to encouraging ing or, in to elicit With continued use of theMCQ, I expe rienced a gradual and profound evolution inmy teaching from transmitting informa tion of replace nitive information deeper Whatever my that my course, encouraged processing. lowed of require guidance As outlined reaction to be one of seemed suggestions students and encouragement, psychology's and principles of particular interest to her, and I suggested that she take full advan tage of this opportunity. Lastly, I suggest ed that for some readings she might take of Some the five answering efforts note-taking another. adjusting to this new learning environment (Hart?n et al. 2002). Informing them about the different types and levels of informa tion processing helps them understand the rationale for theMCQ and take full advan tage questions. I pointed out that questions 2 through 5 permitted (and even encour aged) her to direct her reading, studying, and how quizzes, experience Below Imple the two practice start with two after some question. with I discovered, quizzes, How detailed. one and support, ine, had to do with her belief that I was telling her to be less thorough in her note Students initially are quite anxious about the MCQ. To ease them into the procedure, and note-taking to write notes student's chance-determined question, she did not know where to begin, and she complained surprise Implementation answers time more levels of the types and the aged her to take fewer notes and to direct should studies was about 1996). distrib uted practice suggests that knowledge acquired over many small regular study intervals (for example, weekly quizzes) is retained more readily than knowledge acquired in one study interval (such as for a final exam only) or a few very large intervals (Dempster 1988; Glenberg 1992; 1996). Leeming 2002; Payne andWenger The MCQ clearly incorporates proven of principles cognitive processing. support for MCQ-prompted Empirical deep processing, however, has yet to be demonstrated. her ever, of required by and Shakesby (McDaniel, Waddill, research and ten entire Covering pages, large-size were comprehensive which outlines enhance the MCQ." lined Kuiper, (Rogers, can MCQ, Finally, of both Preparing readings, a practice assigned I hate self-referencing affect, Kirker 1982). The to students question on the that lecture. I have not yet included such a Although question, report for my considered addresses respective the externs' agencies. 98 This content downloaded from 132.178.2.65 on Fri, 10 Oct 2014 16:17:39 PM All use subject to JSTOR Terms and Conditions course Externship including a question experiences With I have that at their the decree COLLEGETEACHING option, when other instructional activities take the priority, can instructor not elect to roll the die, that is, not to administer a if students' learning is better quiz. Or, served the otherwise, can instructor decree a quiz without the roll of the die. A flexible procedure, theMCQ is readily to address a wide variety of designed and learning objectives. teaching words: Key evaluation, quiz, learning New York: els of processing. Journal of Verbal 21:165-74. ing and Verbal Behavior and J. Sappington. C. M., 2000. with required reading assign Compliance ment. 27:58-60. Teaching of Psychology to thank Benassi, Russell Jennifer case Kosits, Edward Martin, Tracey Stine for and William Root, of the manuscript and reading with assistance Victor Fernald, Dodge Perkins, Michael O'Brien, their critical data concerning to Peter analysis. this article Correspon should S. Fernald, Conant of New Hampshire, Durham, University or e-mail 03824, psf8@cisunix.unh.edu. be Hall, NH D. in P. use of advance orga and of retention learning Journal verbal material. of Edu 1960. The the 3 A to apply the results research. American Psy psychological 43:627-34. chologist P. J., and E. Y. Hammer. 1999. In Giordano, class collaborative Practical sug learning: from the teaching trenches. Teach gestions ing of Psychology A. M. Glenberg, effects. 26:42-44. 1992. Distributed In Encyclopedia of R. Squire, practice and learning 138-42. New H. C, D. S. Richardson, E. B. Ricar J. Rockloff, and B. Latane. 2002. do, M. A Focused tool for interactive learning: active chology Leeming, dure class discussion. Teaching of Psy 29:10-13. F. C. improves 2002. The proce exam-a-day in psychology performance and P. S. 1996. Study interest, Shakesby. strategies, text: The application and learning from of In Basic appropriate processing. in research: memory applied Theory ed. D. J. Hermann, C. C. McEvoy, context, P. Hertel, vol. and M. K. Johnson, Herzog, material activities for psychology classes. Teaching 24:105-15. of Psychology S. L. Benton, J. A. Glover, and M., Palmere, R. Ronning. 1983. Elaboration and recall of ideas main in the failure Hart?n, meaningful cational 51:267-72. Psychology B. S. 1956. Taxonomy Bloom, of educational I: Cognitive domain. handbook objectives, Vol. 52/No. of study ed. L. memory, York: Macmillan. REFERENCES Ausubel, nizers in episodic I wish addressed of and Journal memory. of Experimental General 104:268-94. Psychology: F N. 1988. The effect: Dempster, spacing 29:210-12. of Psychology P. J. Waddill, NJ: Erlbaum. 1, 385-97. Mahwah, S. A. student partic 1997. Increasing and productivity in small-group ipation 4:33-72. and E. Tulving. 1975. Depth the retention of words Teaching M. A., Meyers, Facil Corkill, A. J. 1992. Advance organizers: itators of recall. Educational Psychology F. I.M., McDaniel, and Learn Burchfield, Craik, NOTE dence in factors 1:18^4-6. Cognitive Psychology G. L., and J. R. Anderson. 1982. Bradshaw, as an explanation Elaborative of lev coding processing Feenstra, Donna classes. McRoy. 1970. Organizational memory. Review objectives David G. H. Bower, Psychology Payne, D. G, tice effects in prose. Journal 25:898-907. and M. of Educational J. Wenger. in memory: unanswered questions. research: Practical memory ed. D. J. Hermann, C. McEvoy, P. Hertel, and 123-37. Mahwah, 1996. Data, In Basic Prac and theory, and applied applications, C. Hertzon, M. K. vol. 2, Johnson, NJ: Erlbaum. T. B., N. A. Kuiper, and W. S. Kirker. Rogers, 1977. Self referencing and the encoding of information. Journal personal of Personal ity and Ruscio, dom Social Psychology 35:677-88. J. 2001. Administering to increase student's of Psychology B. F Skinner, New behavior. quizzes reading. at ran Teaching 28:204-06. 1953. Science York: Macmillan. and human I kicked 1985. How the lecture Zachry, W. H. in psychology. habit: teaching Inquiry 12:129-31. Teaching of Psychology 99 This content downloaded from 132.178.2.65 on Fri, 10 Oct 2014 16:17:39 PM All use subject to JSTOR Terms and Conditions