C RE SS T/U C LA Technology-Based Assessment for High-Performance Learning Eva L. Baker UCLA Graduate School of Education & Information Studies Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing International Congress for School Effectiveness and Improvement January 2003 Sydney, Australia C RE SS T/U C LA Premises and Goals Assessment is central to the effectiveness of classroom and distance learning and accountability—some basics What are advances in assessment? How does technology fit? How can technology weave instructional and external testing into a coherent system? Can assessment be cost-sensitive and valid? Many examples throughout C RE SS T/U C LA Order of Topics Basics about assessment CRESST research-based models Attributes, benefits Examples Template 1: Paper and pencil Template 2: Computer Template 3: Authoring Technology benefits and requirements C RE SS T/U C LA About Assessment Role of external tests—accountability, evaluation, and system monitoring Instructional uses of tests in classrooms and schools: diagnosis, modeling, formative assessment as a core teaching strategy Variations in curriculum require strands supporting coherence among goals (standards), content, cognitive demands (what thinking skills are required?) Coherence perceived from student, teacher, administrator, and expert views C RE SS T/U C LA External Exams Instructional Assessments Motivated performance Embedded in learning Time-sensitive Adapted to learners Standardized Extended time Shallow sampling Opportunity to revise Stand-alone Contextualized results C RE SS T/U C LA External and Instructional Assessments Must Be More Coherent Horizontally and vertically Conceptual and psychological linkage For accountability systems, classroom measures may be good supplements C RE SS T/U C LA CRESST Assessment Models Do not start with content Focus on aspects of learning and assessment that transfer from content to content area Multi purpose: both formative learning and outcomes Emphasize student-constructed answers Expert performance defines scoring Research substantiates these models across different subjects and people C RE SS T/U C LA Families of Cognitive Demands: Model-Based Assessment Content Understanding Teamwork and Collaboration Communication Learning Problem Solving Metacognition C RE SS T/U C LA CRESST Assessment Models Research-based, and sites are exclusively classrooms, schools, and systems Focus on cognition and learning Combine content-independent and contentdependent knowledge and strategies Reusable and cost-sensitive C RE SS T/U C LA Examples in Knowledge Understanding and Problem Solving Paper-pencil templates Technology-based administration, scoring, and reporting Technology-supported authoring templates and menus For teachers, curriculum experts, test makers C RE SS T/U C LA Assessment of Understanding Deep understanding of primary source materials or key processes Standard reading as part of task Standard directions Standard scoring rubrics based on experts’ performance C RE SS T/U C LA Hawaiian History Assessment Task: Bayonet Constitution Imagine you are in a class that has been studying Hawaiian history. One of your friends, who is a new student in the class, has missed all the classes. Recently, your class began studying the Bayonet Constitution. Your friend is very interested in this topic and asks you to explain everything that you have learned about it. Write an essay explaining the most important ideas you want your friend to understand. Include what you have already learned in class about Hawaiian history, and what you have learned from the texts you have just read. While you write, think about what Thurston and Liliuokalani said about the Bayonet Constitution, and what is shown in the other materials. Your essay should be based on two major sources: 1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution. 2. What you have learned from the readings yesterday. Be sure to show the relationships among your ideas and facts. C RE SS T/U C LA Excerpts from Hawaiian History Primary Source Documents LILIUOKALANI For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they became wealthy, their greed and their love of power increased. Although settled among us, and drawing their wealth from resources, they were alien to us in their customs and ideas, and desired above all things to secure their own personal benefit. Kalakaua valued the commercial and industrial prosperity of his kingdom highly. He sought honestly to secure it for every class of people, alien or native. Kalakaua’s highest desire was to be a true sovereign, the chief servant of a happy, prosperous, and progressive people. And now, without any provocation on the part of the king, having matured their plans in secret, the men of foreign birth rose one day en masse, called a public meeting, and forced the king to sign a constitution of their own preparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race. C RE SS T/U C LA History Explanation Scoring Rubric 1. 2. 3. 4. 5. 6. 7. General impression of content quality Principles or concepts (DD) Prior knowledge (DD) Use of available resources (DD) Misconceptions (DD) Argument (DD?) English mechanics (DI) C RE SS T/U C LA Mathematics Explanation Task (4th or 5th Grade) Imagine a person from a television station has asked you to give a demonstration on TV. You will be on a show to help other students learn about maths. You are asked to explain everything 10-year-old students should know about fractions. Below are some questions you should try to answer. These are questions that students in the TV audience will ask you. For each question you should draw as many pictures as you can to show what you mean. Then write down what you would say about your pictures on TV. Use as many words and pictures as you need. What is a fraction? Why are there two numbers in a fraction? How many fractions are there between 0 and 1? How many fractions are equal to 1/2? What other important ideas should students know about fractions? Show how you would explain these ideas. Use as many pictures and words as you need. C RE SS T/U C LA Scoring for Maths Task Principles Prior knowledge Resources Misconceptions Argument/explanation C RE SS T/U C LA Technology for What? Most available technology-based assessment promotes efficiency rather than expands the boundaries for measurement of learning Tendency to limit attention to data processing requirements We should use technology to extend our understanding of student accomplishment and program and school quality C RE SS T/U C LA Template 2: Knowledge Representation to Assess Content Knowledge and Problem Solving Same tasks for content knowledge Responses not essay but representation of relationships, hierarchies Scored by expert maps Efficient but expands breadth and depth of knowledge measurement C RE SS T/U C LA History: U.S. Depression (CK) C RE SS T/U C LA Genetics: High Performance (CK) C RE SS T/U C LA Bicycle Pump—High Performance (PS) C RE SS T/U C LA Integrating Knowledge Map with Web Search Strategy C RE SS T/U C LA MAIN Story: Integrating Formative and External Assessments Authoring systems (computer-supported guidance) to help teachers create and share various sorts of assessments intended to measure goals included, not covered by, or that need deeper attention than given in external measures C RE SS T/U C LA Strategy to Link External and Formative Assessments Cognitive demands (domain independent) and content knowledge and strategies (domain dependent) Authoring system allows teachers to assess goals not externally measured, or to connect their assessments to external measures but in a more contextualized setting C RE SS T/U C LA How It Works Teacher will get guidance, using research-based templates What is the purpose of the test? What is to be measured? What content? What conditions? What intellectual skills? Scoring rubrics (based on expert performance) will be provided but can be edited Graphical representation, simulations, and essays are current options C RE SS T/U C LA Assessment Authoring Benefits Common floor on assessments created by teachers Systems start with easy, fixed formats related to learning demands, and as teacher sophistication develops, move to more choices (mix and match) Will allow summaries of student work bubbling up from teachers’ formative assessments to validate the external scores Supports collaboration across different teachers C RE SS T/U C LA Authoring Systems Issues Scored work made public (within the school, with privacy provisions, and among schools) Success depends upon teacher subject matter knowledge, access to needed information, and sharing Success may depend on the realistic link to external examinations Generation of paper- or computer-based tasks C RE SS T/U C LA Minimum Requirements Infrastructure Capacity—subject matter Orientation to learning and to results Congruence with external mandates Availability of smart tools Lead to a culture shift C RE SS T/U C LA Distance Learning Applications High-quality performance demanded No bottleneck in scoring Basis of comparing courses Generates online assessments with instant scoring, feedback to student and instructor Aggregates student work C RE SS T/U C LA Window on the Development of Problem-Solving Template 2— Author Screen Assessment Purpose(s) Diagnostic, readiness monitoring, certification Scenario Context, constraints, situation Problem Characteristics Fix, change usual sequence, improvise step(s), combination Problem Identification Menu Stated, embedded, multiply masked, barriers, inconsistent data from multiple sources, time bound, partial identification, prior knowledge requirements C RE SS T/U C LA Author Screen Template 2 (Cont’d) Macro Planning Menu Explicit courses of action, problem subdivision, backup strategies, help seeking, mix of domain-independent and domain-dependent cognitive strategies Trial and Feedback Menu Data capture of process, process feedback, help, iteration Solution(s) Convergent (right answer), multiple acceptable, partially acceptable, divergent—with scoring criteria, sequential, mixed C RE SS T/U C LA Sample Examinee Screen Components Scenario Problem Information acquisition Macro strategy Micro strategies (domain specific) Solution trials Report of performance C RE SS T/U C LA Sample Examiner Information Time spent on problem Trials to criterion Help access Solution paths Generalizability of solution Acceptability of solution Likely explanation for errors (e.g., lack of prior knowledge) Metric (standards for performance) C RE SS T/U C LA Summary Technology-based assessment needs to extend what we can do Authoring systems can help teachers design better, more sensitive tests and projects Technology can help us share findings Technology-based assessment requires the same evidence of technical quality Demand evidence, not business claims, before you buy