Looking at Kalie’s Test Information visualization for cognition amplification Michael Griffin, 6 June 2002 ED299X: Visualizations for Learning, Dr. Roy Pea Introduction Kalie’s Test, developed as part of an NSF-funded project called Primes, and my Master’s Project in Stanford’s School of Education, is a workshop designed to introduce parents of middle-school aged children to topics in standardized testing. In the workshop, parents follow the story of a young girl named Kalie, and learn about the topics through structured exploration of an interactive model of factors related to student performance on tests. Parents apply new knowledge to scenarios about Kalie, role playing activities, personal reflection, and group discussion. To provide a bit of background, Primes is a project run jointly by Shelley Goldman at Stanford University, where I’m a student, and Jennifer Knudsen at WestEd, where I’m also an employee. The project’s goals are to ‘bring parents in the door’ of their children’s school experience. Each Primes experience is intended to build parents' confidence about their own math skills and gain some insight into the innovative teaching methods that schools are beginning to adopt. Primes is designed specifically for parents of middle school aged children in underserved communities. The project consists of 9 workshops (of which Kalie’s Test is the last), plus a TV special and a Parent Guide, any of which can be used independently or as part of a series. Primes workshops tie important math concepts to topics parents find interesting and motivating. In this paper, I intend to document the ways in which the interactive model serves as a visualization of information that amplifies cognition, as per the framework developed by Card, Mackinlay and Schneiderman at SRI International and the University of Maryland (Card, et. al., 1999). I’ll also trace the influence of other relevant theories of visualization as they are manifested in the design of the software. 1 Workshop description and context of use As mentioned in the introduction, Kalie’s Test is one in a series of 2-hour workshops developed as part of the Primes project. Primes has the overarching goal of helping parents of middle-school aged children become more involved in their children’s education. To achieve this, workshops strive to highlight four messages for parents: 1) Parents can do math and, indeed, do it everyday, 2) The math parents do is important in middle school, 3) How mathematics reform is changing the look of today’s classrooms, and 4) How parents can help their children in school and in math. Most Primes workshops present math activities within an interesting or motivating context with the goal of building parents’ confidence in their ability in the domain while highlighting their links to their children’s school experience. Motivating contexts have included car buying, planning lunches and playing games of chance, to name a few. In Kalie’s Test, the motivating context of standardized testing has become the primary learning content for the experience, with math taking a less highlighted role. Primes workshops are designed for use in a wide variety of settings, anywhere parents might be likely to congregate. While schools and teachers can and have hosted workshops, it’s especially important that people in nonschool settings are able to participate as well, settings such as community centers, schools, churches and workplaces. Kalie’s Test is designed specifically for use in a live group setting, in a computer-equipped facility. Materials will be available for download over the Web, but facilities don’t need live internet connections during the workshop. Because of the importance of making Primes available in non-school settings, workshops are designed for facilitators with little or no special knowledge in the domain being presented (e.g., math, or in this case, standardized testing). They’re intended to act as workshops-in-a-box, with expert knowledge typically embodied in video segments, paper handouts and booklets. In the case of Kalie’s Test, this expert knowledge is embodied in an interactive model, built in Macromedia Flash, which plays a central role in the 2 workshop activities and is available to individuals and small groups at computer workstations. Learning problem and learning goals Standardized testing is a difficult subject for parents to find clear information about. Discourse about testing, primarily found in the media and in political debate, is partisan and controversial. Piecemeal treatments of fact are often used to support opinions about testing, and are used to make cases for and against the use of standardized tests. I believe that it’s important for parents to have a schematic understanding of the factors that influence test results, in order to 1) make sense of this discourse, and 2) to engage in conversations with teachers and administrators about how test scores will be used to make decisions about their own children. To address these problems, I believe parents need to be able to understand the factors that influence test results, develop informed opinions, flexibly accommodate new information, develop and defend arguments, and engage in discourse on the subject. The role of visualization in the workshop Kalie’s Test features a number of activities designed to allow parents to learn concepts and practice skills they’ll need to do this. While 2 hours is not enough time for novices in the domain to become experts, it should be enough time for parents to develop schema with which they can continue to learn about the topic, and develop some important initial insights. The workshop activities are divided into 6 sections: introduction, introductory software activities, scenarios, role-plays, reflection, and group discussion. A significant ingredient in the workshop is an interactive model that allows parents to explore many of the complicatedly interrelated factors that affect student performance on standardized tests. In each segment, parents make use of the interactive model to complete answer questions and make decisions that arise from the activities. In exploring the model, parents can discover for themselves the semantic relationships among factors. Parents then use this understanding in group activities to make, critique and defend interpretations of Kalie’s and other children’s test scores. 3 Parents are asked to literally enter into Latour’s ‘agonistic situation’ in this workshop, because that is what they need to do in the real world in order to work with teachers and administrators to make smart decisions about their children’s academic career (Latour, 1986). This visualization becomes an inscription that allows parents to learn these concepts more clearly. Latour’s theory also suggests ways in which the model might be adapted to make it useful as a tool for argument and discussion outside of this workshop, in real interaction between parents and teachers, for example. This possibility will be further explored in the summary of the paper. Figure 1. Integration of the model in workshop activities. Description of the interactive model Before discussing workshop further, it’s important to understand the features of the interactive model that plays such an important role. The model, still in development, is intended to represent an expert view of the world of standardized testing. In these ways, the model serves as the type of visualization described by Gordin and Pea for scientific visualizations (Gordin and Pea, 1995). It incorporates lots of data, resembles the phenomenon it describes, it’s holistic, takes advantage of visual properties to code meaning, it’s animated, it allows for high-speed computation, and it’s spatial. The model has two primary benefits. First, it allows learners to understand the many factors at play when interpreting standardized tests, providing them with a schema in which to organize many otherwise disparate ideas. It bypasses the need to understand technical jargon by presenting the 4 relationships visually, rather than describing them; the effects are quickly evident. In Gordin and Pea’s terms, these attributes allow it to be more accessible to learners, provide a means for authentic inquiry, is empowering, and enables critical and informed citizenry. Further, following the writing o Bruno Latour, it provides a tool for informed argumentation for parents (more on this later).Second, this model off-loads expertise that would otherwise be required by a workshop facilitator. This frees the host to be a participant in the learning activity, rather than a defender of a point-of-view. Figure 2. The software interface at full complexity. See the appendix at the end of this paper for larger, more legible views of the interface in various states. 5 Key features There are four dynamically linked areas in the interface of the Kalie’s Test interactive model: 1) the student distribution area, 2) Kalie’s results area, 3) the testing factors/sliders area and 4) the knowledge map. Figure 3. The main areas in the interface Principles outlined in Ramana Rao’s very readable See and Go Manifesto (Rao, 1999) figure prominently in the design of this interface and will be highlighted where appropriate. Overall, this model exhibits three of the principles prominently: putting graphics first, animating the transitions, and creating stable and consistent spaces. The interface is based on graphics (seeing Kalie and her classmates, for instance, rather than lists of words, numbers or dialog boxes). Further, since all of the elements are dynamically linked, animations are used extensively to show how and to what extent changes in the model are taking place. Last, the interface always shows the same 4 6 areas in the same relationship with each other. As users become more familiar with the interface, they learn they can rely on these relationships. In the current stage of the software, only the sliders are directly manipulable by users. Parents can move sliders up and down. Changes in the sliders cause changes in the knowledge map and in the overall distribution of student results. Changes in the knowledge map affect Kalie’s placement in the distribution. Finally, Kalie’s results are also reported numerically. Figure 4. Propagation of effects in the model 1. Student distribution area The distribution of students’ results is composed of simplified children’s faces. All students with the same score are stacked on that score. This natural sorting shows how statistical patterns are manifested from physical phenomena, giving learners a point of entry into understanding how to interpret a fairly complex histogram. For norm-referenced tests, results appear as a normal (bell curve) distribution. Criterion-referenced appear less regular, highlighting one difference between types of exams. Kalie can be identified as the only with child with a fully completed face, familiar to workshop participants from the beginning of the workshop. Above Kalie’s head is a bracket that represents the reliability of her test score, which emphasizes that her results are not ‘set in stone’ for the test. Kalie’s classmates are identifiable by the tufts of hair on their heads. Users can switch between seeing the entire population of about 250 kids or just look at the 15 kids in Kalie’s class. This allows learners another way of seeing how the histogram was built, of seeing the effects of local curriculum, and of 7 seeing how effects generalize and form clearer patterns with larger sample sizes. For norm-referenced tests, the ‘average’ region is shading, representing the yardstick against Kalie is being measured. For criterion-referenced tests, the cut score is shown instead. The culture (or race) of kids is evident as well, allowing the effects of cultural bias on score results. Changes made in the model dynamically affect this area in a variety of ways. Using the grain of the wood. The distribution of kid’s scores embodies the grain of the information it’s showing, conceptually and mathematically. Spotlight results of computation in the same space. All of the effects users can control are distilled into a few visual representations, including the distribution area. 2. Results area Kalie’s results are shown for norm-referenced tests in terms of total answers correct, percentile rank and placement relative to average. For criterionreferenced tests, the area displays total score, and achievement against a cut score (pass/no pass). These are dynamically updated whenever a change is made to the model that affects Kalie’s results. Show results of computation in same space. All of the effects users can control are distilled into a few visual representations, including the results area. 3. Knowledge map This area, based on the concept of the Venn diagram, represents the intersection of knowledge different types of knowledge that influence Kalie’s score in testing. Circles represent the knowledge Kalie has in math, the knowledge embodied in her math curriculum, and the math knowledge tested. For criterion-referenced tests, there is an additional circle for the math knowledge embodied in standards. Sliders control the relative positioning, overlap, and size of the circles. The most important circles in the diagram are the ones representing Kalie’s knowledge and the knowledge covered by the test. Overlap of these circles determines Kalie’s test score. Other circles can be pulled closer or pushed further from each other to represent relative 8 alignment. 4. Sliders/testing factors area The sliders/testing factors area is the region over which users have direct control. Sliders representing testing factors determine the outcome of Kalie’s test. Factors exist in groups related to Kalie’s knowledge, Kalie’s curriculum, the test itself, and the performance of other kids (for standardized tests) or standards (for criterion referenced tests). The following is a list of factors included in the current model and their effects. Kalie Kalie’s math knowledge: increases/decreases area of Kalie’s knowledge in the knowledge map School skills: increases/decreases alignment of Kalie’s knowledge with curriculum in the knowledge map Test taking skills: increases/decreases alignment of Kalie’s knowledge with the test in the knowledge map Luck, health on test day: increases/decreases Kalie’s results Test What the test measures: increases/decreases area of test in the knowledge map Difficulty level: increases/decreases Kalie’s results, shifts distribution of overall results down/up Cultural bias: dial selects culture to be favored, slider boosts results of selected kids to greater/lesser extent Reliability of scores: narrows/widens bracket over Kalie’s head Curriculum Amount covered: increases/decreases area of curriculum in knowledge map Teaching to the test: increases/decreases alignment of curriculum with the test Other Kids (norm-referenced tests only) Performance: shifts results of other kids up/down, while leaving Kalie’s results as they are Standards (criterion-referenced tests only) 9 Amount covered: increases/decreases area of standards in the knowledge map Difficulty level: decreases/increases alignment with Kalie’s knowledge in the knowledge map Favor direct manipulation. As mentioned above, this is the input area of the model, and the area in which direct manipulation is facilitated. More discussion of direct manipulation his can be found in the summary of this paper. Understanding the effects of each slider takes some thinking, in large part because of the interdependency of variables that aggregate to create the reduction that is Kalie’s test score. Indeed, being able to explain these relationships is one of the main learning goals of the workshop. The following diagram helps clarify these interrelationships. Figure 5. List of possible influences within the knowledge map In the diagram above, some relationships are highlighted and others are grayed out. Highlighted relationships are captured in the model, while gray 10 ones are excluded, usually because they’re highly unlikely. In creating the model, much consideration was given to deciding which factors needed to be included. These decisions are still being made. Another important aspect of this model is the weighting given to each of the factors that is included. We’ll be working with experts on testing to make decisions about this. But we’ll also be making the weightings that we decide on, and the rationale and research that supports those decisions available to all users within the model. This glass box design is essential if the model is to have any credibility in such a contentious domain. Description of workshop activities Introduction. The workshop begins with a 20-minute introduction to standardized testing, in the form of a text and image presentation, along with brief video segments. Topics covered include the workshop plan, an overview of the testing system in the U.S., perspectives of various stakeholders, the benefits of testing, and hot topics. We found in early discussions with parents that many came to the table with questions about testing that come directly from reporting in the popular press. We wanted to make sure that the workshop addressed a wide variety of parents’ concerns, so it seemed essential to give explicit attention to hot topics when it is possible. This sentiment is backed up by the work of Edelson, Gordin and Pea (1999) in their writing about the geographic visualization software WorldWatcher. In developing curriculum activities for the software, they found it was important to identify a motivational context for learning, topics that were meaningful, controversial and open for debate. The topics were also well known to learners, and had direct implications for learners. Certainly, the entire topic of standardized testing fits that bill, but it’s important to focus on specific topics, such as ‘alignment’ or ‘opportunity to learn,’ in order to get traction for learning. These topics are analogous to the ‘greenhouse effect’ in geography chosen by Edelson, et. al. Software activities. Parents next engage in a series of activities that allow them to become familiar with the interactive model, and to begin to develop their understanding of the domain. To construct a model that reflects the complexity of a topic like standardized testing, even one that does so 11 parsimoniously, results in a representation that is necessarily daunting to novices in the domain. To address this issue of accessibility, we devised a number of ‘staging activities,’ again based on the work of Edelson et. al. with WorldWatcher. In these activities, which serve as a tutorial and initial foray into the topics, parents use a ‘training-wheel’ version of the interface, and focus on a limited set of relationships to answer some initial questions, such as “What happens when Kalie’s knowledge in math increases? To her score? In the knowledge map? Does she get a passing grade?” Parents work in small groups and keep track of their findings on preprinted worksheets. Figure 6. The training-wheels interface. The advantages of the training wheels interface are that it allows learners to see how the histogram is assembled, by focusing on a limited number of cases (Kalie and her 14 classmates). Understanding this small set allows 12 learners to understand the larger distribution when it appears. By focusing on a small number of relationships among factors, learners develop the core of a mental model, which can be made more complicated over time. This ‘building process’ will be emphasized for learners by animating the transitions as new factors are introduced and attached to the existing model, again following Rao’s See and Go Manifesto (Rao, 1999). Scenarios. Learners will watch brief videos that set up dilemmas in which Kalie’s involved. For example, Kalie normally does well on school math tests, but she’s done poorly on the math section of a standardized test. Again working in small groups and using a worksheet to structure their inquiry, parents will try to build a rationale for Kalie’s confused parents. Kalie’s story, in this case provides a common meeting ground for parents to discuss personal and sometimes emotional concepts. Role-plays. Similar to the scenario activities, but this time parents take notes individually, then on the roles of various stakeholders to debate what rationale seems best explains dilemmas, from their point of view. This activity gives them practice in building rationales on their own, arguing using the model and seeing complex issues from others’ points of view. Reflection. After these group activities, parents have a chance to reflect for themselves about what they’ve seen, heard and learned about tests, and to think of the ways they want deal with testing for their children. Group discussion. At the end of the workshop, there is time for participants to share what they’ve learned, and a brief video segment that summarizes some of the main themes of the workshop. Parents leave with a small booklet of information and links to more information on the web. Information visualizations for cognitive amplification In this section, I’ll analyze from the perspective of Card, Mackinlay, and Schneiderman’s framework for information visualizations designed for cognitive amplification (Card, et. al., 1999). According to their framework, information visualization is “the use of computer-supported, interactive, visual 13 representations of abstract data to amplify cognition” (Card, et. al., p.7). More specifically, this model serves as an “active diagram,” one that “amplifies the effect of a good visual representation by making it interactive” (Card, et. al., p. 8). Cognitive amplification, again by their definition, occurs when learners/users can use the visualization to: forage for new and relevant information, search for schema, instantiate schema, problem-solve, and author, decide and act. I’d argue that all of these occur in Kalie’s Test when the visualization is activated in a curricular context. In each section of the workshop, parents do each of these things. Parents use the model to amplify cognition in their pursuit of specific goals. In the first training wheels activity, for instance, learners are posed a simple problem: explain what happens when Kalie’s knowledge in math increases. In this first activity, parents learn to be attentive to certain types of phenomena (forage), identify patterns (search for schema), begin to explain patterns (instantiate schema), and generate descriptions (solve problems and author solutions). In later activities learners rely on the model to inform interpretations of testing situations (forage and search for schema), and turn those findings into defendable positions (instantiate schema, solve problems and author solutions). They build from simpler problems to more realistic, independent situations, in the end gaining experience that allows them to use their understanding in authentic, social contexts. In performing each of these activities, learners take advantage of the affordances of the model, as described in the following list. Increased memory and processing resources High-bandwidth hierarchical interaction: The model is high bandwidth in that it presents lots of data, including Kalie’s results, her position in a distribution, influences of various factors and issues of alignment that lie at the root of the situation. The model is ordered into areas, maintaining relationships among objects that help users organize the information in the presentation. For a 14 simple example, learners come to realize that in the distribution, columns of kids have the same score. This physical positioning and relative positioning giving meaning top the display. Parallel perceptual processing: This principle is not especially prominent here, but it does exist. For instance text descriptions of Kalie’s results can be interpreted in parallel with the visual display of Kalie’s place in the distribution. Offload work from cognitive to perceptual system: Certainly the display affords this phenomenon. For instance, users can see Kalie’s results visually, without interpreting her placement through semantic channels. Expanded working memory: When solving an extended problem activity in the workshop, the persistence of objects allows parents to plan actions (experiments or trial settings), see the results and try again, reducing their memory load. Expanded storage of information: The model allows for the storage and presentation of lots of information. Reduced search for information Locality of processing: Because information is grouped together into areas and subsets within areas, less search is required to find patterns and relationships which are important in learning the content of the workshop. This is likely to be especially true for experienced learners. High data density: Lots of data are represented in a single, small space. Spatially indexed addressing: Because effects are propagated from area to area within the interface, it wasn’t practical to keep all related objects in tight proximity. For instance, while Kalie is represented in the distribution, it made more sense to group the sliders for factors related to her in the area with their sliders than have them located near her in the distribution. Labels, reinforced by consistent color-coding overcame this barrier, tying like factors together and relating them to each other within the knowledge 15 map. Spatial addressing was taken advantage of in some areas, such as the reliability bracket above Kalie’s head in the distribution. Enhanced recognition of patterns Recognition vs. recall: In the model, direct manipulation affords this phenomenon. Parents can ‘see’ changes as they happen, and can repeat them when they want, so they don’t have to memorize which relationships obtain among objects. Abstraction and aggregation: Abstraction in the model allows parents to be attentive to patterns and functional relationships, rather than raw data. Visual schemata for organization: The patterns and relationships parents notice allow them to develop schemata that are visual rather than semantic. Value, relationships, trend: These patterns are enhanced at all three levels. Perceptual inference Visual representations make some problems obvious: For instance, the difference between criterion-referenced and norm-referenced distributions is immediately apparent. Graphical computations: maybe when it’s turned into an analysis tool? Sugg for imp… Perceptual monitoring: The model allows multiple changes to be seen at once, through movement and changes in the appearance of the display. Manipulable medium: Because the model is directly manipulable, the model allows exploration of a space of parameter values. More discussion of direct manipulation follows in the ‘future improvements’ section at the end of this paper. Summary Overall, I’d say that while the model was developed without the benefit of attention to many of the models discussed in this paper so far (with the 16 exception of the idea of creating bridging and staging activities early in the workshop, an idea borrowed explicitly from the Edelson, et. al. description of WorldWatcher), it fits extraordinarily well within the frameworks they define. What suggestions for improvement do principles for visualizations suggest? Looking at Rao’s framework, I’d say that there are two areas where the model is relatively weak: in providing the ability to focus while maintaining context, and in affording direct manipulation. In further developing the model, I believe that we’ll need to pay more attention to the focus-context issue: affording users the ability to zoom in on areas of interest, while maintaining a global view, especially for examining the distribution. How to do that will require reflection, but ideas for resolving the problem may lie in the issue of direct manipulation. Direct manipulation is a tricky problem. The model affords direct manipulation over a limited set of objects, namely the sliders. We intentionally avoided letting people directly move objects on the knowledge map and distribution. This is because we wanted to force learners to make sense of the mechanistic relationships in the model. If users were to simply move Kalie’s knowledge around on the knowledge map, how could one make a statement about the way in which it moved? How would the sliders adapt to accommodate such free movement? Similarly, what would happen if Kalie’s knowledge could be moved freely on the map? One way to accommodate this type of action that’s currently being considered would also expand the usefulness of the model. If users could fix test scores, along with Kalie’s race and school achievement, then they could maximize or minimize individual factors (or drag the alignment of types of knowledge in the map) and see how other factors and relationships would have to deform to accommodate such changes. This might help users after the workshop model their child’s situation to point to possible ways of explaining and exploring, for instance, why their own child might be doing poorly on tests. It would help in building rationales, and could be used for diagnosing real-life situations, and pointing out places to look when resolving real-life dilemmas. 17 Bill Finzer’s work with Fathom points to even more possible ways of exploiting direct manipulation (Finzer, 2000). Fathom features can dynamically deform any representation simply by grabbing and dragging on it. This kind of freedom of presentation removes the screen real-estate problem from the design of interfaces. Applied to the model, users might be able to zoom in on Kalie and her neighbors to inspect relationships more carefully, or stretch the knowledge map to make it more prominent, or drag and drop new sliders into or out of the model to make them simpler or more complex. This type of dynamism is especially well suited to a vector-based development platform like Flash, and seems to represent some very exciting possibilities for this summer’s development plans. Aside from dynamic manipulation, the model seems to address Finzer’s other principles well, including touch (avoiding modes), making math useful, outreach to other disciplines. After this workshop is developed, there has been talk about developing similar workshops for teachers and other stakeholders, addressing the idea of ‘designing with teachers in mind.’ I believe one further area of improvement would make for a stronger product as well. Bruno Latour’s perspective on the usefulness of visualizations as inscriptions goes to the heart of the learning goals of the project. The situation of a parent sitting with institutional representatives to decide on the meaning of a test score in relation to a child’s academic progress is a clear example of the agonistic situation Latour focuses on. More thought needs to be given to making the visualization useful as an inscription for parents. Goals would be to make the representation: flat, mobile, immutable, scalable, reproducible, superimposable, importable into written texts, and mergeable with geometry. Many of these are already properties of the model. But because the product was designed for the web, it’s not always available in situations where it might be needed. Paper serves parent-teacher conferences better than the web. To that end, we should consider adding the ability to print out screen shots showing various configurations of factors that represent several interpretations of a child’s situation. It would be important to highlight the weightings (and research supporting the weightings, and experts 18 supporting the research) to make it an acceptable submission for skeptical teachers (and to address the issue of reproducibility). Many other improvements are slated for review as well, including adding specific factors, refining the interface with user studies, expert vetting of the model, and allowing the weighting and addition of factors to be controlled by users (especially education researchers). These considerations, while important to the project, are beyond the scope of this paper. 19 Bibliography Card, Stuart, MacKinlay, Jock D. (Editor), Shneiderman, Ben (Editor). Readings in Information Visualization : Using Vision to Think (Morgan Kaufmann Series in Interactive Technologies) 1999. Chapter 1: Information Visualization. Edelson, D. C., Gordin, D.N., & Pea, R. D. (1999). Addressing the challenges of inquiry-based learning through technology and curriculum design. Journal of the Learning Sciences, 8(3&4), 391-450 Finzer, William. (2000). "Design of Fathom, a Dynamic Statistics Environment, for the Teaching of Mathematics" Paper presented at ICME 9. Gordin, D. N., & Pea, R. D., 1995, Prospects for scientific visualization as an educational technology: Journal of the Learning Sciences, v. 4, p. 249–279. Latour, Bruno (1986). "Visualization and cognition: thinking with eyes and hands." Knowledge and Society, 6 p. 1-40. Rao, Ramana (1999). See and Go Manifesto. Interactions, Sept-October 1999. 20 Appendix 21