Cognitive Systems Tutorial 1 / October 22, 2012 Prof. Dr. Sven Bertel / JP Usability Today’s topics ● Organizational issues & literature ● Introduction into cognition ● Discussion of readings for today: ● Topical paper: Introduction to computational cognitive modeling [Sun, 2008] ● Application paper: Effects of Compressed Online Foveated Video on Viewing Behavior and Subjective Quality [Nyström & Holmqvist, 2010] (Useful slides relating to background for second paper are collected below.) Organizational issues Lectures & Tutorials: Times & Locations ● 14 lectures + 7 tutorials ● Lectures mostly on Wednesdays, 3x on Tuesdays ● Tutorials always on Tuesdays ● Tuesdays: 1.30-3pm Seminarraum 014, Bauhausstraße 11 ● Wednesdays: 9.15-10.45am, Seminarraum 102, Marienstraße 7B This is new. Concept. ● Mix of organization by topic and organization by application/domain ● Putting techniques and methods to use later will be within application contexts, problemcentered ● Rather open format, more discussions, feedback; hands-on, group-based activities ● Regular reading assignments, likely short position papers This is new. Concept. ● Example: first reading assignment ● Topical paper: Introduction to computational cognitive modeling [Sun, 2008] ● Application paper: Effects of Compressed Online Foveated Video on Viewing Behavior and Subjective Quality [Nyström & Holmqvist, 2010] ● Papers, slides, assignments, other resources to be made available online on course page at www.uni-weimar.de/usability ● login: q0gnit1oN passw: rul3Z This is new. Concept. ● Regular assignments in tutorials ● Finals (oral or/and written, tba). ● You will need to pass the assignments to take the final. ● Exact conditions to be presented during first tutorial. ● Feedback most welcome. Suggestions, comments, lamentations, praise. You name it. (The more constructive, the better, of course.) Grading scheme Of overall grade: ● 60%: final exam ● Likely oral ● Potentially in groups ● 30%: tutorials ● Hand-in homework (first one on ACT-R to be announced at, or before, second tutorial) ○ Written part ○ Presentation in class or separately ● In-class work Grading scheme ● 10% related to reading assignments: Selected papers ● Start from selected paper(s) on methods, techniques, projects, systems, … from there: own research on related work ● Concise presentation in class, as groups ● Concise position papers THE calendar TUE WED 10/17 L1 10/23 T1 10/24 L2 10/30 L3 10/31 --- 11/06 T2 11/07 L4 11/13 --- 11/14 L5 11/20 T3 11/21 L6 11/27 L7 11/28 --- 12/04 L8 12/05 --- 12/11 T4 12/12 L9 L – lecture, T – tutorial THE calendar TUE WED 12/18 --- 12/19 L10 01/08 T5 01/09 L11 01/15 --- 01/16 L12 01/22 T6 01/23 L13 01/29 T7 01/30 L14 L – lecture, T – tutorial Basics: useful textbooks ● Perception ● Jeremy Wolfe et al. (2012). Sensation and Perception, 3rd ed. Sinauer Associates, Sunderland, MA, USA ● E. Bruce Goldstein (2010). Sensation and Perception, 8th ed. Wadsworth. * At the university library. Basics: useful textbooks ● Cognition ● John R. Anderson (2010). Cognitive Psychology and Its Implications, 7th Edition. Worth Publishers, New York. * (à contains various references to ACT-R) * At the university library. Specific literature To be announced as we move along. Cognition perception & attention Learned Associations Traffic signs: Learned meanings! Just as: Learned cultural associations ● ● ● ● red - danger, stop, delete green - go, environmentally friendly white - pure, clean, honest black - serious, heavy, death Associations may be different for different cultures Is this designed well? ● ● ● ● ● ● ● Correct Size Correct Shape Correct Location? Correct Weight Correct Color Correct Material But… à Ability to (re-) interpret meaning and act on interpretation! Now: ● What is cognition? ● What are users good and bad at? ● sensory organs (afferent) ● thinking organs + memory ● effectors (efferent) Some of the following slides (Intro to CogSci) are partly based on material of previous semesters of http://cosy.informatik.uni-bremen.de/category/teaching/cognitive-systems Cognitive System Cognition: What goes on in the mind? Experiential cognition (comes with practice, repeating e.g. driving a car) vs. reflective cognition (thinking and reflecting, e.g. deciding on what car to buy) Cognition (Definition I) Cognition: the mental processes used in perception, memory, communication, thinking, reasoning, remembering, judgment and understanding. adapted from Merriam-Webster dicitonary & Psychology: A ConnecText, byTerry Pettijohn Cognition (Definition II) “Cognition” refers to the principles underlying intelligence and intelligent systems with particular reference to intelligent behavior as computation. HA Simon & CA Kaplan, Foundations of Cognitive Science, in MI Posner, ed, Foundations of Cognitive Science, MIT Press 1991. (Sun, 2008; p.3) Cognitive System: Computation Metaphor Input Memory Reasoning Unit Processor Output Beyond the Cognitive System Environment ● Source of input Cognitive system ● E.g. ambient temperature, speech generated by another cognitive system, book, … ● Manipulation of environment ● E.g. store food for the winter, make notes, … ● Communication with other cognitive systems ● E.g. speak, put up traffic signs, … ● External memories ● E.g. notes, book, (spatial) structure of the environment, … Likely, no one remembers the detailed layout of Times Square (à complexity, visual clutter, …) One can still recognize it, navigate on it etc. (à use of information encoded in external structure / representations). 29 Cognition & Usability Why do we need to look at cognition? ● Better understanding & modeling of human performance at tasks ● Limits (What can + can’t a user do?) Limits For example, there exist various limits to human memory. BDTGCPEV 574831962 WLFZMQRA 7 1 V F X L 5 3 B 4 W 7 GCE BTEC GCSE GNVQ AS Cognition & Usability Why do we need to look at cognition? ● Better understanding & modeling of human performance at tasks ● Limits (What can + can’t a user do?) ● Specific properties (e.g. of a certain user / user group) Does Seeing One’s Hand Help? Example discussed during the first lecture: Different users may have different cognitive properties Back of hand Palm Camera shows Keehner, M. (2008). Conflicting Cues from Vision and Touch can Impair Spatial Task Performance: Speculations on the Role of Spatial Ability in Reconciling Frames of Reference. In Spatial Cognition VI. LNCS 5248, Springer. Image: http://www.colorectalhalgroup.com.au/images/handport_drawing.gif Hand Assisted Minimally Invasive Surgery Cognition & Usability Why do we need to look at cognition? ● Better understanding & modeling of human performance at tasks ● Limits (What can + can’t a user do?) ● Specific properties (e.g. of a certain user / user group) ● Needs (How to support cognition for better performance?) Does Seeing One’s Hand Help? Example discussed during the first lecture Different users may have different cognitive properties which may lead to different needs (e.g. in H-C interaction) Back of hand Palm Camera shows Keehner, M. (2008). Conflicting Cues from Vision and Touch can Impair Spatial Task Performance: Speculations on the Role of Spatial Ability in Reconciling Frames of Reference. In Spatial Cognition VI. LNCS 5248, Springer. Image: http://www.colorectalhalgroup.com.au/images/handport_drawing.gif Hand Assisted Minimally Invasive Surgery Cognition & Usability Why do we need to look at cognition? ● Better understanding & modeling of human performance at tasks ● Limits (What can + can’t a user do?) ● Specific properties (e.g. of a certain user / user group) ● Needs (How to support cognition for better performance?) ● Build better tools ● Learn from how humans function Computational models of cognitive processes for use in tools Example: Cognitive simulation of ship navigation by navigator and helmsman for risk assessment K. Itoh, T. Yamaguchi, J. P. Hansen and F. R. Nielsen. Risk Analysis of Ship Navigation by Use of Cognitive Simulation. Cognition, Technology & Work (2001) 3:4–21 Cognition & Usability Why do we need to look at cognition? ● Better understanding & modeling of human performance at tasks ● Limits (What can + can’t a user do?) ● Specific properties (e.g. of a certain user / user group) ● Needs (How to support cognition for better performance?) ● Build better tools ● Learn from how humans function ● Better interaction between humans and tools ● Tailor interaction to cognitive processes of users ● Also when tools take over some cognitive functions Tailor interaction to cognitive processes of users ● E.g. intelligent tutor systems Adapt tutoring to student’s learning progress (CMU’s LISP tutor) LISP, geometry, genetics, physics, … ● Shared human-computer control ● … Andes physics tutor Bremen smart wheelchair Rolland Images: Anderson et al. (1989). Skill Acquisition and the LISP tutor. Cognitive Science, 13, 467-505. / http://www.cs.cmu.edu/~cprose/pslc-slides/ StudyingandachievingrobustlearningwithPSLCResources.ppt / www.ensta-paristech.fr/~stulp/.../kriegbrueckner08adaptation.pdf Development of HumanMachine Systems Cf. Hollnagel, E. & Cacciabue, P. C. (1999). Cognition, technology and work: An introduction. Cognition, Technology and Work, 1(1):1–6. Manual Control ● Display of simple physical measurements ● E.g. pressures, temperatures ● Often: One sensor – one indicator ● Local control ● Manipulation of single components (e.g. valves, switches) ● Machine primarily as amplifier of human physical strength and precision ● Humans effectively just another piece of equipment to support machine functions. Tasks: ● Mainly observation, detection, manual operation ● Occasionally: process tuning or fault management Cf. Hollnagel, E. & Cacciabue, P. C. (1999). Cognition, technology and work: An introduction. Cognition, Technology and Work, 1(1):1–6. Images from http://farm3.static.flickr.com/2329/2293980977_730f3b3fb3.jpg, http://www.oldbeacon.com/beacon /images/inductor_big-4.jpg, Development of HumanMachine Systems Cf. Hollnagel, E. & Cacciabue, P. C. (1999). Cognition, technology and work: An introduction. Cognition, Technology and Work, 1(1):1–6. Supervisory Control ● Technological developments increased machine complexity & demands on operators ● Precision, speed and sustained attention exceeded human capacity à Solution: automation took over manual control ● Operators as supervisors who step in when things go wrong ● Tasks: ● State recognition, fault finding, scheduling of tasks during e.g. start-up and shutdown sequences, process tuning, etc. Cf. Hollnagel, E. & Cacciabue, P. C. (1999). Cognition, technology and work: An introduction. Cognition, Technology and Work, 1(1):1–6. Image: http://www.ebreaker.de/images/015-rollad1401.jpg / http://www.ornl.gov/info/ornlreview/v38_1_05/images/a11_controls_full.jpg Control room at a nuclear power plant (USA, Tennessee Valley Authority) Development of HumanMachine Systems Cf. Hollnagel, E. & Cacciabue, P. C. (1999). Cognition, technology and work: An introduction. Cognition, Technology and Work, 1(1):1–6. Cf. Hollnagel, E. & Cacciabue, P. C. (1999). Cognition, technology and work: An introduction. Cognition, Technology and Work, 1(1):1–6. / Image + caption: http://www.carbibles.com/brake_bible_pg2.html Cognitive Control ● Necessary with yet increasing complexity of systems and machines ● More extensive automation has removed operator even further from the actual process à Focus on cognitive aspects of joint human-machine system A (relatively easy) example of shared control in human-machine interaction: Volvo's collision warning system (CWS) constantly monitors speed and uses a 15° forward field of view radar to determine the distance to any object in front. If the distance begins to shrink but you don't slow down, the system sounds a buzzer and flashes a bright red light in a heads-up display to alert you. The brake pads are automatically placed against the discs and when the driver finally does use the brakes, the system monitors the pedal pressure. If the pressure is determined to be too light, the braking power is amplified by the system. Cognitive Control ● Today: Work is mediated by information technology and allocated between humans and machines ● Usually according to the nature of the process (and its physical implementation) rather than considerations of joint human-machine system functioning ● Creates problems for functioning of joint humanmachine system à not at optimum à Need to provide explicit design for the humanmachine system, e.g. with respect to performance envelopes à Study cognition! (natural + artificial) Cf. Hollnagel, E. & Cacciabue, P. C. (1999). Cognition, technology and work: An introduction. Cognition, Technology and Work, 1(1):1–6. What is Cognitive Science? The critical aspect of Cognitive Science is the search for understanding of cognition, be it real or abstract, human or machine. Donald A. Norman, What is cognitive science?, D. Norman, ed, Perspectives on cognitive science, Ablex, NJ 1981. Domain of Cognitive Science Examples: real abstract human wayfinding in a city imagining your next vacations machine robot avoiding obstacles computer planning a trip Disciplines of Cognitive Science real abstract human psychology anthropology neurosciences linguistics philosophy logics machine artificial intelligence cognitive robotics informatics mathematics Approaches to Cognitive Science real abstract human empirical studies biological models machine synthetic constructions theoretical process models Natural vs. Artificial Cognitive Systems calculating playing games speaking planning perceiving understanding arguing learning recovery natural weak weak/strong strong pretty good very strong very strong strong strong very strong artificial strong strong/weak improving quite good specialized weak weak improving weak Some Cognitive Abilities (1) ● Ability to detect and interpret sensory stimuli Example: On the following slide you will briefly see a number of colored geometric objects. Detect as fast as possible how many blue circles there are. ready? your answer? (5) Cognitive Abilities (2) ● Ability to detect and interpret sensory stimuli ● Tendency to focus on certain sensory stimuli and to disregard others Example: Was there a yellow hexagon among the colored geometric objects? your answer? (no) ● … ● Tendency to focus on certain sensory stimuli and to disregard others ● Ability to extract parts of complex events and to integrate them into a schema that gives meaning to the episode Video: https://www.youtube.com/watch?v=r8DydFeC9BA Cognitive Abilities (3) Cognitive Abilities (3) ● … ● Tendency to focus on certain sensory stimuli and to disregard others ● Ability to extract parts of complex events and to integrate them into a schema that gives meaning to the episode ● Detailed knowledge of the physical characteristics of the environment (by Mary Hegarty) Tying shoelaces How many windows are in your home? Cognitive Abilities (4) ● … ● Ability to extract parts of complex events and to integrate them into a schema that gives meaning to the episode ● Ability to extract meaning from letters and words Examples … C T Jim observed the hunter with the binoculars He was just about to pull the trigger He was looking for the deer. http://www.owlnet.rice.edu/~psyc351/Images/ContextEffects.jpg Examples / Context effects Cognitive Abilities (5) ● … ● Ability to extract parts of complex events and to integrate them into a schema that gives meaning to the episode ● Ability to extract meaning from letters and words ● Capacity to retain explicit knowledge and to integrate it into an ongoing sequence Example … This slide shows all circles of the picture you saw a while ago, except for the blue ones. Give me the number of circles in the original figure. Cognitive Abilities (6) ● … ● Ability to extract meaning from letters and words ● Capacity to retain explicit knowledge and to integrate it into an ongoing sequence ● Ability to form an image of a “cognitive map” For example … Cognitive maps (Kosslyn, Ball, & Reiser, 1978) Cognitive maps often preserve some relationships found in the space which they represent. Which city is further south Santiago de Chile or New York? Which city is further west Santiago de Chile or New York? 73°56′19″W 70°39.02′W Mental maps are often not veridical. 73 Cognitive Abilities (7) ● … ● Capacity to retain explicit knowledge and to integrate it into an ongoing sequence ● Ability to form an image of a “cognitive map” ● Ability to solve a problem For example … § Three missionaries and three cannibals want to cross a river from side A to side B. § They have a boat that holds no more than two persons at a time. § The life of the missionaries is endangered when the cannibals outnumber the missionaries at any location. How can the six missionaries safely cross the river? Image: http://thiscircularparade.com/wp-content/uploads/2012/01/1aIllustrated_world-108_edited-1.jpg Missionaries and Cannibals Problem Cognitive Abilities (8) ● … ● Ability to form an image of a “cognitive map” ● Ability to solve a problem ● Translation of a high-level instruction into a complex motor response For example … “Quickly run up the stairs, please.” Cognitive Abilities (9) ● … ● Ability to solve a problem ● Translation of a high-level instruction into a complex motor response ● Ability to recall quickly from long-term memory specific information that is immediately applicable to the present situation For example … What are your parents’ names and dates of birth? … Cognitive Abilities (10) ● … ● Translation of a high-level instruction into a complex motor response ● Ability to recall quickly from long-term memory specific information that is immediately applicable to the present situation ● Ability to translate visual events into spoken language For example … Describe what happened in the movie you saw a couple of minutes ago … Cognitive Abilities (11) ● … ● Ability to recall quickly from long-term memory specific information that is immediately applicable to the present situation ● Ability to translate visual events into spoken language ● Tendency to store linguistic information in a general form For example … Can you recall the last task regarding the colored objects I presented? The task was: “Give me the number of circles in the original figure.” Cognitive Abilities (12) ● … ● Ability to translate visual events into spoken language ● Inability to perform perfectly Inability to Perform Perfectly ● In a complex open world it is impossible to guarantee perfect solutions ● We need ways to discover imperfections ● We need ways to recover from imperfections ● Such “imperfections” are part of the operating principles of cognition! Core cognitive aspects ● ● ● ● ● Perception and recognition Attention Memory Reading, speaking and listening Problem-solving, planning, reasoning and decision-making, learning In relation to Nyström & Holmqvist, 2010: Resolution of the human visual system is differs across visual field. High acuity in foveal vision, low acuity in peripheral vision. no.of receptors per square mm Density of Rods & Cones in the Visual Field temporal fovea Angle (degrees) nasal From Coren, Porac, Ward., Sensation & Perception Rods & Cones Image: http://eyetracking.me/wp-content/uploads/2009/04/field_of_vision.png Foveal, parafoveal, peripheral vision Figure 2.10 The “rule of thumb” Image: Yang, S. & McConkie, G. W. (2005). New directions in theories of eye movement control during reading. In G. Underwood (ed.), Cognitive Processes in Eye Guidance, (pp. 105–130). New York: Oxford University Press. Repeat: Field of View Discussion of reading for today: In relation to Nyström & Holmqvist, 2010: Eye movements Targeting eye movements: ● Smooth pursuit ● Saccade ● Fixation Other: ● Vergence: The two eyes move in opposite directions, as when both eyes turn towards the nose ● Reflexive: Automatic and involuntary eye movements Image / content: Wolfe et al. Types of Eye Movements Targeting eye movements: ● Smooth pursuit: The eyes smoothly follow a moving target Video: The University of Utah 2001 under a Creative Commons Attribution-NonCommericalShareAlike 2.5 License // http://www.youtube.com/watch?v=sKrvQgoR2uk Types of Eye Movements Targeting eye movements: ● Saccade: A rapid movement of the eyes that changes fixation from one object or location to another Video: The University of Utah 2001 under a Creative Commons Attribution-NonCommericalShareAlike 2.5 License // http://www.youtube.com/watch?v=sKrvQgoR2uk Types of Eye Movements Types of Eye Movements Targeting eye movements: ● Saccade: A rapid movement of the eyes that changes fixation from one object or location to another ○ Saccadic amplitude: angular distance travelled ○ Fastest movement of the human body ○ Ballistic: top velocity proportional to amplitude (for all but very large amplitudes) ○ Amplitudes > ca. 20° usually with head movements, but much higher amplitudes possible Saccadic suppression: The reduction of visual sensitivity that occurs when we make saccadic eye movements ● Saccadic suppression eliminates the smear from retinal image motion during an eye movement à When do we see? Image / content: Wolfe et al. Eye Movements Types of Eye Movements Targeting eye movements: ● Fixational eye movement: ○ ○ ○ ○ Rather stationary period, between saccades, nearly all visual processing occurs during fixations common duration depends on various factors (task, level of mental processing, stimulus, …) Saccadic suppression: The reduction of visual sensitivity that occurs when we make saccadic eye movements ● Saccadic suppression eliminates the smear from retinal image motion during an eye movement à When do we see? Image / content: Wolfe et al. Eye Movements Image: http://1.1.1.3/bmi/www.learning-systems.ch/multimedia/fix.gif // http://upload.wikimedia.org/wikipedia/commons/e/ef/Reading_Fixations_Saccades.jpg Reading text Image: p. 205, Eyetracking Web Usability by Jakob Nielsen and Kara Pernice. Sequence of gaze positions Eye movements ● Reading off of information: sequential, piecemeal, selective ● Sequence may be influenced by ○ Structure, content, visual features, … (external) ○ Reasoning, representations, memory, task, …(mental) Videos by http://www.eyeseecam.com/ Seeing through somebody else’s eye movements EyeSeeCam Videos by http://www.eyeseecam.com/ Seeing through somebody else’s eye movements EyeSeeCam In relation to Nyström & Holmqvist, 2010: Eye movements depend on task. Image: Repin: The unexpected visitor. Source: Wikimedia Eye movements during different tasks (Yarbus, 1967) Eye movements during different tasks Context ● Also the other way around: analysis of visual inspection sequence may provide information on involved ● Mental processes ● Mental representations Different sequences of inspection under different tasks http://www.useit.com/alertbox/fancy-formatting.html One Question to Ask: Do Users Look at the Relevant Parts? Holsanova, J., Holmqvist, K. & Holmberg, N. (2008): Reading information graphics: The Role of Spatial Proximity and Dual Attentional Guidance. Applied Cognitive Psychology (2008). Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/acp.1525. Multimedia Aspects: Integration of Text & Images Eye movements may depend on used representation formats (here: text/diagrams) and their placement. In relation to Nyström & Holmqvist, 2010: Eye movements differ between people: E.g. due to levels of expertise. (Remember that N&H’s approach depends on similarity of (first) eye movements between people.) Image: Yang, S. & McConkie, G. W. (2005). New directions in theories of eye movement control during reading. In G. Underwood (ed.), Cognitive Processes in Eye Guidance, (pp. 105–130). New York: Oxford University Press. Eye Movements & visual search Fig. 5. The scan paths of two different observers, when reading the set of CT images shown in Fig. 4. Top row: result for a consultant radiologist who specialises in lung imaging. Middle and bottom rows: results for a novice observer before and after training. Image: Yang, S. & McConkie, G. W. (2005). New directions in theories of eye movement control during reading. In G. Underwood (ed.), Cognitive Processes in Eye Guidance, (pp. 105–130). New York: Oxford University Press. Differences in visual search patterns of expert and novice radiologists. In relation to Nyström & Holmqvist, 2010: Eye movements are influenced by properties of inspected scene / image etc. (such as changes in contrast, shapes, movements, etc. Cf. feature detectors in early vision (area V1)). At least, when first viewing a scene (cf. first part of study of Nyström & Holmqvist). Following example of computation of saliency map from Itti & Koch, 2001. Image: http://kybele.psych.cornell.edu/~edelman/Psych-4320/IttiKoch01-fig1.jpg In relation to Nyström & Holmqvist, 2010: Focus of eye movements (specifically, of fixations, a type of eye movements) does not need to be co-located with focus of attention. Good demonstration of this via the Posner cueing paradigm. Image: Wolfe et al. Posner Cueing Paradigm Posner Cueing Paradigm ● Cue? A stimulus that might indicate where (or what) a subsequent stimulus may be. Image: Wolfe et al. ○ Valid ○ Invalid ○ Uninformative Peripheral cue Image: Wolfe et al. Posner Cueing Paradigm Symbolic cue Image: Wolfe et al. Posner Cueing Paradigm Image: Goldstein, 8th edition Procedure for (a) the valid task and (b) the invalid task in Posner et al.’s (1978) precueing experiment. (c) The results of the experiment: average reaction time was 245 ms for valid trials but was 305 ms for invalid trials. Image: Wolfe et al. Posner Cueing Paradigm Attention! Attention. ● Attention? Any of the very large set of selective processes in the brain. Impossible for nervous system to handle all inputs at once à mechanisms that restrict processing to a subset ● Selective attention? The form of attention involved when processing is restricted to a subset of the possible stimuli. On ● Locations ● Features, ● ... Image: http://img.labnol.org/di/images/SpotLightf_178D/mousecursorspotlight.gif Spotlight metaphor for how visual attention gets allocated Image: Goldstein Spotlight metapher of attention: Sequential scene perception Focus of attention falls within foveal vision. Image: http://eyetracking.me/wp-content/uploads/2009/04/field_of_vision.png Overt attention Focus of attention elsewhere in visual field (does not fall within foveal vision). Image: http://eyetracking.me/wp-content/uploads/2009/04/field_of_vision.png Covert attention