NGOMSL Modeling Psyc645 -- Week 10 Wayne D. Gray and Thomas Mayfield George Mason University 1 Human Factors & Applied Cognitive Program Next Week -- TUESDAY!! NGOMSL#1 Produce NGOMSL methods from the trace of the VCR (my trace not yours) Are no pops in NGOMSL. Verifies may be suitable replacements. Yours to decide. NGOMSL#2 Create NGOMSL methods for the ManTel task Should be one set of methods that can be applied to both interfaces and to 1 or 2 queries Obviously, not all methods need to be used in each analysis Produce an NGOMSL trace for Dialog-Box 1 query and PopUp 2 queries Readings Paul Green’s 1999 Human Factors Ergonomics Society conference article George Mason University 2 Human Factors & Applied Cognitive Program Psyc645 -- Reprise of GOMS lecture 1 Levels of analysis based on Newell’s Time Scale of Human Action Important to concept is that behavior at one level can be decomposed into behavior at next lower level Show what is added at various levels Important to the argument that I am trying to make to convince you that analyses at increasing lower levels entail decreasing degrees of generality (or increasing degrees of specificity) Programming VCR is a generally understood concept, but programming my VCR is very different than programming George Lukas’ VCR. Setting Tom’s watch or Mary Ann’s watch is very different than setting the clock in my car Implications? Behavior is not as unconstrained as we would like to believe. As we get more and more specific in our description of behavior we find more and more constraints on behavior. George Mason University 3 Human Factors & Applied Cognitive Program Psyc645 - Overview of GOMS lecture 2 NGOMSL analyses Unlike KLM, NGOMSL is a serious notation for discussing interactive behavior at levels lower than the unit task Not simply a tool for analyzing design but a vehicle for representing low-level performance Today we will work through several examples George Mason University 4 Human Factors & Applied Cognitive Program NGOMSL Natural Language GOMS based on structured natural language notation and a procedure for constructing them models are in program form Control Structure: Hierarchical goal stack Serial or parallel: Serial Level of Analysis: In principle, as necessary for your design question George Mason University 5 Human Factors & Applied Cognitive Program NGOMSL - why? More powerful than KLM. Much more useful for analyzing large systems More built-in cognitive theory Provides predictions of operator sequence, execution time, and time to learn the methods George Mason University 6 Human Factors & Applied Cognitive Program NGOMSL - Overall Approach Step 1: Perform goal/subgoal decomposition Step 2: Develop a method to accomplish each goal List the actions/steps the user has to do (at as general and high-level as possible for the current level of analysis) Identify similar methods/collapse where appropriate Step 3: Add flow of control (decides) Step 4: Add verifies Step 5: Add perceptuals, etc. Step 6: Add mentals for retrieves, forgets, recalls Step 7: Add times for each step Step 8: Calculate total time George Mason University 7 Human Factors & Applied Cognitive Program NGOMSL - Example Car clock Presetting radio stations simple perverse George Mason University 8 Human Factors & Applied Cognitive Program NGOMSL--Car Radio example (1) Provides predictions of methods and operators used to complete a task. If you provide estimates of operator-duration, you can get predictions of error-free expert performance time. George Mason University 9 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (2) Goal/subgoal hierarchy set clock set mode turn knob depress knob (and keep it depressed) set hour set minute press tune “-” button press tune “+” button release knob George Mason University 10 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (3) Assumptions User's hands are NOT on the buttons at the beginning. Time to move hand & arm to time change level is estimated by 2ft "reach" from Barnes (1963): 410 msec. D = device time, time for clock to move forward one number, = 500 msec (we estimate this on the next slide) I am assuming a 12-hr clock. Seems good assumption for a car clock. Radio is off at beginning and end. Begin knowing the time you want to set the clock to. George Mason University 11 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (4) D = device time, time for clock to move forward one number, = 500 msec 290 Visual Perception perceive info (X) 50 Cognitive Operators 50 verify info (X) initiate release (X) 100 Right Hand release (X) Minimum time required to perceive clockTime=targetTime and to release the toggle lever George Mason University 12 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (2) • (Duplicate slide) • Note that method for accomplishing the top level goal will have three subgoals and one “release” physical operator set clock set mode turn knob depress knob (and keep it depressed) set hour set minute press tune “-” button press tune “+” button release knob George Mason University 13 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (5) Method for goal: SET-CLOCK Accomplish goal: set Mode to Step 1 set Clock Decide: If curHr°targetHr Then: Accomplish goal: Change Hour Step 2 display Decide: If curMin°targetMin Then: Accomplish goal: Change Step 3 Minute display stmt time oper 0.1 op time tot time 0.10 0.1 0.10 0.1 0.10 0.1 0.10 Step 4 Release "vol/on/off" knob 0.1 B Step 5 Return with goal accomplished 0.1 0.10 assumptions 0.20 0.10 Need methods for each of these three subgoals George Mason University 14 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (6) Method for goal: set Mode to set Clock stmt op tot time op time time 0.1 0.10 Step 1 Recall "vol/on/off" is modeSwitch 0.1 M Decide: If location of modeSwitch is not known then Locate Step 2 modeSwitch 0.1 M Reach & Grasp "volume/on/off" Step 3 knob Step 4 Turn knob 0.1 R 0.1 T Home thumb to "volume/on/off" Step 5 knob Step 6 Press and hold knob Step 7 Return with goal accomplished 0.1 H 0.1 B 0.1 assumptions retrieval from LTM or equivalent (location and 0.50 0.60 perception of labels?) needed For this example we assume that location is NOT known. If known then tot. time for this step would be the stmt time, 1.20 1.30 0.10 sec source: Barnes, 1963, reach = 0.48 0.58 410 msec, grasp = 070 msec 0.34 0.44 source: Barnes, 1963 400 msec seems too high here, might use 100 msec for 0.40 0.50 button press instead. 0.10 0.20 0.10 George Mason University 15 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (7) stmt time Method for goal: Change <time> display Decide: If location of setTime switch is not known then Locate Step 1 setTime switch Decide: If finger is NOT on setTime switch then: Reach to Step 2 setTime switch Step 3 Step 4 Retrieve-from-LTM that <time> = "+ or -" Step 5 Determine current setting Press and hold setTime switch to <time> Step 6 Step 7 Step 8 Hold until <curTime> = <targetTime>, (500 x #digits) Verify display <time> Return with goal accomplished oper op time 0.1 tot time 0.10 assumptions 0.1 M 1.20 1.30 assume that loc is not known 0.1 R 0.41 0.51 0.50 retrieval from LTM or equivalent (location and perception of labels?) 0.60 needed and takes - 500 msec 0.1 M 0.1 P 0.32 0.1 B 0.10 W 0.1 M 0.1 1.20 based upon cpm-goms analysis it should take 420 msec for eye to move to clock 0.42 and for user to perceive the hour. cpm-goms calc. is -250 msec, go with 0.20 KLM System response time is 500 msec/digit, this is enough time for Ss to perceive and initiate keyUp response. 1.30 0.10 George Mason University 16 Human Factors & Applied Cognitive Program NGOMSL-- Car Radio example (8) time for SetClock goal time for setMode goal time for setHour from 1 to 10 time to setMin from 56 to 50 total 0.70 3.82 9.03 31.53 45.08 George Mason University 17 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (1) George Mason University 18 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (2) Goal hierarchy for optimal design PreSET Network Show Set Mode to Radio Locate Station Set PreSET etc Select AM/FM seek & identify press & hold preset button until sound returns Note differences between this and previous slide. George Mason University 19 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (3) MODEL 1: SETTING A PRESET TO A NETWORK SHOW OPTIMAL DESIGN Method for goal: preset network show Step1: Decide: IF radio is not playing, THEN Accomplish goal: set-mode-to-radio Step2: Decide: IF show N.E. intended show, THEN Accomplish goal: locate-station Step3: Accomplish goal: set-preset Step4: Return with goal accomplished Method for goal: locate-station Step1: Decide: IF band N.E. desired band, THEN Accomplish goal: select AM/FM Step2: Accomplish goal: seek&identify Step3: Return with goal accomplished Method for goal: select AM/FM Step1: Reach to button for desired band Step2: Press button for desired band Step3: Verify band correct Step4: Return with goal accomplished George Mason University 20 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting a station (4) Method for goal: seek&identify Step1: Reach to right side of "Seek" button Step2: Press "Seek" button Step3: Decide: IF show N.E. intended show, THEN Goto 2 Step4: Verify show correct Step5: Return with goal accomplished Method for goal: set-preset Step1: Reach to desired "Preset" button Step2: Press and hold "Preset" button Step3: Wait until sound returns Step4: Remove finger from "Preset" button Step5: Return with goal accomplished George Mason University 21 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (1) Goal hierarchy for perverse design PreSET Network Show Locate Station Set Mode to Radio Set PreSET etc Select AM/FM Enter seek mode Delete old PreSET Initiate seek seek & identify Exit seek MODE Enter delete MODE Enter PreSet Button Assign new PreSET Exit delete MODE Enter PreSet MODE Enter PreSet Button Exit PreSet MODE George Mason University 22 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (2) Radio for Perverse Preset Design SEEK MODE AM FM EJ SEEK – ON TUNE ERASE PRESET + OFF 2:41 MEMORIZE PRESET BASS 1 2 3 4 5 6 TREBLE – + – + BALANCE L R FADE L R PUSH CLOCK George Mason University 23 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (3) MODEL 2: SETT ING A PRESET T O A NET WORK SHOW PERVERSE DESIGN Method for goal: preset network show (s/a optimal) Step1: Decide: IF radio is not playing, THEN Accomplish goal: set-mode-to-radio Step2: Decide: IF show N.E. intended show, THEN Accomplish goal: locate station Step3: Accomplish goal: set-preset Step4: Return with goal accomplished Method for goal: locate-station (s/a optimal) Step1: Decide: IF band N.E. desired band, THEN Accomplish goal: select AM/FM Step2: Accomplish goal: initiate-seek Step3: Return with goal accomplished Method for goal: select AM/FM (s/a optimal) Step1: Put finger on button for desired band Step2: Press button for desired band Step3: Verify band correct Step4: Return with goal accomplished George Mason University 24 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (4) Method for goal: initiate-seek Step1: Accomplish goal setMode-SEEK Step2: Accomplish goal: seek&identify Step3: Accomplish goal: exit seek mode Step4: Return with goal accomplished Method for goal: setMode-<mode> Step1: Reach for <mode> button Step2: Press <mode> button Step3: Verify <mode> correct Step4: Return with goal accomplished Method for goal: seek&identify (s/a optimal) Step1: Reach to right side of "Seek" button Step2: Press "Seek" button Step3: Decide: IF show N.E. intended show, THEN Goto 2 Step4: Verify show correct Step5: Return with goal accomplished Method for goal: set-preset Step1: Accomplish goal: delete-previous-preset Step2: Accomplish goal: assign-new-preset Step3: Return with goal accomplished George Mason University 25 Human Factors & Applied Cognitive Program NGOMSL-- PreSetting perverse (5) Method for goal: delete-previous-preset Step1: Accomplish goal: setMode-deletePreSet-on (setMode-<mode>) Step2: Accomplish goal: enter-preset-button Step3: Accomplish-goal: setMode-deletePreSet-off (setMode-<mode>) Step4: Return with goal accomplished Method for goal: assign-new-preset Step1: Accomplish goal: setMode-preset-on (setMode<mode>) Step2: Accomplish goal: enter-preset-button Step3: Accomplish-goal: setMode-preset-off (setMode<mode>) Step4: Return with goal accomplished Method for goal: enter-preset-button Step1: Reach to desired "Preset" button Step2: Press "Preset" button Step3: Return with goal accomplished George Mason University 26 Human Factors & Applied Cognitive Program NGOMSL--comparison of optimal and perverse designs Optimal number of different methods Perverse 6 11 6 18 number of different steps 21 38 number of steps required 21 66 number of methods used George Mason University 27 Human Factors & Applied Cognitive Program Cost-of-Knowledge Characteristic Function Card, S. K., Pirolli, P., & Mackinlay, J. D. (1994). The cost-of-knowledge characteristic function: Display evaluation for direct-walk dynamic information visualizations. In B. Adelson, S. Dumais, & J. Olson (Eds.), ACM CHI'94 Conference on Human Factors in Computing Systems (Vol. 1, pp. 238-244). New York: ACM Press. George Mason University 28 Human Factors & Applied Cognitive Program George Mason University 29 Human Factors & Applied Cognitive Program CENTURY-METHOD = GOAL: DO-TASK GOAL: GET-DATE TURN-TO [MANUSCRIPT] GET-DATE GOAL: ACCESS-DAY-CALENDAR GET-YEAR . . . if necessary GOAL: SELECT-CENTURY (1700Хs) POINT-TO (Century=1700-1790s)) => Century-display GET-YEAR . . . if necessary GOAL: SELECT-DECADE (1710Хs) POINT-TO (1710-1719)) => Decade display GET-YEAR . . . if necessary GOAL: SELECT-YEAR: (1719) POINT-TO (1719)) => Year-display GET-MONTH . . . if necessary GOAL: SELECT-MONTH: (November) POINT-TO (November)) => Month-display GET-DAY . . . if necessary GOAL: SELECT-WEEK: [??] POINT-TO [23] => Week-display GET-DAY . . . if necessary GOAL: SELECT-DAY: [23] POINT-TO (23)) => Day-display George Mason University 32 Human Factors & Applied Cognitive Program But, Card, Pirolli, & MacKinley did not do a complete GOMS analysis Only did enough of an analysis to determine that the “volume part” of the task was the repeated accesses of menus for each component (century, decade, year, month, week, day) George Mason University 33 Human Factors & Applied Cognitive Program Cost-of-Knowledge Function Small amounts of knowledge can be accessed quickly with access costs increasing as amount of knowledge accessed increases ≈6.9 s for 1 day ≈10.4 s for 7 days ≈14 s for 30 days ≈17.5 s for 365 days ≈21 s for 3,562 days ≈24.6 s for 36,525 days George Mason University 34 Human Factors & Applied Cognitive Program CD = =Eliminate need “week” to select improve “week”cycle can do time = Eliminate Spiral as&is BA = SC w/ 2 Calendar s cycle time (was 3.5) that when select day from the month calendar Design Improvements? Speed up system response time (SRT) Time to access = 3.3 + 3.5 * Ncycles Imply in text that it takes 1 s to point to item and 2.5 s for system to bring up item Eliminate some of the cycles -- seem redundant Get rid of the week cycle Have users bring up month and directly select day from the month calendar rather than first selecting week and then month Issue - Implicit estimate of 2.5 s srt seems tediously long Suggest need for a full NGOMSL analysis George Mason University 36 Human Factors & Applied Cognitive Program NGOMSL Methods for Spiral Calendar Top method and one subordinate Method for goal: Set-Date Step 1 Accomplish goal: get-date Step 2 Accomplish goal: Access-day-calendar Step 3 Return with goal accomplished Total Time Method for goal: get-date stmt op time op t/f time 0.1 0.1 0.1 0.1 0.1 Step 1 Decide: IF page not turned, then turn page 0.1 R Step Step Step 2 Read Date 3 Home hand to keyboard 4 Return with goal accomplished 0.1 M 0.1 H 0.1 Total Time tot time 0.10 0.10 0.10 0.10 0.40 assumptions 0.10 T 0.88 1.2 0.4 source: Barnes, 1963, reach = 410 msec, grasp = 070 msec, + 400 est for flip. ** could analysis this further e.g. 0.98 CPM-GOMS use M for estimate of time to read and 1.30 put into memory 0.50 0.10 2.98 George Mason University 37 Human Factors & Applied Cognitive Program NGOMSL Methods (cont’) Method for goal: access-day-calendar Step 1 Decide: IF forgot date, then Read Date Step 2 Accomplish Goal: Set-<timeUnit> Decide: If current_day N.E. target_day, then Step 3 GO TO Step 1 Step 4 Verify date Step 5 Return with goal accomplished Total Time 0.1 0.1 M 0.1 Method for goal: Set-<timeUnit> Step 1 Point to <timeUnit> Step 2 ClickOn <timeUnit> Step 3 Wait for System Response Step 4 Return with goal accomplished 0.1 0.1 P 0.1 BB 0.1 W 0.1 Total Time 0.1 0.1 M 0.1 F 1.20 1.20 1.10 0.20 1.00 0.10 0.10 0.10 0.10 1.30 0.10 1.80 0.10 1.20 0.30 1.10 0.10 2.80 George Mason University 38 Human Factors & Applied Cognitive Program 0.10 Method for goal: Set-Date 0.10 Step 1 Accomplish goal: get-date 0.10 Method for goal: get-date 0.98 Step 1 Decide: IF page not turned, then turn page 1.30 Step 2 Read Date 0.50 Step 3 Home hand to keyboard 0.10 Step 4 Return with goal accomplished 0.10 Step 2 Accomplish goal: Access-day-calendar 0.10 Method for goal: access-day-calendar 0.10 Step 1 Decide: IF forgot date, then Read Date 0.10 Step 2 Accomplish Goal: Set-<timeUnit> 0.10 Method for goal: Set-<timeUnit> 1.20 Step 1 Point to <timeUnit> 0.30 Step 2 ClickOn <timeUnit> 1.10 Step 3 Wait for System Response 0.10 Step 4 Return with goal accomplished 0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 1 0.10 Step 1 Decide: IF forgot date, then Read Date 0.10 Step 2 Accomplish Goal: Set-<timeUnit> 0.10 Method for goal: Set-<timeUnit> 1.20 Step 1 Point to <timeUnit> 0.30 Step 2 ClickOn <timeUnit> 1.10 Step 3 Wait for System Response 0.10 Step 4 Return with goal accomplished 0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 1 0.10 Step 1 Decide: IF forgot date, then Read Date 0.10 Step 2 Accomplish Goal: Set-<timeUnit> 0.10 Method for goal: Set-<timeUnit> 1.20 Step 1 Point to <timeUnit> 0.30 Step 2 ClickOn <timeUnit> 1.10 Step 3 Wait for System Response 0.10 Step 4 Return with goal accomplished 0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 1 1.30 Step 4 Verify date 0.10 Step 5 Return with goal accomplished 0.10 Step 3 Return with goal accomplished How well does NGOMSL fit the data? 25 y = 0 .852 5x + 2 .431 5 2 R = 0.99 NGOMSL (se c) 20 15 10 5 0 0 5 10 15 20 25 30 Empirical Data (sec) Pretty well by r2 George Mason University 40 Human Factors & Applied Cognitive Program How well does NGOMSL fit the data? em pirical data card m odel NGOMSL 30 Tim e in Seconds 25 20 15 10 5 0 1 2 3 4 5 6 Num ber of Cycles Pretty well compared to the equation they derived from the data George Mason University 41 Human Factors & Applied Cognitive Program How well does NGOMSL fit the data? em pirical data card m odel NGOMSL NGOMSL w/2.5 s srt 35 30 Tim e in Seconds 25 20 15 10 5 0 1 2 3 4 5 6 Num ber of Cycles But not too well when I assume a 2.5 sec system response time George Mason University 42 Human Factors & Applied Cognitive Program What Happened? Did the modeler cheat? Let’s examine the assumptions of the model Critical assumptions concern the time per cycle and the elements that compose the cycle Card et al., empirically derived 3.5 s per cycle The NGOMSL model, which assumed a 1 s SRT, predicted 3.1 s per cycle George Mason University 43 Human Factors & Applied Cognitive Program 3.1 s 3.1 s 3.1 s 0.10 Method for goal: Set-Date 0.10 Step 1 Accomplish goal: get-date 0.10 Method for goal: get-date 0.98 Step 1 Decide: IF page not turned, then turn page 1.30 Step 2 Read Date 0.50 Step 3 Home hand to keyboard 0.10 Step 4 Return with goal accomplished 0.10 Step 2 Accomplish goal: Access-day-calendar 0.10 Method for goal: access-day-calendar 0.10 Step 1 Decide: IF forgot date, then Read Date 0.10 Step 2 Accomplish Goal: Set-<timeUnit> 0.10 Method for goal: Set-<timeUnit> 1.20 Step 1 Point to <timeUnit> 0.30 Step 2 ClickOn <timeUnit> 1.10 Step 3 Wait for System Response 0.10 Step 4 Return with goal accomplished 0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 1 0.10 Step 1 Decide: IF forgot date, then Read Date 0.10 Step 2 Accomplish Goal: Set-<timeUnit> 0.10 Method for goal: Set-<timeUnit> 1.20 Step 1 Point to <timeUnit> 0.30 Step 2 ClickOn <timeUnit> 1.10 Step 3 Wait for System Response 0.10 Step 4 Return with goal accomplished 0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 1 0.10 Step 1 Decide: IF forgot date, then Read Date 0.10 Step 2 Accomplish Goal: Set-<timeUnit> 0.10 Method for goal: Set-<timeUnit> 1.20 Step 1 Point to <timeUnit> 0.30 Step 2 ClickOn <timeUnit> 1.10 Step 3 Wait for System Response 0.10 Step 4 Return with goal accomplished 0.10 Step 3 Decide: If current_day N.E. target_day, then GO TO Step 1 1.30 Step 4 Verify date 0.10 Step 5 Return with goal accomplished 0.10 Step 3 Return with goal accomplished Where is the problem? For NGOMSL to accept 2.5 s SRT means either accepting a very bad fit, or changing the model But, what can be changed? My guess is that they never measured the SRT of their system In any event it is incredible that they assumed that all of the cycle time not devoted to pointing was attributable to SRT George Mason University 45 Human Factors & Applied Cognitive Program Two Morals to this Story? First Even good researchers can be careless when they do not check their assumptions against a model (& yes, Card, Pirolli, and Mackinlay are VERY good researchers) Second Before investing lots of effort into jazzing up system response time, do a profile analysis with a complete GOMS model George Mason University 46 Human Factors & Applied Cognitive Program Next Week -- TUESDAY!! NGOMSL#1 Produce NGOMSL methods from the trace of the VCR (my trace not yours) Are no pops in NGOMSL. Verifies may be suitable replacements. Yours to decide. NGOMSL#2 Create NGOMSL methods for the ManTel task Should be one set of methods that can be applied to both interfaces and to 1 or 2 queries Obviously, not all methods need to be used in each analysis Produce an NGOMSL trace for Dialog-Box 1 query and PopUp 2 queries Readings Paul Green’s 1999 Human Factors Ergonomics Society conference article George Mason University 47 Human Factors & Applied Cognitive Program End of NGOMSL George Mason University 48 Human Factors & Applied Cognitive Program