3461 S04
Module 4
COSC
Windows-Icon-Menus-Pointers
(WIMP) Elements
Types of Interactions
•
•
•
•
•
•
•
Command-based
Natural language based (written, spoken)
Menu-based
Question-asking/answering (query dialog)
Field Completion
Windows-Icons-Menus-Pointers (WIMP)
Point-and-click
2
Point-and-Click
• Point-and-click:
– Interaction that requires only single clicks of a
mouse button
– Common in multimedia systems (e.g., web
browsers)
– Popularized by WWW pages
– Also used for systems based on touch screens,
such as information kiosks
– Interaction style closely tied to hypertext
3
WIMP-based Interaction
• Windows Icons Menus Pointers
– Windows, Icons, Mice, Pull-down menus
(according to some)
• Currently the most common environment
for interactive computing
• Examples:
– Microsoft Windows, Apple MacOS,
X-Windows (for unix)
4
WIMP-based Interaction
• A user should be able to interact using…
– only the mouse
– only the keyboard
– combination of mouse and keyboard
• Combination of mouse and keyboard is most
convenient
– text input via a mouse is difficult (but not
impossible!)
– clicking and selection actions with the keyboard are
more difficult than with mouse (but not impossible!)
5
Comparison to Point-and-Click
• Both point-and-click and WIMP interfaces
have buttons
• Point-and-click philosophy is simpler
• WIMP-based interaction not limited to
mouse
6
Outline
•
•
•
•
Windows
Icons
Menus
Pointers
7
Windows
• Windows have an appearance and a behaviour
• Appearance
– occupy an area of the screen
– can contain text, graphics, menus, toolbars, and so
on…
• Behaviour
– they act like individual terminals for applications
– can be moved, resized, closed, minimized, maximized
– can accept focus
• What is it about their appearance that signals
their behaviour?
8
Windows
• How is behaviour of windows
determined?
– By the system’s window manager
– aka windowing system,
user interface management system (UIMS)
9
Window Managers
• How essential is a window manager?
– When user opens a new window, how is
its position on the screen determined?
– Application-specific decisions could
lead to chaos on the screen
• Instead, layout policy of window manager
used
• Layout policy may be fixed or may be
selected by user
10
Anatomy of a Window
indicator of focus (blue)
size controls
title bar
menu bar
contents
scroll bar
feedback
size control
11
System for Organizing Windows
• The display of multiple windows constrained by
screen real-estate
– Multiple windows may exist simultaneously, but they
might not all be fully shown
• Possible layouts include:
– Overlapping – One window partially obscures another
– Tiled – Adjoin, but don’t overlap
– Cascading – A sequence of overlapping windows, each
offset from the preceding according to a rule (e.g., 10
pixels to the right and below)
12
Overlapping Windows
13
Tiled Windows
14
Cascading Windows
15
The Notion of “Focus”
• Scenario: multiple windows are open
– input action that users performs is applied to window that
has focus
• What are the visual clues of focus?
– the colour of the title bar of active window can signify focus
• Blue or some other colour indicates focus; Grey or some other
colour indicate window is inactive
– What other attribute? (hint: it may require redrawing)
• To change focus:
– mouse: click in it (note that “in” needs to be defined somewhere)
– keyboard: system-dependent
16
Resizing a Window
What has changed?
• scrollbar added
• menu has wrapped
• toolbar modified (buttons lost)
17
Window Size
• Size of window is considered a state of the window
– What other type of states do windows have?
• Windows have three size states:
– Maximized
• Fills available space
• Can’t resize the window
– Minimized/Iconified
• Reduced to a title bar or icon
• Can’t resize (icon?)
– Normal/Restored
• Window occupies a region
• Can resize representation of window: the window width and height
may be adjusted
18
Size State Transitions
b1=minimize
b2=maximize
b3=restore down
St
St+1
St-1  { Maximized, Normal/Restored }
Minimized
Minimized
Maximized
Maximized
Normal/Restored
Normal/Restored
a=resize
19
Control of Window Size State
• Buttons in upper-right corner of window are
context sensitive:
Maximized
Restored
Minimized
Minimize Restore Close
Minimize Maximize Close
Restore Maximize Close
• Why isn’t this at odds with the design principle
of consistency?
20
Control of Window Size State
• Resize - what visual clues are provided?
Handle in lower-right
corner of window
(what signals that this
is a handle?)
Drag to resize
21
Control of Window Size State
Handle on edges
(what signals that this
is a handle?)
Drag either edge to resize height
Drag either edge to resize width
22
Outline
•
•
•
•
Windows
Icons
Menus
Pointers
23
What is an Icon?
• Webster’s dictionary defines an icon
as a pictorial representation
• An icon is one type of sign
• A sign is:
– something that “stands for” or
“represents” another thing
– the other thing is called the referent
24
What is an Icon?
• How are icons used?
– Representation of windows that are
minimized
– What other uses have we already discussed?
• The act of reducing a window to an icon is
called iconifying or minimizing
– A window may be restored by clicking on its
icon
• If not with an icon, how else might a
minimized window be represented?
25
Examples of Icons
– Disk drives
– Available applications
Folders
Files
– Miminized applications
Commands
– Minimized windows
States
26
Tradeoffs for Icons
Advantages:
– small (use small amount of screen space)
– can be intuitive
– can persist
• serve as reminders of available dialogs, applications, or
commands that may be restored or invoked
Disadvantages:
– How strong does the perceptual resemblance need to
be for iconic representations of minimized windows?
• Can minimized windows get lost?
• Can minimized windows get confused with other
minimized windows?
27
Outline
•
•
•
•
Windows
Icons
Menus
Pointers
28
What is a Menu?
• A menu is an organizational device for presenting a set
of buttons
– buttons correspond to imperative commands
– represent operations that can be performed by the application at
a given time
• A toolbar is also an organizational device for presenting
a set of buttons
• Menus vs. toolbars
– Recognition vs. Recall (which ability do humans find easier?)
– Cost of navigation (input actions)
• Review: what input actions are used with menus?
– Keyboard
– Pointing device
29
Location of Menus
• Menu bar located directly below the title bar
– Optional, but most application windows have them
– Why this location as opposed to other ones?
title bar
menu bar
30
Location of Menus
• Hidden Menu
– called popup menus
– right mouse click to access
– context sensitive
31
Menu Design Techniques
• Why are menus difficult to design?
– Need an understanding of users’ mental
model
– Need to manage tradeoff between space
constraints and cognitive burden
32
Features of Menus
•
•
•
•
•
Cascades (sub-menus)
Groupings
Dialog boxes
Icons/Textual Elements
Support for keyboard input
– Mnemonics
– Accelerators
33
Cascades
• Design technique:
– Group associated buttons together, then place
in a sub-menu
• When submenu is selected:
– The submenu “cascades”
– Several layers of cascading menus may be
used
– Why do we want this?
• How is this behaviour signalled?
Example
34
Example
File menu
File menu with focus on Preferences
Visual indicator
for cascading
menu
35
Cascades
• Submenus form a hierarchy
• What are the tradeoffs?
– Each level in the hierarchy requires more
input actions from user, greater cognitive
load/burden
36
Groupings
• Similar items are grouped together in a
menu
• How is the grouping signalled?
– negative space (empty space), or
– separator (horizontal line)
Example
37
Example
Color menu
Image adjust group
Channel group
Visual indicator for
group
Palette group
Color depth group
38
Dialog Boxes
• Recall that imperative control has
a <verb> <noun> structure
• The specification of the <noun> may require
additional input from the user
39
Dialog Boxes
• A menu choice may trigger a dialog box
• The dialog box may contain:
– messages
– editable fields
– buttons, etc.
• Types of dialog boxes
– File (open, new, save as)
– Print
– Color chooser
Example
• How are such application functions triggered?
Why do we want to signal this?
40
Example
File menu
Visual indicator for
dialog box
“New” dialog box
41
Icons in Menus
• Recall: menus are collections of buttons
– How might these buttons be labelled?
• Text
• Icons
• Text and Icons
Example
42
Example
Format menu
Font icon
Paragraph icon
43
Keyboard Access to Menus
• Menus need to support access by a pointing
device and a keyboard
– Why?
• Who might use the keyboard instead of the
pointing device?
– Those interested in speed (expert/frequent users)
– Those who cannot use the pointing device
44
Mnemonics and Accelerators
• A chord is a combination of key presses
– e.g., shift-f12, ctrl-P, alt-F, ctrl-shift-x, meta-x
– the initial key is typically a function/modifier
keys (shift, control, alt)
– the final key is usually an alphanumeric key
45
Mnemonics and Accelerators
• A mnemonic is a keyboard shortcut to a visible
button
– Convention: Alt-letter chord, with underlined single
letter
– The full menu hierarchy may be accessed using only
the keyboard, but it may require a sequence of
mnemonics
• An accelerator is a keyboard shortcut to a button
that is not visible
– Convention: Ctrl-letter or Shift-letter chord
– Shortcuts to bypass the menu hierarchy and directly
invoke a menu option
Example
46
Example
To invoke the “Full Screen Preview” command in the “View” menu
Technique 2:
Technique 1:
Using mnemonics:
View menu
Using accelerator:
Alt-v
+
p
Shift-Ctrl-A
47
Popup Menus
• Many menus accessed from menu bar
– invocation action: left mouse-click
– left mouse-clicks on non-menu regions
associated with other application functions
• What does right mouse-click do?
– invokes pop-up menu
– type of menu depends on context of the click
– popup menu is sensitive to context
48
Example
Context sensitive popup menus on Windows 98 desktop
Right-click on Start
button
Right-click on
background
49
Example
DemoMenu.java
50
Example
DemoMenu2.java
51
Example
DemoLookAndFeel.java
Metal (java)
Motif
Windows
52
Outline
•
•
•
•
Windows
Icons
Menus
Pointers
53
What is a Pointer?
• A pointer is an input device that affords:
– Position control (may need an on-screen tracker)
– Selection (via buttons)
• Examples:
mouse, trackball, joystick, touchpad,
finger, stylus, light pen
• Pointer used to interact with GUI components
54
Direct vs. Indirect Input
• Direct input
– No spatial displacement between input device and
display
– Selection via tapping or pressing
– Tracker generally not needed
– Examples: finger, stylus, light pen
• Indirect input
–
–
–
–
Spatial displacement between input device and display
Tracker needed
Selection via button presses
Examples: mouse, trackball, joystick, touchpad
55
Selection Primitives
• How many buttons are needed on
pointing devices?
– Generally at least two buttons
• Selection primitives
– Primary button (default = left)
• Single click – select
• Double click – launch
• Drag – select region
– Secondary button (default = right)
• Click – invoke context-sensitive menu
56
Tracker
• Tracker is an on-screen symbol
– it follows (“tracks”) the user’s motion (via the
input device)
• Tracker can indicate two things:
– Position – on-screen position provides
feedback to user
– State – the symbol can change, to reveal
information about the current state of the
system or GUI component
57
Tracker
• The tracker is an on-screen symbol
– Symbol is an image in bitmap format (x by y pixels)
• The bitmap can partially overlap with buttons
– when is the tracker considered “in” the region of the
button?
• One pixel in the image is defined as the hot spot
– Selection determined by the coordinate of the hot
spot
• Trackers need to have an obvious hotspot
– an issue when designing custom trackers
58
Tracker Examples
• Examples from MS Windows
Where is the
hot spot?
59
Example
DemoCursorControl.java
Note:
Cursor is not captured
with “print screen”. Run
the program to observe
the cursor changing.
60
One Context for WIMP-based
Interaction
• The old: command-line interaction (terminals)
• The new: WIMP-based interaction
• What about all of the legacy systems?
– Old software systems are often called legacy systems
– Adage: If it ain’t broke, don’t fix it!
– Legacy systems are very hard to replace or update,
due to investment and momentum
• X-Windows “on top of” UNIX mixes the old
with the new
• Common Solution:
command-line interaction in a window
61
Outline
•
•
•
•
Windows
Icons
Menus
Pointers
• Window Managers
• More Widgets
62
Atomic Components
• An atomic component is:
– a low-level entity
– an entity that interacts with the user
– an entity that cannot hold other components
• Examples:
– buttons (JButton),
– text fields (JTextField)
– combo boxes (JComboBox)
• Other components:
– JFrame and JPanel
– Are these atomic?
63
Containers
• Containers hold components
– All components are placed in containers
– Might be top-level or intermediate
• Top-Level
– E.g., JFrame is a top-level container
• It exists mainly as a place for other components to paint
themselves
– Other top-level containers: dialogs (JDialog) and applets
(JApplet)
• Intermediate
– E.g., JPanel is an intermediate container
• Its purpose is to simplify the positioning of interactive
objects, such as buttons or text fields
– Other intermediate containers: scroll panes (JScrollPane)
64
and tabbed panes (JTabbedPane)
Containment Hierarchy
• A window is made up of a number of
components
– atomic components (e.g.,buttons, text fields)
– composite components (e.g., containers)
• How might the relationship of these components
be expressed?
– by a containment hierarchy
– based on screen geometry of objects
– represents the hierarchy/nesting of the objects
65
Example 5.11
Display Screen
“F:\cs160\Public” window
Inner Window
title bar
horizontal scroll bar
contents area
“CDJukebox” folder
“Home Ent…” folder
…
size control
…
“Web Newspaper” window
66
…
Example 5.12
Display Screen
Outer Win [black]
93.54
Inner Win [green]
9
4
8
5
1
2
3
0
+
=
-
7
6
ENT
Result Win [tan]
Result String
Keypad [Teal]
= button
- button
+ button
0 button
67
Example 3.3 (revisited)
DemoSwing.java
How to determine the
containment hierarchy?
Keystroke combination:
CTRL-SHIFT-F1
68
Example 3.3 (revisited)
DemoSwing.java
DemoSwingFrame[frame0,0,0,224x208, ...
javax.swing.JRootPane[,4,23,216x181, ...
javax.swing.JPanel[null.glassPane, ...
javax.swing.JLayeredPane[null.layeredPane,0,0,216x181, ...
javax.swing.JPanel[null.contentPane,0,0,216x181, ...
javax.swing.JPanel[,0,0,216x181, ...
javax.swing.JButton[,50,50,116x27, ...
javax.swing.JLabel[,50,77,116x27, ...
javax.swing.JButton[,50,104,116x27, ...
E.g.
javax.swing.JButton[,50,104,116x27,
layout=javax.swing.OverlayLayout,
alignmentX=0.0,alignmentY=0.5,
border=javax.swing.plaf.BorderUIResource$CompoundBorderUIResource@cbf30e,
flags=296,maximumSize=,
minimumSize=,preferredSize=,defaultIcon=,
disabledIcon=,disabledSelectedIcon=,
margin=javax.swing.plaf.InsetsUIResource[top=2,left=14,bottom=2,right=14],
paintBorder=true,paintFocus=true,pressedIcon=,
rolloverEnabled=false,
rolloverIcon=,
rolloverSelectedIcon=,selectedIcon=,
text=Exit,
defaultCapable=true]
69
Example 3.3 (revisited)
DemoSwing.java
JButton
JFrame
Root Pane
Layered Pane
Content Pane
JPanel (container)
JLabel
JButton
70
Java’s Containment Hierarchy
In JFC/Swing, what is the basic building
block for a window? The JFrame
***
Content Pane is most important for our purposes
71
Example 5.14
Title
Atomic component
(text field)
Atomic component
(buttons)
1.2345
7
8
9
4
5
6
1
2
3
0
+
-
=
ENT
JFrame
content pane
JPanel
JPanel
JPanel
72
Containment Hierarchy for
JFC/Swing
JFrame
Content pane
JPanel
JPanel
JPanel
JButton
JButton
JLabel
etc.
etc.
JTextField
73
Outline
•
•
•
•
Windows
Icons
Menus
Pointers
• Window Managers
• More Widgets
74
Validating Input
•
•
JTextField provides one mechanism by which user can input data
What type of information might users provide?
– Examples:
• Numeric value (e.g., weight, age)
– Values belong to a continuum
– Acceptable values fall within a range
• Postal or zip code
• Date, time
•
How to handle input?
– Should it be in the format required by the application?
– The distinction between valid and invalid data
– Semantic Error vs. Format Error
•
When to Intervene:
– Solicit Input  Accept  Proceed to next step
•
Feedback: popup message, generate audio alarm, system
behaviour…
75
Examples 5.15, 5.16
DemoInputValidation1.java
DemoInputValidation2.java
76
Navigation
• The process of moving from one component to the
next
– Applies to both displayed and non-displayed components
• Recall: The attribute of being in focus or not is a state
(an attribute of all components)
– Feedback re: currently active component given to user
• e.g., I-beam cursor, special highlighting
• Options for Navigation
– Pointer: Click
– Keyboard: Tab or Enter
– Desired Behaviour: advance focus to the selected component
• Before leaving current field, should entry be accepted and validated?
• Should sequence pf next focus be determined by application or user?
77
Example 5.17
DemoTextField.java
78
Editable Combo Boxes
• Like a combo box, except allows the user
to directly type in an entry
• Much more complicated due to…
– Validating input
– Navigation
79
Example 5.18
DemoComboBox2.java
Similar to DemoComboBox.java (Example 4.5), except uses an
editable combo box to specify font size.
80
Toolbars
Recall (Module 4, slide 13):
• A collection of buttons
– typically arranged in a row or column
• The collection is persistent
• Appropriate for commonly used functions
– Provide convenient access to common menu
commands(contrast with menus)
• Meaning typically signaled by an icon rather than text
• Commands are displayed (requires ‘recognize’
rather than ‘recall’), but this requires display space
81
Example 5.19
DemoToolBar.java
82
Tool Tips
• A tool tip:
– a small text field that “pops up“ when the
cursor pauses over a GUI component
• What should the text do?
– provides a terse description of the command
associated with, or the purpose of, the
component
– be a supplement to the text or icon label
83
Tool Tips
• Advantages
– Aids in command recognition
– Does not consume screen space until shown
• Disadvantages
–
–
–
–
Can’t be selective
Can occlude other components
Terse wording may be inadequate
Frequent changes in appearance/disappearance
annoying to some users
84
Example 5.19 (revisited)
DemoToolBar.java
NB: cursor not captured by screen capture utility
85
Sliders
• Device for soliciting input
– domain of input values must correspond to a
pseudo-continuous variable
• Examples
– RGB values in a color chooser
– Time delays for animations
86
Example 5.20
DemoSliderBB.java
87
Example 5.21
DemoSlider.java
88
Scrollbars
Anatomy:
Arrow button
Track
Elevator (aka Knob)
Arrow button
89
Example 5.22
DemoScrollBarBB.java
90
Viewports and Scrollable Clients
Source: JScrollPane API Specification
91
Example 5.23
DemoScrollPane.java
92
Example 5.24
DemoScrollPane2.java
93
JList
• A component that allows a user to select
one or more items from a list
94
Example 5.25
DemoList.java
95
Example 5.26
DemoListMultiple.java
96
JTable
• A component that displays regular twodimensional tables of cells
– entries can be edited
97
Example 5.27
DemoTable.java
98
Dialog Boxes
• File Chooser
• Color Chooser
• Custom Dialog Box
99
Example 5.28
DemoFileChooser.java
100
Example 5.29
DemoMenu2.java
101
Example 5.30
DemoCustomDialog.java
102
Window Managers
• Need to know the window manager
when implementing user interfaces
– user will need certain functions
– the window manager provides the
framework
103
Structure of Window Manager
• Base layer:
– Implements the basic infrastructure
– Has an output model (graphics primitives)
– Has an input model (keyboard, mouse)
• UI layer:
– Handles all “visible” aspects
– Presentation (e.g., what is on top?)
– Commands (window & content manipulation)
• Mapping of input actions to applications
104
Structure of Window Manager
• When building a UI:
– use services of windowing system where possible
(rather than writing custom code)
105