CALIFORNIA STATE UNIVERSITY, NORTHRIDGE
SIMUI,TANEOUS PROCESSING OF AUDIO
AND VISUAL INPUTS IN A VIGILANCE TASK
A thesis submitted in partial satisfaction of the
requirements for the degree of Master of Arts in
Psychology
Human Factors/Applied Experimental
by
Francine Harriet Landau
August, 1981
The Thesis of Francine Harriet Landau is approved:
Dr. vVilliami:Hlsoncjoft
Dr. Tyler Bla}e
/Dr. M~J>CSanders
California State University, Northridge
i
TABLE OF CONTENTS
Page
ABSTRACT ..
.. i i i
It~TRODUCTION •••••••..•.•.••••.•.•••.••••.•• 1
METHOD ...... .
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
............................... .7
Apparatus . .. . ... . . . . . .. . . . . . .. . . ... . .. . 7
Subjects.
................... .7
Materials. .. . . . . . . . . . . . .. .... . 8
Design .•..
Stimulus
Procedure ........•.••.•....•....•..... 1 0
RESULTS •••••...•••••......•.
I
••••••
I
I
•••••
11
Table 1 ................................ 13
Table 2 .
•
•
•
•
•
•
•
•
•
•
•
•
Ill
••••••••••••••••
.14
Figure 1 ............................... 15
F i gu.re 2 • • • . • • • • , • • . • . • • . • ., • • • . • . . . . . . 1 7
Figure J .
Figure 4.
DISCUSSION.
•
•
•
•
•
•
•
•
•
•
•
•
•
0
••••••••••••••
.18
.. . ... . . .. . .. .. .. . ......... . .19
.............................. .20
REFERENCES • .. • • • • • • • • • • • • • • • • • .............. 3 2
ii
ABSTRACT
SIMULTANEOUS PROCESSING OF AUDIO
AND VISUAL INPUTS IN A VIGILANCE TASK
by
Francine Harriet Landau
Master of Arts in Psychology
Percent of correct detections was used to investigate
whether parallel processing was possible with auditory and
visual stimuli being presented simultaneously.
Sixty-four
male and female college students participated in a vigilance task.
Three factors, type of stimulus, signal pres-
entation, and response were manipulated in a betweengroups factorial design.
Stimuli were verbal or psycho-
physical; responses were verbal or manual; and signals
were presented synchronously (both input modes) or asynchronously (one input mode).
It was found that semantic
stimuli were harder for subjects to process accurately
than psychophysical stimuli and that detection scores
dropped in the second trial block, reflecting a vigilance
decrement.
Asynchronously presented auditory targets ir1
the second half of the session elicited the worst detection performance.
Several phenomena such as habituation,
visual dominance, and semantic interference were
iii
discussed which could account for the poor performance of
that group.
Evidence suggests that subjects were not
processing information in a parallel fashion but instead
were quickly switching attention between inputs or using
other optional attentional strategies that were available
as a result of the implementation of signal presentation.
iv
A classic question in the attention and performance
literature is whether two simultaneous inputs can be processed at the same time.
Broadbent's filter theory (1958)
proposed that there is a central processing mechanism,
i.e., attention, which is capable of handling information from only one "channel" at a time.
To deal with
simultaneous multiple inputs, attention is switched from
channel to channel, sampling information for a brief,
finite interval from each channel sequentially.
Atten-
tion is switched by a filter which handles inputs serially.
That is, one input is processed first, and the
filter only later retrieves the other item from storage.
If processing of the first stimulus in the central
system takes too long, the second stimulus will be lost
from sensory or memory storage.
In 1954 Broadbent in-
vented the split-span experiment in which subjects monitored, recalled, shadowed (verbally repeated inputs), or
replied to one or both of two simultaneous speech messages.
Much of the research supporting Broadbent's
theory was done with this kind of experiment (Ivlowbray,
1962, 1964; Moray & O'Brien, 1967; Trei.sman & Geffen,
1968).
The predominant finding in these experiments was
that subjects were only able to process information from
one channel at a time or that information from dual channels was processed serially.
In a typical split-span experiment, 'rreisman and
1
2
Riley (1969) had subjects shadow digits from one of two
dichotically presented lists and to detect infrequent
letters on either ear.
The subjects were to
i~mediately
stop shadowing when they heard the critical item.
Sub-
jects detected 76 percent of letters on the attended ear
and 33 percent on the other.
Although this result sup-
ported Broadbent's sequential processing hypothesis,
researchers felt that the experiment demonstrated that
attention cannot be divided between concurrent stimuli
if the listener is biased toward one channel by the
instruction to shadow one of the messages.
A subsequent
study by Tulving and Lindsay in 1970 corroborated this
result.
In contrast to these experiments, other studies
demonstrated that parallel processing of concurrent
verbal stimuli was possible.
A study by Moray (1959)
showed that people often noticed their spoken name while
they were listening to another message.
l.'Unio and
Kahneman (1973) found that reaction time was approximately the same in both a divided attention and focused
attention task.
Further, subjects detected 77 percent
of the targets in the divided attention task in which
they had to press a key whenever a target word was heard
while listening to dichotically presented word lists.
Although evidence for parallel processing of diehotically presented stimuli is sparse, it is clear that under
J
p '
some conditions we can cope with two informative inputs at
least partly in parallel.
The variations among type of
stimulus, response, and results for these experiments were
so great that Treisman (1969) postulated a new hypothesis
to help structure the heterogeneous findings.
She sug-
gested that independent analyzers (Sutherland, 1959) code
different aspects or dimensions of incoming stimuli such
as color, pitch, loudness, and spatial location.
If tasks
involve inputs which converge on the same analyzing system, it would seem that serial processing must take place.
On the other hand, if separate analyzing systems are invoked by one or more inputs, parallel processing may be
effected.
This theory predicts that the subject should
be able to monitor two messages as easily as one, provided
that the messages do not reach the same analyzer.
Yntema
and Schulman (1970) supported this logic when they found
that subjects had less difficulty keeping track of states
of different variables than monitoring different states
of the same type of variable.
Using this hypothesis as a departure, experiments
began to include the case in which different modalities
were used as input sources for the divided attention
taslcs.
Allport, Antonis, and Reynolds (1972) found that
subjects could carry out two tasks in parallel (shadowing speech while sight reading music) with little or no
interference.
Although it is evident that parallel
4
processing can take place, researchers were interested in
determining when dual inputs were most efficient, and for
what kind of task.
Although the findings in experiments using complex
verbal stimuli were inconsistent, there was a preponderance of evidence supporting the superiority of dual mode
inputs in vigilance tasks when stimuli were simplistic
and psychophysical, e.g., audio stimuli that changed frequency or intensity, visual stimuli that became brighter
or changed color.
Typically, these experiments had sub-
jects monitoring inputs for extended periods (90 minutes)
and the subjects were trained for the task to eliminate
As early as 1958, Klemmer demon-
any practice effects.
strated that a simultaneous presentation of auditory and
visual signals (each reinforcing the other) against a
background noise condition produced more efficient reaction time and higher detection rates than either a single
visual or audio signal.
\j) and~olquhoun
Tyler, Waag, and Holcomb (1972)
(1975) all found that redundant inputs from
both audio and visual sources elicited superior detection
performance in a vigilance
task.~
In all these experiments the dual mode condition was
represented by simultaneous redundant presentation of
stimuli and signals on both audio and visual inputs, i.e.,
the signal was presented synchronously on both input
sources every time.
In a 1965 study, McGrath used a
5
different implementation of signal presentation:
back-
ground stimuli were presented in both modes once every
three seconds but the signal was presented on only one
source aperiodically (asynchronously).
His investigation
also included a distinction between "easy" detection tasks
in which subjects detected the signal 90 percent of the
time versus a "difficult" detection task in which subjects could detect the signal only
65 percent of the time.
With stimuli being presented simultaneously and signals
asynchronously, McGrath found vigilance performance enhanced for the easier task only.
The lack of signal re-
dundancy and the uncertainty about where the signal would
originate may have introduced additional processing demands which would have been responsible for the differential enhancement of detection.
In a 1979 dual mode study, Whitaker investigated how
response time in a monitoring task was influenced by set
size of response choices.
The implementation of the dual
task condition included both a verbal response to an audio
input and the addition of another visual tracking task
which was done concurrently.
This evidently exceeded the
capacity of the subject to perform parallel processing
because response time was much slower for the dual task
condition.
Thus, the range of parameters facilitating
parallel processing in vigilance tasks may be Yery small.
The enhancement effect of dual inputs found in the
6
vigilance literature is in direct opposition to results
with verbal split-span experiments where reliable parallel processing of dichotically presented inputs has not
been demonstrated.
One important difference seems to be
whether signals appeared simultaneously (synchronously)
or alternately (asynchronously) on both input sources.
Fbr this reason the present study investigated both synchronous and asynchronous signals presented in a dual
mode task.
It was hypothesized that synchronous presen-
tation of signals would enhance detection performance.
Another aspect of dual mode tasks which appeared to influence the possibility of efficient parallel processing
was the type of stimulus used.
In the verbal learning
literature previously reviewed, the content of research
was predominantly semantic.
stimuli were psychophysical.
In the vigilance literature,
These two stimuli types
were combined in the present study as two levels of a
stimulus factor.
It was hypothesized that simplistic
inputs would be easier to process than semantic inputs.
Each area of research was also characterized by one set
of responses.
The verbal learning literature typically
used variations of verbal responses while, in the vigilance literature, motor responses were used most often.
Using the empirical evidence from the vigilance experiments as a rationale, it was predicted that motor responses would interfere less than verbal responses with
simultaneous processing.
Method
Subjects
The subjects were 19 male and 45 female college sophomores and juniors with normal vision and hearing who were
required to accumulate experimental credits for their
Psychology courses.
Subjects were screened on a pretest
of the task and had to obtain a detection score of 60
percent in order to be accepted into the experiment.
Only
one person was dropped for this reason.
Apparatus
The equipment included a Panasonic TV monitor and
p;ortable video recorder NV3085 with tape meter; two pairs
of Telephics TDH-49 monaural earphones with separate volume controls, one for the subject and one for the experimenter; a two button response box connected to a 12.7 em
x 17.8 em lightboard indicator with a red and green light
bulb.
Each push button, labeled with the appropriate res-
ponse word ("see" or "hear") was connected to a light bulb
which would light when the subject pushed the button.
power supplies were used:
Two
a Panasonic AC adaptor NV-B40
and a BRS electronics model PS-12b.
Design
Three independent variables, type of stimulus, signal
presentation, and response were investigated in a 2x2x2
between groups factorial design.
7
Since stimuli were
8
presented both visually and aurally, the two levels of
stimulus type were dots and printed words for visual presentation, and tones and spoken words for auditory presentation.
The two levels of signal presentation were syn-
chronous and asynchronous; in the synchronous condition,
two signals were presented simultaneously, one visually
and one auditorily.
In the asynchronous condition, the
signals were presented aperiodically, either on the TV or
over the earphones, but never at the same time.
Responses
could be either button pushes or verbal reports ("see .. or
"hear").
The assignment of subjects to conditions and the
order of words and signals was randomly determined.
Sig-
nals were sequenced during the session so that no temporal
or spatial pattern was discernible and they occurred, on
the average, once every 2.15 minutes.
The dependent var-
iable was percent of correct detections.
Stimulus Materials
Four tapes, each .30 minutes long, were produced for
combinations of the two independent variables, stimulus
type and signal presentation type.
Each tape contained
900 stimulus presentations for both the visual and auditory modes.
t
Each stimulus was presented for a duration of
second and the interstimulus interval was 1.5 seconds.
The following is a description of the contents of the
four tapes.
Synchronous-Dot-Tone.
A 12.7 mm black dot subtending
9
a visual angle of 1° 12' appeared on the center of a
white background on the TV screen every two seconds.
At
the same time, a 300 Hz, 50 db. tone was played every two
seconds over the earphones.
The signals in this condition
were a 19.05 mm black dot, subtending a visual angle of
1° 48' and a 300 Hz, 70 db. tone.
Fourteen signals were
aural and fourteen were visual, totaling 28.
These sig-
nals were always presented at the same time.
Asynchronous-Dot-Tone. A 12.7 mm black dot appeared
on a white background every two seconds on the TV screen
while a 300 Hz, 50 db. tone was heard through the earphones at the same time.
A 19.05 mm black dot, the visual
signal, appeared seven times during the 30 minute run.
A
300Hz, 70 db. tone, the audio signal, occurred seven
times during the session but never at the same time as
the visual signal.
Synchronous-Verbal-Verbal. A monosyllabic word, e.g.,
"know", "dish", "cool", appeared every two seconds in the
center of the TV screen.
At the same time a monosyllabic
word was heard over the earphones at exactly the same rate
and duration.
The signal was a word of the animal class
such as "dog", "snake", or"cat".
The words subtended an
average visual angle of 4° 10'.
There were 14 visual and
14 audio signals always presented at the same time, totaling 28.
As~nchronous-Verbal-Verbal.
The stimuli and signals
10
were the same as the previous condition, however, a total
of 14 signals were presented, seven by earphones, and
seven on the TV.
Audio and visual signals never appeared
at the same time.
Illumination in the room while subjects were taking
part in the experiment was 2766.3 lx; the luminance contrast between stimuli and background in the dot-tone and
verbal conditions were 94 and 90 percent respectively.
Because of an afterimage problem with the visual stimuli,
a second light field of the same intensity as the background came on at the offset of the stimulus and went off
at the onset of the stimulus.
Because of an anticipated
summation phenomenon on the auditory tapes, a 25 db.,
1kHz bandwidth of white noise was played during the ISI.
This white noise was also present on the verbal stimulus
tapes.
Procedure
The subjects were asked to sit directly in front of
the TV screen which was approximately 69 em away from
their face.
The type of visual stimulus was displayed
on a card, either a dot or word, and then the type of
signal was explained.
For the dot-tone condition, the
signal was the larger dot; for the word condition, it
was an animal word.
The appropriate response upon iden-
tifying a signal was explained and then subjects practiced identifying four visual signals.
After practice
with the visual stimuli, the auditory stimuli and signals
were demonstrated using the earphones.
viTi th the TV off,
the subjects practiced identifying four auditory signals,
either a louder tone, or an animal word.
If they had
trouble hearing and/or seeing two of four signals, they
were thanked, given credit, and dropped from the experiment.
After the subjects practiced monitoring each mode
separately, they practiced monitoring both modes together
for seven minutes.
If they failed to detect at least 60
percent of the targets, they were not used in the experiment.
The experimenter then changed tapes while the par-
ticipants took a five minute break in another room.
The
experimental run took 30 minutes, and at the end of the
session, subjects were given a debrief sheet.
Results
Using a three-way factorial ANOVA, it was found, as
hypothesized, that dot-tone stimuli were easier to process
than verbal stimuli, F (1,56)
= 205.323,
p <.001.
Since
19 of the 32 subjects in this condition scored 100 percent, it was felt that the homogeneity of variance assumption for ANOVA could not be met.
In addition, this ceil-
ing effect would have made all other statistical results
suspect.
Therefore, only the verbal stimulus data were
used and a new four-way ANOVA was performed.
The two re-
maining between subject factors--response mode and signal presentation--were combined with two repeated measures
11
12
--trial blocks and stimulus mode.
For the trial blocks
factor, the experimental session was divided in half,
giving two detection scores.
These detection scores were
again separated into auditory and visual target detection
scores for the stimulus mode factor.
Although false alarm
data was collected, the highest group mean for false
alarms was 7.0 in a session that contained 900 stimulus
presentations.
This data was not included in any further
analysis since these responses were too infrequent to
yield meaningful interpretations.
Using percent of cor-
rect detections as the dependent variable, Table 1 contains the means and standard deviations for the four-way
analysis data.
Table 2 contains the F ratios ~s well as eta 2 for
the four-way ANOVA data.
A significant trial blocks by response interaction
was found which is illustrated in Figure 1.
A Newman-
Keuls test indicated that correct verbal responses dropped
significantly from trial block 1 to trial block 2, Grit.
Diff. (32)
= 17.19,
p<. .05, while correct button responses
did not change with time on task.
All other comparisons
among trial block and response groups were not significant.
Although the next two significant interactions identified by the ANOVA will be reported here, it should be
noted that they are in turn modified by another variable
13
Table 1
Mean Percent and Standard Deviations for Correctly Identified
Audio and Visual Targets in Two Trial Blocks
Trial Block
2
1
Synchronous
Stimulus
:.:ode
Audio
7isual
Response
Mode
Verbal
92.85
(
Button
87.49
Verbal
Button
7.64)a
73.2
(19.37)
(14.17)
91.08
(20.10)
92.85
( 7.64)
'?4.99
(19.84)
78.56
(22.90)
87.49
(14.17)
Asynchronous
Audio
Tisual
1
Verbal
9.5.84
{11. 77)
58.J4
(34. 51)
Button
8J.J4
(1.5.42)
62.49
(JJ,05)
Verbal
84.J8
(22.90)
90.63
(12.94)
Button
81.78
(18.35)
79.69
(18.32)
aStandard deviations in parentheses.
14
Table 2
ANOVA Ratios for Four-Way Target Detection Study
Source
s~
eta2
.!U:
!§.
!
62.)0
1
62.)0
.09
Presentation
{Syn-Asyn)
88:3.58
1
88).58
1.22
1. 5:3
Response x Presentation
5:31.79
1
531.79
,7J
.92
20261.4.8
28
72:3.62
Trials (Blk 1-Blk .2)
:3135.:33
1
:31:35-JJ
11.77
.002
5.4-J
Trials x Response
1701.6:3
1
1701.63
6.)9
.017
2.95
Trials x Presentation
425.96
1
425.96
1.60
.74
Trials x Response x
Presentation
869.97
1
869.97
).27
1.50
7456.97
28
266.)2
Mode (Audio-Visual)
:3:31.21
1
:3:31.21
1.21
.57
Mode x Response
189.88
1
189.88
.69
,JJ
l11J.JJ
1
111J.JJ
4,06
41.52
1
41.52
.15
Error
768?.52
28
274.55
Trial x Mode
2424.69
1
2424.69
229.78
1
229.78
15JJ.89
1
15JJ.89'
5.11
4-<>7.20
1
407.20
1.36
Error
8404.19
28
)00.15
Total
57692.22
127
Response
· (Manual-Verbal)
Error
Error
Mode x Presentation
Mode x Response x
Presentation
Trial x Mode x Response
Trials x Mode x
Presentation
Trial x Mode x Response
x Presentation
l2.
.1
.054
1.93
.07
a.oa .ooa
4,2
.40
.77
.0)2
2.66
.70
~
15
95
93
91
91.48
89
87
Percent
85
Correct
83
Detections
81
82.79
80.23
79
77
75
74.29
73
Trial Block
Trial Block
1
2
Figure 1. Interaction of trial blocks and
response type with group means.
16
in a significant three-way interaction.
Stimulus mode and
signal presentation interact significantly and are graphed
in Figure 2.
Synchronous audio and visual signals were
equally hard to detect as were asynchronous visual signals.
However, the percent of correctly detected asyn-
chronous audio signals was much lower than any of the
other three groups.
Since a third variable, trial blocks,
further modifies this interaction between stimulus mode
and signal presentation, no other ad hoc comparison tests
were made.
A significant trial blocks by stimulus mode interaction was also found and this is illustrated in Figure
J.
Audio signals were harder to detect in trial block 2 than
in trial block 1.
Subjects were able to detect visual
signals equally well in trial block 1 or trial block 2.
Again, the probability of detecting the two types of stimuli depended on whether the subject was identifying asynchronous or synchronous signals in trial block 1 vs.
trial block 2.
in Figure 4.
This three-way interaction is illustrated
The interaction of trial blocks and stimulus
mode was not significant with synchronously presented signals but was with asynchronously presented signals.
A
Newman-Keuls test indicated that asynchronous audio signals became much harder to correctly identify in trial
block 2 than asynchronous visual signals in that trial
block, Crit. Diff. (16) = 24.75, J2.C..05.
Both synchronous
1'1
95
93
91
89
87
Percent
85
Correct
83
Detections
86.15
Visual
83.47
84.12
81
79
77
75
75.0
73
Asynchronous
Synchronous
Presentation
Figure 2. Interaction of signal presentation
and stimulus mode with group
means.
18
91
89
89.88
87
85
Percent
83
Correct
81
Detections
79
Visual
84.39
83.20
77
75
73
Audio
71
71.26
69
Trial Block
Trial Block
1
2
Figure 3. Interaction of trial blocks and
stimulus mode with group means.
19
91
89
87
·as
90.17
85.7
a:;
Percent
Trial Block 2
Trial Block 1
a. Synchronous Presentation
Correct
Detections
82.14
81.18
81
95
9:3
91
89
87
89.59
as
a:;
81
79
77
75
73
71
-Visual
85.16
8J.07
69
67
65
63
61
59
60.41
Trial Block 2
Trial Block 1
b. Asynchronous Presentation
Figure 4. Intaraction of trial blocks, signal
presentation, and stimulus mode with
group means.
audio and visual signals were slightly harder to detect in
trial block 1 than trial block 2, but not significantly
so.
The greatest drop in performance occurred between
trial block 1 and trial block 2 with asynchronous audio
targets, Crit. Diff. (16)
= 29.18,
p
~
.05.
Performance
with asynchronous visual targets did not change significantly from trial block 1 to trial block 2.
Thus, the
hardest signals to detect in this experiment were asynchronous audio signals in the second trial block.
The
three variables, stimulus mode, signal presentation, and
trial blocks, modified the effect of each other so that
differential performance occurred depending on the specific combination of levels experienced.
Discussion
The original hypothesis that semantic material is
more difficult to process than simple stimuli such as dots
and tones was supported in this experiment.
The mean per-
cent of correct detections for verbal stimuli was 82.1 as
opposed to 95.8 for the dot-tone stimuli.
As previously
mentioned, only verbal data were used in the analysis because of a ceiling effect in the dot-tone data.
The hypo-
thesis that motor responses would have a generally positive influence on detection behavior was not supported in
this experiment.
In addition, no main effect of signal
presentation was found.
In general, the subjects in all
groups, except one, had little difficulty processing
20
21
inputs from dual sources.
The mean percent correct detec-
tions ranged from 73.2 to 95.0.
If subjects could not
process both inputs, one would expect that performance for
at least one input would be reduced to chance.
This was
not the case and these results support other research
(Allport, Antonis, & Reynolds, 1972; Colquhoun, 1975;
McGrath, 1965; McLeod, 1977; Treisman & Davies, 1973;
Tyler, Waag, & Holcomb, 1972) which indicated that detection performance in dual mode tasks was as good or better
than in single mode tasks.
A significant effect of trial blocks indicated that
performance in the first helf of the session was superior
to performance in the second half.
tions dropped from 92.1 to 86.0.
Mean percent of detecEvidently, subjects can
process simultaneous stimuli, but it is not clear for how
long.
This drop in accurate detection has been inter-
preted as reflecting the commonly found vigilance decrement.
This effect has been noted and explored since
Mackworth first investigated the vigilance task in 1950.
Many theories have been advanced to explain this phenomenon; all of them predict vigilance behavior fairly well
but none of them explain all vigilance data (Loeb &
Alluisi, 1980).
A very cursory review of the current
major theories explaining the vigilance decrement will be
described here with no attempt to ascribe priority.
Arousal and habituation are two neurological models
22
which have been investigated in the literature.
Arousal
is a measure of how wide awake the organism is, of how
ready it is to react (Berlyne, 1960).
The ascending retic-
ular activating system and the diffuse thalamic projection
system are portions of the brain thought to mediate
arousal.
There is an optimum level of arousal for per-
formance, but with time on task, arousal tends to decline,
with subsequent deterioration in performance.
Habituation is a decline in perceptual sensitivity of
the observer because of repeated exposure to non-signal
stimuli (Kahneman, 1973), again producing a decrement in
detection performance.
In this context, habituation of
the orienting response, a response to novel stimuli such
as a signal, has been noted by researchers (Coquery, 1978;
Kahneman, 1973; Mack, 1970; Stroh, 1971).
As an indica-
tion of neurological involvement, the magnitude of evoked
potentials within the brain has been measured.
Haider
(1967) found a decline in amplitude of evoked cortical
potentials with a concomitant decline in percent of correct detections.
Other researchers have found similar
results (Davies & Krkovic, 1965; Mackworth, 1970; Picton,
Campbell, Baribeau-Braun, & Prouix, 1978; Stroh, 1971;
Wilkinson, Morlock, & Williams, 1966).
A third hypothesis suggests that subjects expect
unrealistically high rates of signal presentations and
that decrements occur because signal rates are lower
23
(Baker, 1959; Colquhoun & Baddeley, 1964; Craig, 1978:
Deese, 1955; Thomas, 1971).
A general probability match-
ing trend of the subject becomes more obvious during the
experimental run as he progressively tries to match his
propensity to detect a stimulus with the ratio of signals
to non-signals (Loeb & Alluisi, 1980).
Using a signal detection framework for interpretation
of vigilance decrements, Swets in 1977 concluded that subjects adopt a stricter criterion as sessions go on in
which they learn to better identify non-signals from signals and are less inclined to detect a stimulus as a signal.
Again, there is much support for this model
(Broadbent & Gregory, 1965; Colquhoun, 1967a; Egan,
Greenberg, & Schulman, 1961; Loeb & Alluisi, 1980).
Most probably all these models reflect ongoing processes within the vigilance situation which could account
for the drop in accuracy in this experiment.
However, an
investigation by Parasuraman (1979) seems particularly
relevant to this study.
He used similar parameters of
task, load, and stimuli type to determine whether the,
vigilance decrement was the result of a loss of perceptual sensitivity or a change in response criterion. Event
rates of non-target stimuli were presented either once
every two seconds or once every four seconds.
The tasks
were either successive discrimination, i.e., the signal
was presented alone after many repetitions of non-signal
stimuli, or simultaneous discrimination where the signal
was presented within a field of non-signal stimuli.
Parasuraman found that the decrement was due to a loss of
perceptual sensitivity when signal discrimination loads
memory (successive discrimination) and when stimulus
events occur rapidly (once every two seconds).
More interesting is the finding that with time on
task, verbal responses had a negative influence on performance as opposed to motor responses which elicited similar
performance for both time blocks.
This result partially
supports the initial hypothesis that verbal responses may
be interferred with by verbal encoding of stimulus inputs.
McLeod (1977) analyzed response interference in dual mode
tasks and concluded that the similarity of task and response could predict the level of interference which would
be reflected in poor performance.
Using two levels of
difficulty in an arithmetic task, he found no debilitating
effect on manual responses while significant interference
was obtained using verbal responses.
The finding that
dimensions of tasks that use common analyzers generate
the most interference has been found by many investigators
(Kinsbourne & Hicks, 1978; McLeod, 1977; Treisman & Riley,
1969).
Evidently, as interference from verbal inputs and
fatigue increased, verbal responses may have demanded more
processing time which subjects did not have with the fast
presentation of stimuli in this study.
25
The question of whether subjects can receive and process parallel inputs has to be addressed in determining
the cause of the
three~way
interaction between trial
blocks, stimulus mode, and signal presentation.
Most in-
vestigators support the notion that subjects can receive
information in parallel but disagree on the possibility of
higher order parallel processing.
The evidence, here,
suggests subjects can process dual inputs in the synchronous condition but their performance may have been due to
optional attentional strategies and not to valid higher
order processing in parallel.
Because in the synchronous
mode signals were always presented on both inputs at the
same time, subjects became aware that when an audio signal
was presented, a visual signal was also being presented.
This cueing not only heightened short term attention but
alleviated the demand for constant attention to both modes
throughout the session.
Subjects could switch attention
from one mode to another at will, thereby attenuating the
effects of habituation, boredom, and fatigue.
This strat-
egy may offer an explanation for the positive findings in
the vigilance literature where redundant signals were
efficiently processed in dual mode tasks.
It may also
explain why performance was maintained as well as it was
in trial block 2 for those subjects seeing only synchronous signals.
Subjects had much more difficulty processing
26
asynchronously presented audio signals in the second half
of the experiment.
It is more certain that subjects were
forced to either process inputs in parallel in the asynchronous condition or switch attention very quickly (250
msec) between inputs because of the lack of any other
effective strategy.
In trial block 1 subjects were able
to process both input sources but in trial block 2 performance seems to break down, especially for audio signals.
There are two general phenomena which debilitate performance in vigilance tasks and both were present in the
asynchronous condition.
uncertainty.
These are temporal and spatial
Temporal uncertainty, the irregular pres-
entation of infrequent signals, has been the subject of
many vigilance experiments and has elicited fairly consistent results with both reaction time and correct percent of detections as dependent variables.
When there is
no pattern for signal presentation, subjects don't know
how to allocate attention and this results in missed signals
(Adams & Boulter, 1964; Alegria & Bertelson, 1970;
Baker, 1959b; McCormack & Prysiazniuk, 1961; Webber &
Adams, 1964).
Although temporal uncertainty may have a
transient arousal effect, it is debilitating in tasks of
long duration.
The concept of spatial uncertainty, not knowing on
which source the signal will appear, has been investigated
by Adams and Boulter (1964) in conjunction with the
27
effects of temporal uncertainty.
Three signal sources
were manipulated with differing levels of temporal certainty.
It was found that latency was minimal for either
spatial and temporal certainty, but maximal for both kinds
of uncertainty.
Kulp and Alluisi (1967) investigated the
same phenomena and had concluded that the contributions of
different sources of uncertainty were linearly additive.
These two sources of uncertainty may explain why subjects were under cumulative stress by the second time
block, but it does not explain why this stress interferred
with the detection of audio signals only.
This decrement
in detection for auditory signals in vigilance tasks has
been documented throughout the literature (Baker &
Harabedian, 1962; Buckner & McGrath, 1965; Gruber, 1964;
Rollins, Jr. & Hendricks, 1980).
Parsuraman (1979) com-
pared 13 dual input investigations finding a vigilance
decrement.
the
s~~dies.
He found an auditory decrement in eight of
Craig, Colquhoun, and Corcoran (1976),
attempting to account for this effect, proposed that, as
sessions went on, subjects adopted a more conservative
criterion for detecting auditory targets, with a concomitant drop in correct detection.
Therefore, the worst
performance in this experiment may have been the result of
subjects using a stricter criterion for identifying audio
signals at the same time signal presentation uncertainties
were creating additional stress for the subjects.
28
Although these are plausible explanations for the
poor processing of asynchronous audio signals, another
hypothesis will be discussed to account for the drop in
accuracy for aural signals.
An assumption made by this
researcher and supported by the slightly better detection
performance of asynchronous visual signals in trial block
2 was that subjects consciously attended the visual display and processed visual inputs first.
Posner, Nissen,
and Klein (1976) have reported on the tendency of subjects
to attend to visual inputs in preference to auditory inputs when both are presented.
This appears to be related
to the superior alerting capacity of the auditory system
and the relatively weak capacity of the visual system to
alert the organism.
Subjects, assuming they will be auto-
matically alerted by auditory inputs and compensating for
the weak alerting capacity of visual inputs, deliberately
keep their attention tuned to the visual inputs.
In a
1974 study, Colvita asked subjects to press one key whenever a light came on and another key whenever a tone
occurred.
When the signals were presented simultaneously,
the subjects generally responded to the visual inputs and
were often unaware that an auditory signal had occurred.
When informed that signals could occur on both modes, subjects still attended to the visual inputs and detection
for those stimuli was better.
If subjects were watching
for the visual signal, they had approximately 250 msec
29
to receive and encode that stimulus.
Within the next 250
msec the subject had to receive and encode the auditory
stimulus.
If this sequence was somehow disrupted, the
visual signal had the higher probability of being processed properly.
Interference in cognitive processing may
have occurred at two levels, short-term memory and/or
semantic analysis, and could have caused information from
the second stimulus to be lost.
Lindsay (1970) reported
on a short-term memory study using dual inputs in which
subjects were asked to report judgments about psychophysical stimuli in a specific order.
The dimension reported
second suffered a substantially larger impairment than the
first dimension reported on a trial.
In another dual task
study, Lindsay (1970) commented that subjects perform more
efficiently when attending to different dimensions of the
same type of stimulus than attending to the same dimension
of different stimuli.
This implies that specific ana-
lyzers with limited capacity are available, subsequently
restricting the amount of processing per unit time.
In
this study, semantic analysis of the visual word, which
was received first, may have interferred with the semantic
analysis of the auditory word which was processed second.
Therefore, problems with both short-term memory and semantic analyzers may have caused the decline in auditory
detection performance.
This result lends support to the
single channel filter theory of Broadbent which states
30
that multiple inputs are serially processed.
In summary, this experiment provides results which
are congruent with both vigilance and verbal learning
research utilizing dual inputs.
With redundant signals,
processing of dual inputs is possible because of alternative attentional strategies which optimize switching
techniques.
Information is not being processed simulta-
neously from input to output.
Motor responses typically
used in vigilance experiments did not interfere with cognitive processing in that research and did not interfere
with semantic processing in this study.
When subjects in
the asynchronous condition were forced to process inputs
in parallel, they could do it for a short period of time.
Again, it is hypothesized that they accomplished this by
quickly switching attention.
Interference in limited
capacity analyzers or processors soon made this technique
unsuccessful.
The verbal learning experiments reflected
this situation where both stimuli and responses were analyzed by the same cognitive structures and performance was
subsequently degraded.
Further research should attempt to isolate where in
the input-output process semantic interference is most
likely to occur, and whether coupling manual responses
with semantic inputs would eliminate that interference and
facilitate performance.
In addition, evidence from this
study implicates the auditory system as more vulnerable
31
to habituation with extended task time.
This result
should be explored and verified so that future investigations can account for this phenomenon when designing
dual mode tasks.
32
References
Adams, J.A., & Boulter, L.R. Spatial and temporal uncertainty as determinants of vigilance behavior. Journal
of Experimental Psychology, 1964, £1, 127-131.
Alegria, J., & Bertelson, P. Time uncertainty, number of
alternatives and particular signal-response pair as
determinants of choice reaction time. Acta
Psychologica, 1970, ]1, 42-50.
Allport, D.A., Antonis, B., & Reynolds, P. On the division of attention: A disproof of the single channel
hypothesis. Quarterly Journal of ExDerimental
Psychology, 1972, 24, 225-235.
Baker, C.H. Attention to visual displays during a vlgllance task. British Journal of Psychology, 1959, 2Q,
30-36. (Acta Psychologica, 1970, 11• 122-131.)
Baker, C.H. Towards a theory of vigilance. Canadian
Journal of Psychology, 1959b, 1lr 35-42. Cited by
D.R. Davies & G.S. Tune, Human Vigilance Performance,
New York1 American Elsevier Publishing Company, Inc.,
1969.
Baker, C.H., & Harabedian, A. Performance in~ auditory~
vigilance task while simultaneousl* tracking a visual
target. Technical Report No. 740- 2, 1962. Cited by
J. Mackworth, Vigilance and Attention: A Signal
Detection Anproach, Baltimore: Penguin Books, 1970.
Berlyne, D.E. Conflict, arousal and curiosity.
McGraw Hill, 1960.
New York:
Bernstein, I.H. Can we see and hear at the same time?
Acta Psychologica, 1970, Jl, 21-35.
Broadbent, D.E. Perception and communication.
Pergamon Press, Ltd., 1958.
London:
Broadbent, D.E., & Gregory, M. Effects of noise and of
signal rate upon vigilance analysed by means of decision theory. Human Factors, 1965, L• 155-162.
Colquhoun, W.P., & Baddeley, A.D. The role of pretest
expectancies in vigilance decrement. Journal of
Experimental Psychology, 1964, 68, 156-160.
JJ
Colquhoun, W.P. Sonar target detection as a decision
process. Journal of Applied Psychology, 1967a, ji,
187-190.
Colquhoun, W.P. Auditory, visual, and dual mode displays~
for prolonged sonar monitoring in repeated sessions.
Human Factors, 1975, !1(5), 425-4]7.
Colvita, F.B. Human sensory dominance.
Psychophysics, 1974, 16, 409-412.
Perception and
Coquery, J.M. Selective attention as a motor program. In
J. Requin (Ed.), Attention and performance VI. Hillsdale, New Jersey: Lawrence Erlbaum Assoc., 1978.
Craig, A., Colquhoun, W.P., & Corcoran, D.W. Combining
evidence presented simultaneously to the eye and the
ear: A comparison of some predictive models.
Perception and Psychophysics, 1976, 12(6), 47]-484.
Craig, A. Is the vigilance decrement simply a response
adjustment toward probability matching? Human
Factors, 1978, 20, 442-446.
Davies, D.R., & Krkovic, A. Skin conductance, alpha
activity and vigilance. American Journal of
Psychology, 1965, ~. 304-306. Cited by D.R. Davies &
G.S. Tune, Human Vigilance Performance, New York:
American Elsevier Publishing Company, Inc., 1969.
Davies, D.R., & Tune, G.S. Human vigilance Derformance.
New York: American Elsevier Publishing Company, Inc.,
1969.
Deese, J. Some problems in the theory of vigilance.
Psychological Review, 1955, 62, 359-386. Cited by
J. Mackworth, Vigilance and Attention: A Signal
Detection Approach, Baltimore: Penguin Books, 1970.
Deutsch, J.A., & Deutsch, D. Attention: some theoretical
considerations. Psychological Review, 1963, 1Q, 8090. Cited by D. Kahneman, Attention and Effort,
Englewood Cliffs, New Jersey: Prentice-Hall, Inc.,
1973.
Egan, J.P., Greenberg, G.Z., & Schulman, A.I. Operating
characteristics and the method of free response.
Journal of the Acoustical Society of America, 1961,
Jl, 993-1007.
34
Gruber, A. Sensory alternation and performance in a vigilance task. Human Factors, 1964, 6, 3-12.
Haider, M. Vigilance, attention, expectation, and cortical evoked potentials. In A.F. Sanders (Ed.),
Attention~ ~erformance, Amsterdam:
North Holland
Publishers, 19 7.
Kahneman, D. Attention and effort.
Hall, Inc., 1973.
-
New Jersey:
Prentice-
Keele, S. Attention and human performance. Pacific
Palisades, California: Goodyear Publishing Inc.,
1973.
Klemmer, E.T. Timesharing between frequency-coded auditory and visual channels. Journal of Experimental
Psychology, 1958, jQ, 229-235.
Kulp, R.A., & Alluisi, E.A. Effects of stimulus-response
uncertainty on watchkeeping performance and choice
reactions. Perception and Psychophysics, 1967, ~,
511-515.
LaBerge, D. Identification of two components of the time
to switch attention: A test of a serial and a parallel mod~l of attention. InS. Kornblum (Ed.),
Attention and performance IV, New York: Academic
Press, 1973.
Lindsay, P·.H. Multichannel processing in perception. In
D. Mostofsky (Ed.), Attention: contemporary theory
and analysis, New York: Appleton-Century-Crofts,
1970.
.
Loeb, M. On the analysis and interpretation of vigilance:
some remarks on two recent articles by Craig. Human
Factors, 1978, 20(4), 447-451.
Loeb, M., & Alluisi, E.A. Theories of vigilance: A
modern perspective. Proceedings of the 24th Annual
Meeting of the Human Factors Society, 1980, 600-604.
Mackworth, N.H. Researchs in the measurement of human
performance. In L. Sinaiko (Ed.), Selected Papers
gn Human Factors in the Design ~ Use of Control
Systems, London: Dover Publications, 1955.
Mackworth, J.F. Vigilance and attention: _A signal detection approach. Baltimore: Penguin Books, 1970.
35
McCormack, P.D., & Prysiazniuk, A.W. Reaction time and
regularity of interstimulus interval. Perceptual and
Motor Skills, 1961, 12• 15-18.
McGrath, J.J. Performance sharing in an audio-visual
vigilance task. Human Factors, 1965,
141-153.
z,
McLeod, P. A dual task response modality effect: support
for multiprocessor models of attention. Quarterly
Journal of Experimental Psychology, 1977, ~~ b51-667.
Moray, N. Attention in dichotic listening' affective
cues and the influence of instructions. Quarterl~
Journal of Experimental Psychologr, 1959, 11, 56- o.
Moray, N., & O'Brien, T. Signal-detection theory applied
to selective listening. Journal of-the Acoustical
Society of America, 1967, 42, 765-772.
Moray, N. Where is capacity limited? In A.F. Sanders
(Ed.), Attention and performance 1, Amsterdam:
Humanities Press, 1970.
Mowbray, G.H. Simultaneous v1s1on and audition: the
comprehension of prose passages with varying levels
of difficulty. Journal of Experimental Psychology,
1953, 46, 365-372.
Mowbray, G.H. Some human perceptual imits, APL Technical
Digest, 1962, 21-24. Cited by D. Kahneman, Attention
and Effort, Englewood Cliffs, New Jersey: PrenticeHall, Inc., 1973.
Mowbray, G.H. Perception and retention of verbal information presented during auditory shadowing. Journal
of the Acoustical Socie~ of America, 1964, 36, 1459-
1465.
Ninio, A., & Kahneman, D. Reaction time in focused and in
divided attention. Journal of Experimental Psychology,
1973 in press. Cited by D. Kahneman, Attention and
Effort, Englewood Cliffs, New Jersey: Prentice-Hall,
Inc., 1973.
Parasuraman, R. Memory load and event rate control sensitivity decrements in sustained attention. Science,
1979, 205, 924-926.
Picton, T.W., Campbell, K.B., Baribeau-Braun, F., &
Prouix, M. The neurophysiology of human attention:
A tutorial review. In J. Requin (Ed.), Attention and
36
performance IV, Hillsdale, New Jersey:
Erlbaum Assoc., 1978.
Lawrence
Posner, M.I., Nissen, M.J., & Klein, R.M. Visual dominance: an information processing account of its
origins and significance. Psychological Review, 1976,
~.
157-171.
Proctor, R.W., & Proctor, J.D. Task modality and attention. Journal of Experimental Psychologt: Human
Perception and Performance, 1979, 2{4), 10-624.
Rollins, Jr., H.A., & Hendricks, R. Processing of words
presented simultaneously to eye and ear. Journal of
Experimental Psycholof): Human Perception and
Performance, 1980, 6(1 , 99-109.
Stroh, C.M. Viglance: the problem of sustained attention. New York: Pergamon Press, 1971.
Sutherland, N.S. Stimulus analysing mechanisms. In
Mechanisation of thought processes, (Vol 2), London:
Her Majesty's Stationery Office, 1959. Cited inS.
Kornblum (Ed.), Attention and performance IV, New
York: Academic Press, 1973.
Swets, J.A. Signal detection theory applied to vigilance.
In R.R. Mackie (Ed.), Attention and performance IV,
New York: Academic Press, 1973.
Thomas, E.A. On expectancy and the speed and accuracy of
responses. InS. Kornblum (Ed.), Attention and permance ]Y, New York: Academic Press, 1973. --Treisman, A.M., & Geffen, G. Selective attention and
cerebral dominance in perceiving and responding to
speech messages. Quarterly Journal of Experimental
Psychology, 1968, 20, 139-151.
Treisman, A.M.,-& Riley, J.G. Is selective attention
selective perception or selective response? A further test. Journal of Exnerimental Psycholo~r, 1969,
12_,
27-34.
Treisman, A.M., & Davies, A. Divided attention to ear and
eye. InS. Kornblum (Ed.), Attention and performance
IV, New York: Academic Press, 1973.
Tulving, E., & Lindsay, P.H. Identification of simultaneously presented simple visual and auditory stimuli.
In A.F. Sanders (Ed.), Attention and performance I,
37
Amsterdam:
Humanities Press, 1970.
Tyler, D.M., 1iJaag, W.L., &Holcomb, C.G. rllonitoringperformance across sense modes. An individual differences approach. Human Factors, 1972, 14(6), 539-547.
Webber, C.E., & Adams, J.A. Effects of visual display
mode on six hours of visual monitoring. Human
Factors, 1964, 6, 13-20.
Wilkinson, R. T., Morlock, H. C., & lrJilliams, H. L. Evoked
cortical responses during vigilance. Psychonomic
Science, 1966, 4, 221-222.
Yntema, D.B., & Schulman, G.M. Response selection in
keeping track of several things at once. InS.
Kornblum (Ed.), Attention and Qerformance I,
Amsterdam: ·Humanities Press, 1970.