Ot:M^ty
T55.4
.W2
'?3
Finding Out What Goes On in a
Software Development Organization
I
Dewayne E. Perryi
Nancy A. Staudenmayer^
Lawrence G. Votta,
November
Jr.i
WP
1993
# 99-93
INTERNATIONAL CENTER
FOR RESEARCH ON
THE MANAGEMENT OF TECHNOLOGY
S^E%^
Massachusetts Institute of Technology
<^lngn Qrhnnl nf ^^gnanQmQnt
The International Center for Research
on the Management of Technology
Finding Out What Goes On in a
Software Development Organization
Dewayne E. Perry^
Nancy A. Staudenmayer^
Lawrence G. Votta,
November
WP # 99-93
1993
1.
AT&T
2.
Massachusetts Institute of Technology
Bell Laboratories
Sloan
©
Jr.^
WP#
3626-93
1993 Massachusetts Institute of Technology
Sloan School of Management
Massachusetts Institute of Technology
38 Memorial Drive, E56-390
Cambridge,
02139-4307
MA
M.l.T.
JAN
LIBRARIES
5 1994
RECEIVED
Acknowledgement
We would like to recognize the extremely hard work of our collaborators who made edl these
experiments possible: M.G. Bradac, D.O. Knudson, and D. Loge. Professor Tom Allen at MIT brought us
together, while Peter Weinberger, Eric Sumner and Gerry Ramage provided the financial support for
this work. Professor Marcie Tyre of MIT was particularly helpful, providing extensive input along the
way and pointing us in the direction of relevant organizational theory literature. W.J. Infosino's early
suggestions regarding experimental design issues and A. Caso's editing of the paper are also much
appreciated. Finally,
subjects
and
we would
their colleagues
like to acknowledge the cooperation, work and honesty of the study
and management for supporting our work with their participation.
ABSTRACT
Time and motion
enterprise.
We
studies constitute a proven approach to understanding and improving any engineering
believe software processes are no different in this respect; hou'ever, the fact that
software development yields a collaborative intellectual, as opposed to physical, output
careful
Ccills
for
and creative measurement techniques.
In attempting to answer the question 'where does time go in software development?' we have been
experimenting with two relatively uncommon forms of data collection in the software development
field: time diaries and direct observation. This paper describes the latter in which we drew upon
experimental techniques from the behavioral sciences to observe engineers developing software in a
large organization.
We
have found that both methods of research are feasible and yield useful information about time
The major source of discrepancy is granulcirity: most software developers are not capable of
retrospectively reporting the large number of unplanned interruptions and transitory events that
typically characterize their working day. We were able to quantify the effect of such social processes
utilization.
using the observational data.
1.
Introduction
In 1975, Frederick P. Brooks,
exercise in
complex (human)
experiments
described software construction as inherently a systems effort, "an
Jr.
inter-relationships" [Brooks75].
Yet now, almost twenty years
later,
most
focus on the mechanical aspects of programming, not the social.' This focus continues
still
despite the fact that Brooks and other leading researchers in the field have continued to emphasize the
importance of the human aspects
in
system development [BrooksSO; Brooks87; Curtis86; Ershov72;
Leves92; Shneid86; Weinb71], and despite the
amount of unexplained variance
in
fact that interim process studies
human performance
[Benj92;
have indicated a large
Boehm88; Brooks80; Cusum91; Sack68;
Vos84].
The few
studies that have investigated the
student programmers or
informative
and
questioned.
How
useful,
artificial
their
tasks
in
human
laboratory
aspects of
settings
programming have
[Curtis86].
are
such samples and tasks?
What
on
Although these have been
relevance to large scale software development
representative
relied primarily
kinds
is
being
increasingly
of problems,
unique
to
organizational environments, are being ignored by focusing on these small and artificial domains?^
For example, work
exists in a temporal context.
How time
is
partitioned, scheduled and used
have both
dramatic and subtle influences on organizations and the people in them [McG90; Schr87; Vinton92].
Within the domain of software development, much recent effort and attention have been devoted
need for increased market response time. Yet there has been surprisingly
behavior
at the individual level
ability to act.
little
to the
research on time related
and on the connection between individual actions and an organization's
The empirical evidence on
the reverse relationship
is
similarly unsatisfactory.
Does time
pressure spur people to improved productivity [Andr64; Andr76] or create an overload of competing tasks
[Pelz63]?
1.
2.
It is an interesting historical question as to why this focus has occurred.
Some researchers attribute it to the EE background of the
disapiine [Kraft79]. Others view it has an example of choice of research methods determining the questions to be studied
[BrooksSO],
One
unfortunate result of this narrow focus
amount of myth
that
is that
programmers have generated an enormous
have some comments to make about such accepted truisms as: (1)
software programming is an isolated type of activity.
the few studies that have investigated
needs to be challenged [Weinb71].
developers don't like (and cannot) be observed and (2)
We will
The
literature yields a history
of provocative (and troubling) anecdotes on time utihzation during
software development. In particular, several prominent authors have written that a significant proportion of
project effort
a
work week
is
devoted
typically absorbed
personal days, leaving
Mayer68].
A
non-programming
to
little
activities,
with some estimates indicating as
as
50%
of
by machine downtime, meetings, paperwork, company business, sick and
time for actual programming and debugging efforts [Boehm88; Brooks75;
close examination of the references, however, seems to indicate that
based on one unpublished 1964 dissertation by E.F. Bairdain on
[Mayer68].
much
Details of the sample and
methodology used
how
to
all
of these claims are
software developers spend their time
generate these findings are not readily
available.
The
present study
is
therefore part of an on-going effort to understand what professional software
developers actually do as opposed to what they say they do or are thought to do [Weinb71].
time as a critical yet under-explored aspect of
life in
We
focus on
software organizations.
In an earlier study (a prototype for our current large-scale experiment)
we analyzed
data on one
software developer's daily activities through the use of a retrospective diary [Brad93a]. In our current
experiment,
we
are gathering data on a large sample of developers via daily retrospective self-reports to
informant biases.
for
We found that although the
macroscopic analyses,
largely
due
and
field
often fails to reflect
well calibrated to observations
what an individual
is
doing
when used
at a finer resolution.
This
software developers vary in the degree of granularity they apply to their
collectively impact the
In the next section,
we
briefly
research questions. In Section
The
is
is
self-
during the course of a working day. Although these types of interactions are typically of
minor duration, they
study.
self-reported data
to assess
do not report the unanticipated work requests and unplanned interruptions they both
reports and, in general,
initiate
it
to the fact that
we used
This paper describes a direct observation methodology
gather the data [Brad93b].
data analysis
and develop a variance
is
3,
development process
review our prior work and choice of methodologies before posing specific
we
describe the experimental design and execution of the observational
divided into two pans: in Section 4,
factor; in Section 5,
interactions not reported by
to a significant degree.
we
most developers.
we
first
calibrate observations to self-reports
investigate the principle source of variance: the transitory
We
conclude with recommendations of
how
to use the data
and with proposals for further research.
2.
2.1
Motivation
Methodological Approach
Time
studies have a rich history and have been extremely valuable to our understanding of
organize work, and
how
to build
complex services and products [And64; McGr90; Pars74]. The
time study breaks a task into a set of subtasks
activities
and the elapsed time interval
until
at the individual
completion.
how
traditional
or project level and tracks the sequence of
When combined
with cost information, a model of
the task can then be assembled that enables an organization to reduce cost and production time by
effectively focusing
its
to
technology and resources. Such a model can also
associated with the process such as scheduling bottlenecks or instances
managers
alert
when
to
more
problems
the process
is
being
circumvented.
This paper builds on earlier work in which
we designed
a modified time card and asked software
developers to record their daily activities. Our preliminary studies are described by Bradac et
Brad93b], and a sample of the survey instrument
and occasional unannounced
visits
is
shown
had convinced us
that
in
al.
[Brad93a;
Appendix A. Although periodic interviews
no conscious misrepresentation occurred, we
sought to check the reports of time usage submitted by software developers via direct observation.'' As
noted above, a few prior studies have reported aggregate
on time usage
statistics
but none, to our knowledge, have been based on actual observation.
for such an approach,
however,
Alternative methods
do
is
exist,
work done
and
we
in the early 1960's in
An
in
software development
important and significant precedent
Japanese software factories [Cusum91].
considered using an observer with a video camera instead of an
observer taking notes. Although there are precedents for using video cameras [Guin90a; Guin90b]
it
would be inappropriate
for a
number of
reasons. First, our study population
was not used
we
felt
to being
observed. The subjects were receptive to the notion of participating in an experiment and quickly became
3.
For example, subjects may unintentionally forget significant events when under the pressure of production. Also, we are implicitly
relying on a subject's definition of what was most significant about a day consisting of (often) many different events. Finally, we
were concerned about the consistency of the survey instrument, both across and within subjects, and the adequacy of the data
resolution. H. Russell Bernard, et
al.
gives an excellent
summary of much of the informant accuracy
research [BemSO].
comfortable with the observer, but the introduction of video equipment would have distorted their behavior
(not to mention that of their peers and overall
have
to
Finally,
did
work
progress). Second, over
300 hours of video tape would
be watched and interpreted, significantly increasing the cost and duration of the expenment.'*
we were
interested in obtaining information about
— why they made
certain choices
of video would have precluded
This approach in and of
oiu'
itself
and
how
they decided
why developers used
their
time the
among competing demands on
way
they
their time.
Use
asking questions about the subject's choices.
cannot (necessarily)
capable of accomplishing. Yet, as noted by
Wolf et
al.,
tell
us what developers should do nor what they are
"in order to
improve processes and design new ones,
necessary to obtain concise, accurate and meaningful information about existing processes" [Wolf93].
it is
That
is,
by understanding how and why programmers use
their time the
way
they do,
we
will
be better
positioned to identify tools and methods that enable them to perform tasks in less time.
Research Questions
2.2
The
specific research questions
To what
1.
actually
—
—
3.
sought to address were as follows.
extent are the daily retrospective seif-reports an accurate representation of
what
happened during the developer's day?
what are the sources and extent of bias?
what are the strengths and weaknesses of the two methodologies?
What types
2.
we
of events are not being captured in the self-reports
and how
significant are they?
Observing Software Developers in Action
The
We
first
part of this section describes the experimental setting, selection of study subjects,
and design.
then proceed to describe the actual execution of the experiment. This includes both the
initial
preparation of the study subjects and research procedures as well as the data collection proper.
4
—
we have touched on an important point in experimental design the costs versus the accuracy of the experiment. This is a
when designing a study. Although outside the major theme of this paper, it is important to realize that
our cost for doing this experiment was =700 person hours at S60 per person hour. At a minimum, reviewing 300 hours of tape
would have increased our costs by =43% (assuming we viewed all 300 hours of tape).
Note
thai
common
tradeoff problem
-5-
Experimental Setting
3.1
The
is
subjects for our time studies build software for a real time switching system [Cols92].
a successful product with over 10 million
41 different subsystems.
New
noncommentary source
lines
(NCSL) of C
hardware and software functionality are added
to the
The system
code, divided into
system approximately
every 15 months.
A unit of functionality
project
with
customer will pay for
that a
management. They vary
many complex hardware
in size
from a few
circuits developed.
The development organization responsible
is
called a feature, the fundamental units tracked by
NCSL
with no hardware developed to 50,000
Most software
for product
software developers and 500 hardware developers.
is
built using a real
time operating system.
development consists of approximately 3,000
This software development organization
ISO 9001 standard [ISO9001] and has been assessed
registered to
NCSL
as
is
currently
an SEI level 2 development
organization [Hump89].
3.2 Selection of
Study Subjects
Five software developers were chosen at random from the group participating in the self-reporting
experiments.
No
one
who was asked
refused to participate.
The remaining 10 developers from
the self-
reporting experiment served as one control group, allowing us to assess the impact of observing on self-
reporting.
We
also had nine
months of prior diary reports on these two groups which could be used
comparison with post-observation
experiments were also included
entries.
in the
Two software developers who
for
were not part of the self-reporting
observation experiment as an additional form of control: comparison
with the other subjects will enable us to assess the impact of self-reporting, given observation.
We
applied a purposive sampling scheme, selecting subjects at
dimensions
for
were
we
felt
would be most
(1) the project factors
significant.
The two major
factors
random
we
felt
yet stratifying along those
were most important
to control
of organization, project phase and project type and (2) the personal factors
of age, gender, race, individual personality and years of experience.
Our
was
goal
not to conduct a comparative study but rather to obtain a broad base of observations and to
decrease the likelihood of idiosyncratic findings.^
In
1.
summary, our sample consisted of one treatment and two control groups:
Treatment Group
1
contained subjects
who had
participated in the self-reporting experiments for 9
months. They continued to complete their time diaries and were simultaneously observed.
2.
Control Group
contained subjects
1
months. They continued to
3.
fill
who had
participated in the self-reporting experiments for 9
out their diaries but were not observed.
Control Group 2 contained subjects
who had
not participated in the self-reporting experiments but
were observed.
33 Experimental
We
Design
applied a social experimental design that allowed us to efficiently control
many
factors with as
few
observations as possible and to determine whether the presence of an observer significantly changed the
developer's behavior.
It
combined elements from
(a) a partial factorial design, (b) a
design [Judd91]. Figure
1
three standard behavioral science experimental designs:
repeated measure design, and (c) a replicated interrupted time series
depicts the
2x3
partial factorial design.
observation predictability and observation type. The design
of the six possible
The
is
partial
at
because
we
variables are
collect data in only four
cells.
intent behind the first variable (observation predictability)
observing
The two independent
random we would hinder
impress the researcher. Second,
we
was twofold.
First,
we hoped
that
by
the study subjects ft-om adjusting their schedule so as to favorably
sought to give the study subjects some sense of control over the
experience. Prior research in psychology has confirmed the notion that having control (or the perception
thereof) over one's environment produces physical and psychological well-being [Langer89].
5.
We
aware
sample size is quite small and probably inadequate for statistical validity but, following the logic of
is bener than none." This work falls within the second category of nested results Brooks cites as
necessary and desirable for progress in software development: reports of facts of real user behavior even though observed in
under-controlled, limited sample expenences. What distinguishes this from just "any data" is the methodological ngor we sought to
apply to the study and the relevance of the results to real world software development.
are
(Brooks88],
we
that our
believe "any data
Y
Variable
—(Observation Type)
One Hour
One Hour
Observing
(Y = l)
With Questions
(Y = 2)
Without Questions
(Y = 3)
Scheduled
(X = l)
XlYl
none
none
Unscheduled
(X = 2)
X2Y1
X2Y2
X2Y3
Full
Day
Variable X
(Observation
Predictability)
Figure
1
2x3
Partial Design for
Observing Software Developers
The figure displays the independent variables of the original experiment definition. This
was
X
later simplified
by using only
full
day observations, for reasons described
represents the independent variable of observation predictability:
unscheduled by the subject.
Y
in the text.
scheduled and
represents the observation types: a full day, snapshot with
The
questions, and snapshot only.
design
original
is
partial
because no scheduled
observations involving snapshots, either with or without questions, are performed.
The use of
multiple controls
was
deliberate and deserves further explanation.
We
were particularly
cognizant of the so-called "Hawthorne Effect," the notion that the mere fact of having subjects self-report
and/or be observed might alter their behavior and distort our conclusions [Pars74].
mechanisms
several alternative control
In
by
particular,
observing
our
subjects
measurements), each subject served as his/her
the design further allowed us to
therefore built in
in order to assess the significance of these possible distortions. This
included both standard hold-out samples (Control groups
design.
We
compare
the
own
same
1
and 2) as well as features of the experimental
more than once under each condition (repeated
control.
The
replicated interrupted time series aspect of
subject over time and different subjects at the
same
time.
This enabled us to assess potential threats to validity posed by maturation, history and testing. Finally, as
noted
earlier,
we had
extensive pre-observation self-reported data to compare with that reported once the
observation experiment began.
The
original design
was quickly modified
to only a full
day observation type (collapsed into a 2x1
design), for three reasons: (1) the subjects were in three different locations, creating excessive travel costs
for the observer; (2) the arrival of an observer in the
middle of the day created too many awkward
warm up
-8-
periods and thus unduly interfered with on-going work; and (3)
it
was
day to properly interpret a single event and the subject's response to
Each subject was observed a
of five
total
full
hours of observation over a 12 week period).
subject.*
The remaining
three days
often necessary to observe a
whole
it.
days (9-10 hours per day, on average, for 315 to 350
Two
total
of the five days were chosen and scheduled by the
were assigned by random draw without replacement, and the subjects
were not informed of when they would occur.'
3.4
Experimental Definition
We
went through several important steps
to prepare for observing the software developers.
We
refer to
these as the experimental definition since they involved assembling information and defining procedures for
our observations.
3.4. J
Preparation of and Guidelines for Observation Participants
Because we were aware
that
some people might be uncomfortable about being observed, we
considerable time beforehand explaining the purpose of the study to the subjects.
observer as a student
—
there
to
learn
how
development. The subjects were reminded that there are no right or wrong ways to work
was not
to
judge but
positioned the
spends his/her time when doing software
subject
the
We
spent
(i.e.,
our purpose
understand behavior within a given environment).
to
We were particularly sensitive to issues of anonymity and confidentiality
[Judd91]. All data
under an ID code known only to the researchers. Each subject was also given a
right to halt or discontinue observations at
examine the observer's notes; and the
list
was entered
of his/her rights: the
any time or withdraw from the study altogether; the right
right to ask the researcher not to record something.
None
to
of these
situations occurred.*
6.
Interestingly, subjects often forgot
when they had scheduled such
the morning. This reassures us that subjects
7.
8.
The
logistics behind this
were not
trivial.
sessions and were subsequently surprised to see the researcher
were not too intimidated by the prospect of being observed
For example, vacations had
her schedule to accommodate subjects
who worked
procedure was established for what to do
in the
In faa, the only person
schedule) feature.
who
asked to look
at
flexible hours.
to
be blocked out in advance, and the observer had to adjust
lab sessions were also conducted off -hours, and a
Many
event that a developer did not come into
the notes
was
a
in
woA.
manager who was extremely worried about the
slate
of
his (behind
3.4.2
Data Checklist
To
insure that information about the software developers
collection.
It
was uniform, we created a
checklist for data
consisted of demographic information (age, gender, race, family obligations outside of work);
educational and professional experience; current organization and project status; work habits; job security
level;
and overall job
We
satisfaction.
This data was collected
Lane
and
Lindquist's
one-hour interview.
Myers Briggs
also administered three survey instruments: a
Kaufman,
in a
Polychronic
Attitude
Index
Monochronic/Polychronic Orientation Scale [Blue92]. The PAI attempts
toward performing more than one activity
attitude
which a department or organization
is
at a time.
personality assessment [Keir84];
(PAI);
and
Bluedom's
to capture an individual's general
The Bluedom
scale measures the extent to
polychronic and was given to each subject, his/her immediate
manager, and a peer.
Finally,
we copied each
subject's desk calendar for
two months and noted
all
scheduled meetings,
classes and vacation days.
3.4.3 Rules for the Researcher
The observer endeavored
to function as a
"human camera," recording everything with
as few initial
preconceptions about relative importance as possible. She was also required to frequently read a half page
list
of reminders designed to keep the observation procedure consistent.
used for
We
all
Because a single observer was
seven subjects, issues of inter-observer variability were avoided [Judd9I].
used continuous real time recording for non-verbal behavior and interpersonal interactions. During
those interims
developer
when
a developer
at regular intervals
was working
basis for
files.
we used
a time sampled approach: asking the
"what are you doing now?" Daily observations were recorded
notebooks, unique to each subject.
standard computer
at the terminal,
Each evening,
This allowed us to readily
the
fill
in
in small spiral
raw notebook observations were converted
observations while
two kinds of summary sheets described below. As data came
still
in, it
fresh
to
and served as the
was added
to a loose-leaf
notebook, with separate sections for each subject. This helped us stay organized over time and facilitated
communication among the researchers.
-10-
Such an inductive method of
analysis,
which involved the
joint collection, coding and analysis of data,
has been cited as one of the best ways to discover theory from data [Glaser67].
3.5 Miscellaneous
Remarks About Observing
Almost without exception, the
nothing to see" (from subjects) or
from the
As
truth.
first
response to
"How
study was
"You
described below, a software developer's day
was
(e.g.,
filled
want
to observe
me? There's
with events and activities that
The observer accompanied
group and department meetings, affirmative action presentations,
broad spectrum of events
just
boring" (from fellow researchers). In fact, nothing could be further
varied considerably across individuals and time.
to
this
the subjects everywhere (e.g.,
tutorials, lab sessions)
and witnessed a
promotions, mental and physical exhaustion, celebrations). In addition to
observing, there were plenty of opportunities to talk with the developers and their colleagues about
the organization,
4.
and we found them
to
life in
be remarkably willing to do so.
Calibration Analysis
In any experiment, the researcher needs to understand the sources of variance in the data. This allows
model of the experiment and answer the question, "What
the researcher to build a probability
probability of being
interpret
data
is.
on how long
The sources and
4.1 Description of
it
takes developers to perform given tasks, and
properties of variance are also important in their
—
It is
Wolf and Rosenblum
direct
own
to
know how
we need
reliable
right [BrooksSO].
left
(see
example
in
of time
we compared two
one comparison
sets
sheet, with a
Appendix B) and our summarized observations on
important to recognize that the observer recorded data contains an impressive amount of
micro-level detail, often
9.
own usage
the self-reported data and the observer's notes. Figure 2 displays
software developer's report on the
the right.
we need
efficient,
Data
In order to calibrate software developers' assessment of their
of data
the
enables other researchers to reproduce the experiments and to
it
agreements or disagreements. To make the software development process more
to gather data
this
wrong?" Furthermore,
is
down
to three
minute
intervals.' In order to
is very imponant and have designed and used an alternative
captunng and correlating short duranon events with the actions of people fWolf93].
note that an appropriate level of granularity
observaaon method
that is
opQmized
for
form an effective comparison, we
-11-
therefore
summarized
that detail into
major blocks of
activities. '°
Note
that in
forming these interim
summaries, we have ignored many transitory, short duration events which the developers had to
initiate
and
respond to during the course of a working day. Those types of interactions are analyzed separately
Section
5.
DIARY
in
-12-
to the right are instances
where a developer actually worked
the majority of the observations are clustered around
obvious exception
is
subject
2A who
500
to
less than he/she reported.
550 minutes or
twice worked more than
1
hours.
1
He
8.3 to 9.2
As
seen in figure
working hours."
3,
An
an acknowledged local expert
is
often called upon to help solve critical issues in the lab. Note that most subjects appear to be relatively
consistent in the accuracy of their reporting. Subject 2B, for example, tends to over-represent total working
time by about one hour whereas subjects IB and IC are remarkably accurate.
0)
3
C
o
o
o
E
F
r(D
o
O
o
<o
a.
a>
CC
^.
(S
Oo2
•Q
in
700
600
500
Subject Reported Time (minutes)
Figure 3 Comparison of Reports of Total Working Time
This plot compares each software developer's self-reported
minutes) with that actually observed. Each developer
is
total
identified
work (in
unique ID (e.g.,
time
by a
at
2A), with 5 data points (observation days) per individual. The 45 degree line indicates
perfect agreement between observer and subject.
when
their reported
time
is
two occasions subject 2A under reported
represented working time was 2.8%.
1 1
A
less (greater) than that
his
subject
is
said to under (over) report
observed (above (below) the
work
time.
line).
The average amount of
On
over-
These calculations merely compare begin/end times; we have not subtracted out breaks or lunches from the totals. In general, there
was a strong correlation m the reliability of all such measures; i.e.. a developer who accurately reported total time and activity also
tended to accurately record the number and duration of his breaks.
-13-
The average amount of over-represented working time
reported
saw)
work time
—
the positive difference between the subject's
given day and that recorded by the observer (as a function of what the observer
in a
— was 2.8%.
The second component of
the error
model
reconcile one of the subject's self-reported
what extent would
the the
list
the fidelity of the self-reports. If
is
we were
to try
of activities on a given day with that actually observed, to
two viewpoints agree?
We
obtained such a measure by dividing the number of
minutes that a subject and the observer agreed about what the developer was actually doing by the
number of minutes worked
that
day
and
(as recorded
total
by the observer).
Figure 4 displays that fraction plotted as a function of the date the subject was observed. The vertical
line
segments represent the confidence bounds around each
and subjects ranged from 0.95 to 0.58 for subjects IB and
the variation between subjects
We
found
that the
is
2B
ratio. '^
respectively.
entries the subject
made
clusters further indicate that
per day, the greater the agreement between the
observer and subject. Figure 5 compares the number of entries
Of course, some
The
the observer
greater than the variation within any subject's set of observations.
more diary
daily activity blocks extracted
The agreement between
from the observation notes
made by
for each
a given subject with the number of
day of observation.
days are more eventful than others, but, as indicated in Figure
on average, be segmented into about 12 different
activity blocks
5,
a developer's day can,
(from the observer's perspective). The
developers, however, varied in the level of detail they applied to their diaries. Subject 2B, for example,
always noted one major activity whereas IB and IC's level of
detail
observer. Interestingly, the time questionnaires also indicated that
approximately equaled that of the
IB and IC were
the
most monochronic
of the study subjects.
When we examined
variation
12
Two
was
the large
the content of what developers were or were not reporting, the major source of
number of unexpected events
that occurred during the course of a developer's day.
independent researchers compared each of the activity blocks noted by the observer with the subject's diary entries that day
and assigned each such block to one of three categories: agree, disagree, or maybe. The length of the line segment is the amount of
time in the day where we could not definitively decide whether there was agreement or disagreement between the two reports.
-14-
a
o
a
E
o
a
<
*
o
d
Date
Figure 4 Rate of Agreement Between Observer and Subjects
The
fidelity rate, calculated as the
number of minutes
that the subject
about what the subject was actually doing divided by the
that day, is plotted
by the date of observation. The
bounds around each estimate. The
subject variation
(See, for example, the
is
fidelity varies
total
and observer agree
number of minutes worked
vertical lines represent the confidence
between 0.95 and 0.58, and the between
clearly greater than the within subject variation.
two afternoon
entries of the right
hand side of Figure
2).
Although most developers
did not record such interruptions in their retrospective reports, they were a ubiquitous part of the
development process from the observers' perspective. In the next section, we use the observation data
to
quantify this qualitative impression that developers are frequently interrupted.
5.
Communication Analysis
Gerald Weinberg once posed the provocative question, "Does
developer runs into during the day?" [Weinb71]
He argued
it
matter
how many
people a software
that although the task of writing
code
is
usually
assigned to an individual, the end product will inevitably reflect the input of others. '^ Others have noted
13.
Note the distinction between what developers say they prefer and how they actually behave. According to Weinberg, most
programmers would probably claim to prefer to work alone in an undisturbed envirotunent, yet he estimated that they aaually
spend about 2/3 of their time working with others. Similarly, we observed a tremendous amount of voluntary collaboration among
the developers in our study despite frequent protests of "too many mterrupuons." Although we cannot distinguish whether the
behavior is driven by preferences or the requirements of a complex task, it does highlight the false portrayal of programnung as an
isolated type of activity.
15-
J2
O
o
o
CM
m
$•
u
in
CO
n
O
o
_
m
-
<D
E
Z
10
Number
15
of Self-Reported Activity Blocks
Figure 5 Comparison of Number of Entries
This plot compares the number of major activity blocks observed versus the number of
diary entries recorded by each developer.
numbers was
to
1
We
found that the closer the
ratio of the
two
(represented by the dashed line), the higher the fidelity of the self-
reporting.
that
programs have a dual nature: they can be executed
entities
[Solow84].
Both points
critical factor in organizational
Most communication
reflect the fact,
for effect
and they can be read as communicative
acknowledged by
theorists, that information flow is a
success [Allen77].
studies,
however, address a very narrow range of interactions that occur
in the
course of a collaborative work effort. For example, they are typically restricted to only one media channel
or focus on exchanges that are planned-in-advance and of relatively long duration [Kraut90]. Moreover, the
empirical data often consists of asking subjects
who
they talk to the most and thus risks confounding
frequency with duration or impact [Baum90; BemSO]. The present study offers a unique opportunity to
address such deficiences in that
media channels.
it
tracks
all
communication
activity, at the individual level, across multiple
-16-
5.1 Description of
Data
Figure 6 presents a sample of the communication
summary
the daily observations of their interactions across four
it
was
we
prepared on each subject, based on
major channels (audix, electronic mail, phone and
in-person visits).'* Drawing on the methodology of [Wolf93]
terms of whether
sheet
we have
sent or received by the study subject.
SUBJECT ID
broken each interaction
down
in
17-
o
0.
w
o
sc
o
O
o
3
g-
'E
3
-18what
widely believed to be people's cognitive limitation [Miller56].
is
Subject
We
2D
stands out in contrast to the rest of the sample with a median
attribute this primarily to office layout (he shared space with
machine drew a
lot
of
traffic).'*
number of
daily contacts.
1 1
two other people and
their local coffee
This subject's Meyers-Briggs personality profile also indicated a strong
preference for social interaction.
The
outliers in the boxplot
diagram are particularly
represents a day in which develof)er
The other
field request.
number of unique
outliers also
started to
correspond
to
The
highest point (17 unique contacts)
work on a code modification motivated by
to modifications
interfaces approximately doubled
were requests for authorization
calls to a help
2C
interesting.
of existing code, and
from the baseline of
7.
in
The majority of
change code owned by another developer. Just
a customer
each case, the
these contacts
slightly less frequent
desk for passwords or information about a particular release of the software;
were
calls to the lab
requesting available time slots for testing; and exchanges with peers about process procedures in general.
Note
that these contacts
were not technically related per
se.
That
is,
the solution
was usually not the
motivating issue driving this behavior. Rather, the developers needed help implementing the solution.
We next examined the number of messages being sent and received each day
channels (Figure
8).
Note
that the distributions of sent
approximately normal, reassuring us that the sample
presence of reciprocal interactions."
total
As noted
is
and received
visits
across the different media
and phone messages are both
not significantly skewed and also suggesting the
in the far right set
of boxes, a developer typically received a
of 16 messages and sent a total of 6 messages diuing a working day. Ignoring email for the moment,
the most ubiquitous form of contact in this
work environment was in-person
visits.
They were used
approximately two to three times as often as the other channels.
One
16.
of the most surprising results concerned the usage of electronic mail.
corporations are
in general, is the impact of having an office mate? Three of our subjects shared an office with a single colleague in the same
work group, three subjects had private offices and the remaining individual shared an office with two peers. Prior studies have cited
an 8-11% productivity gain with private offices [Boehm83], and although we cannot assess such a claim here, the presence of a
What,
close peer
is
definitely an important driver of interruptions.
gossip, and there
17.
Many
We
do not
was
typically a large
explicitly track
communication threads
smgle problem) here [Wolf93].
Office mates often debriefed one another about meetings or local
amount of "what ir or "do you know how" type of questions exchanged back and
(a
group of related communication events
all
forth.
devoted to the discussion of a
•19-
8164><
n
49-
O
o
36
Q.
"T"
<n
a>
25-
re
CO
(A
0)
16
9
E
3
^^-
I
r-f-.
4
I
!
uJ.
I
I
1
^_L_;
r
audix
s
r
.,s
r
u
phone
Media Type
email
c_L_,
.
.
s
r
visit
,,
s
all
Figure 8 Number of Messages Being Sent and Received Across 4 Media Channels
The number of messages being sent and received, broken down by media channel and
whether they were received or
initiated
by the study subject.
root transformation in order to stabilize the variance.
We
have applied a square
Each box contains data on
all
7
study subjects across 5 days of observation per individual.
starting to
implement
this
new form of communication, and we
intensive nature of this organization, to see a large
received
many such messages
(a
amount of email
fully
expected, given the computer
traffic.
median of 9 per day), they sent very few
(a
Although our study subjects
more, the content of these email messages was rarely technical. Most of the
organizational
news (announcements of upcoming
talks, recent
per day). What's
median of
traffic
was devoted
to
product sales, or celebratory lunches;
congratulatory messages on a "job well done") or process related information (mosdy announcements of
process changes!)
We attribute
draft a
I
this
phenomenon
to several factors. First,
it is
difficult
and time consuming to coherently
complex technical question or response. As noted by one developer "Email
type out a coherent description of the problem,
I
is
too slow; by the time
could have called or walked over and gotten the answer."
Secondly, the ambiguity of software technology necessitate a type of iterative problem-solving that
suited to the email venue [Sack68; Tsai87].
Our
subjects
may
also have been
somewhat
is ill-
reluctant to release
-20a written recommendation or opinion without having control over
maturity of this email system
was undoubtedly
a factor. That
its final
is,
distribution. Rnally, the relative
electronic mail in this development
organization appears to have evolved into something of a broadcast medium.
It is
the most efficient
way
for
the organization to distribute information to the technical population, but the very fact that such messages
have flooded the system makes developers reluctant
to use
it
for pressing technical issues.
(D
c»
(0
0)
m
o
o
Q.
(0
o
r
email
audix
phone
Media Type
.
s
.
visit
all
Figure 9 Duration Per Contact By Media Channel
The duration per contact broken down by media channel and whether
received or initiated by the study subject.
the
message was
We have applied a square root transformation in
order to stabilize the variance. Each box contains data on
all
7 study subjects across 5
days of observation per individual.
Figure 9 plots the duration of messages in each media channel. Looking across
communication, approximately
68%
of the interactions are of less than 5 minutes
with research done in the early 1980' s
at
Xerox Pare [Abel90].
It
The
duration of a sent versus received visit (approximately 6 versus 3 minutes)
the
undoubtedly
fact
reflects
that
sent
email
messages are of
relatively
in duration.
forms of
This agrees
also confirms anecdotal evidence
supplied by independent smdies of this population [Baum90; Kelley93].
Similarly,
all
is
difference in the median
attributed to travel time.
longer duration
than those received
compositional factors. Not surprisingly, audix messages are very brief
(1
minute), but
-21-
phone messages are also unexpectedly short (2-3 minutes).
m
all
is
a particularly non-intuitive result in that these are
the
media channels;
visits
Finally, note the existence of significant outliers
of close to one hour and phone calls of 30 minutes are not uncommon. This
all
unplanned and unanticipated forms of interaction.
-22-
6.
Conclusions
The primary motivation behind
this study
capture process information? In so doing,
was
to
answer the question: are time
we experimented
uncommon forms
with two relatively
collection in the software development field: time diaries and direct observation.
methods of research are feasible and
acknowledge, however,
engagement with
that neither
the task. This
is
useful,
method can
diaries a reliable
We
way
to
of data
concluded that both
depending on the questions one wishes to address.
We
satisfactorily assess the subject's level of concentration or
an important component which
is
beyond the inunediate scope of
this
study.
In addition to exploring
new methodology, however, we
also sought to investigate under-developed
arenas in software research: the social structure, environment and culture of a real organization of software
developers.
It
is
our belief that
all
three elements (organization, process and technology) need to be
addressed in order to obtain a complete picture of the development process.
Indeed,
we found
that
elements of the organization were equally
not more) important than
(if
technology. In particular, the data on the number of inter-personal contacts a software developer needs to
make during
a typical working day strongly suggests that technical problems are not the real issue in this
organization. Rather, these software developers need to apply just as
who
to contact within their organization in order to get their
much
effort
work done. Most
and attention to determine
importantly,
quantify what had previously been predominately qualitative impressions about
life
we were
able to
as a software developer
in this firm.
7.
Future Research
As noted by
"deceptively
statistics,
18.
as
the organizational theorists
March and Olsen,
mundane" [March76]. The prosaicness
we have done
here, but also
lies in
the empirical focus on time allocation
is
the necessity of developing not only aggregate
some explanation of
the underlying process.
In subsequent
Although we did not observe a tremendous amount of socializing, exchanges often combined technical and personal
communication.
-23research
we
plan to identify
some of
the structural determinants of the allocation of time within this
organization.
An
additional
imporunt area
for further study concerns the individual
benefits, of these patterns of time usage
level of concentration, yet they
may
and organizational
costs,
and
and communication. Interruptions obviously affect an individual's
also serve important educational
and motivational roles within the
organization.
As technology
researchers.
detail,
progresses,
new forms of
Some such advances
instrumentation are rapidly becoming available to field
are enabling researchers to capture information at a very fine level of
while others are easing the manual burden associated with empirical studies. Yet, surprisingly,
see very
little
traditional
experimentation with these
paper surveys.
We
new
capabilities;
most
developers a device similar to a Sharp
interactive type of diary).
Or one might prompt
would probably be most
efficient).
we have
considered giving
to record their daily activities
on
it
(a
subjects to record their immediate activity by
(A combination of
Not only
knowledge about software development, but they also
research questions.
For example,
Wizard™ and asking them
randomly paging them throughout the day.
personality,
on interviews or
believe other approaches, of varying levels of technical sophistication, are
called for, particularly in the case of software development.
more
field researchers still rely
we
the
above two methods, targeted by
will such experiments
improve our existing
offer the prospect of access to previously unexplored
-24Attachments;
Appendix
A
Appendix
B
References:
[Abel90]
M.J. Abel.
Teamwork,
"Experiences
J.
an
in
Exploratory
Distributed
Organization,"
Intellectual
in
Galegher, R. E. Kraut and C. Egido (eds.), Erlbaum Publishing Company, NJ,
1990.
Managing
Flow of Technology.
MIT Press,
MA,
[Allen77]
T.J. Allen.
[Andr64]
P.M. Andrews. "Scientific Performance as Related to Tune Spent on Technical Work,
Teaching and Administration," Administrative Science Quarterly, Vol 9, September 1964,
pp. 182
[Andr76]
-
Cambridge,
1977.
193.
F.M Andrews. "Time
and Frank Andrews
[Baum90]
the
Donald Pelz
Pressure," in Scientists in Organizations, second edition,
(eds.).
367
-
J.L.
Baumann. "Results of
The
Institute for Social Research,
Ann
Arbor, MI, 1976, pp.
382.
Survey,"
AT&T
the
5ESS Switch Software Development Information Sources
Memorandum 55412- 900124-OlTM. January
Bell Laboratories Technical
24, 1990.
[Benj92]
R.I
Benjamin and
Review, Vol 33,
[BemSO]
J.
1992, pp. 7
H.R. Bernard, P.D. Killworth and L.
Social Networks, Vol
[Blue92]
Blunt. "Critical IT Issues:
No 4, Summer
2,
No 3,
pp. 191
-
Seller.
-
The Next Ten
Years," Sloan
Management
19.
"Informant Accuracy in Social Network Data,"
218.
"How Many Things Do You Like To Do At
Monochronic and Polychronic Time," Academy of Management
A.C. Bluedom, C.F. Kaufinan and P.M. Lane.
Once? An Introduction
Executive, Vol
6,
No 4,
to
1992, pp. 17
-
26.
[BoehmSS] B.W. Boehm and P.N. Papaccio, "Understanding and Controlling Software Costs," IEEE
Transactions on Software Engineering, Vol 14, No 10, October 1988, pp. 1462 - 1477.
[Brad93a]
M.G. Bradac, D.E. Perry and L.G.
Votta, "Prototyping a Process Monitoring Experiment,"
Proceedings of the 15th International Conference on Software Engineering, Baltimore,
MD,
1993.
[Brad93b]
M.G. Bradac, D.E. Perry and L.G.
[Brooks75] F.P. Brooks,
Park,
CA,
Jr.
Votta, Draft, September 1993.
The Mythical Man-Month. Addison-Wesley Publishing Company, Menlo
1975.
[Brooks87] F.P. Brooks,
Jr.
"No
Silver Bullet: Essence and Accidents in Software Engineering,"
Computer, April 1987, pp.10
[BrooksSS] F.P. Brooks,
Jr.
-
IEEE
19.
"Plenary Address: Grasping Reality Through Illusion," Proceedings of
CHISS., May 1988. March 1988.
[Brooks80] R.E. Brooks. "Studying Programmer Behavior Experimentally: The Problems of Proper
Methodology," Communications of the
ACM.
Vol 23.
No 4,
April 1980, pp. 207
-
213.
[Chamb83] J.M. Chambers, W.S. Cleveland, B. Kleiner and P.A. Tukey. Graphical Methods for Data
Analysis. Duxbury Press, Boston, MA. 1983.
.
-25[Cols92]
J.S.
Colson,
AT&T
E.M.
Jr.,
Management
"Total Quality
Prell,
Large Software Project,"
for a
TechnicalJourruil, May/June 1992.
W.
Curtis, "By the Way, Did Anyone Study Any Real Programmers?" Empirical Studies of
Programmers, E. Soloway and S. Iyengar (eds.) Ablex Publishing Corp., 1986, pp. 256 -
[Curtis86]
262.
[Cusum9 1
]
M. A. Cusumano. Japan 's Software
Human
[Ershov72] A. P. Ershov. "Aesthetics and the
ACM, Vol
[Glaser67]
No
15,
7,
Factories.
July 1972, pp. 501
-
Oxford University
Press,
NY, 1991
Factor in Programming," Communications of the
505.
B.C. Glaser and A.L. Strauss. The Discovery of Grounded Theory. Aldine Publishing
Company, Chicago, IL, 1967.
[Gum90.a] R. Guindon. "Designing the Design Process: Exploiting Opportunistic Thoughts," HumanComputer Interaction, Vol 5, 1990, pp. 305 - 344.
[Guin90.b]
"Knowledge
R. Guindon.
Exploited
by
Experts
Software
During
InterruUionalJoumal Man-Machine Studies. Vol 33, 1990, pp. 279
[Hump89]
W.S. Humphery,
Reading,
[ISO9001]
MA,
Software
the
System
Addison- Wesley
Process,
Design,"
304.
Publishing
Co..
1989.
Quality Systems
Installation,
Managing
-
and
—
Model for Quality Assurance
Servicing,
Design/Development, Production,
in
International Organization for Standardization, International
Standard ISO 9001: 1987(E).
[Judd91]
CM. Judd,
E.R. Smith and L.H. Kidder. Research Methods in Social Relations, 6th edition.
Holt, Rinehard
[Keir84]
and Winston,
Inc.,
Chicago, IL, 1991.
D. Keirsey and M. Bates. Please Understand Me: Character and Temperment Types,
Promethueus Nemesis Book Company, CA, 1984.
[Kelley93]
R. Kelley and
J.
Caplan.
"How
Bell Labs Creates Star Performers,"
Review, July-August 1993, pp. 128
[Kraft79]
P. Kraft,
Harvard Business
139.
"The Industrialization of Computer Programming: From Programming
Production," Case Studies in the
[Kraut90]
-
R.E. Kraut and
J.
Labor Process, A. Zimbalist
(ed.),
1979, pp.
1 -
to
Software
17.
Galegher. "Patterns of Contact and Communication in Scientific Research
Collaborations," in Intellectual Teamwork, pp. 149
-
171.
[Langer89
E.Langer. Mindfulness. Addison- Wesley, Inc., N.Y. 1989.
[Leves92]
N.G. Leveson. "High Pressure Steam Engines and Computer Software,"
ACM, May
15,
1992.
March and
[March76]
J.G.
[Mayer68]
D.B. Mayer and A.W. Stalnaker. "Selection and Evaluation of Computer Personnel-the
J.P. Olsen.
Ambiguity and Choice
in Organizations.
Research History of SIG/CPR," Proceedings 1968
ACM
Bergen, 1976.
National Conference, pp. 657
-
670.
[McGr90]
J.E.
[Miller56]
G.A. Miller. "The Magical Number
McGrath. "Time Matters
in
Groups," Intellectual Teamwork.
7,
Plus or
Minus
2:
Processing Information," Psychological Review, Vol 63,
[Pars74]
H.M. Parson, "What Happened
at
Some
No
2,
Limits on Our Capacity for
March
1956, pp. 81
-
97.
Hawthorne?" Science, Vol 183, March 1974. pp. 922
-
932.
[Pelz63]
D.C. Pelz. "Dither and Time
139
[Sack68]
-
in the
Motivation of Scientists," The Chemist., April 1963, pp.
149.
H. Sackman, W.J. Erikson and E.E. Grant. "Exploratory Experimental Studies Comparing
On-Line and Off-Lme Programmmg Performance," Communications of the
ACM,
Vol 11,
26-
No
[Schr87]
I,
January 1968, pp.3
J.B. Schriber
-
11.
and B.A. Gutek. "Some Time Dimensions of Work: Measurement of an
Underlying Aspect of Organizational Culture," Journal of Applied Psychology, Vol 72,
4, 1987, pp.
642
[Shneid86] B. Shneiderman,
"Empirical
E.
Soloway and
K. Ehrlich.
of Programmers: The Territory,
Soloway and Iyengar, 1986, pp. 1-12.
Studies
Destinations," Keynote Address in
[Solow84]
"Empirical
Studis
Transactions on Software Engineering, Vol 10,
[Tsai87]
No
of Programming
5,
1984, pp.595
-
Paths
and
IEEE
Knowledge,"
609.
L.S. Tsai. "Overt and Covert Problem Solving: Reversing the Direction of a Swinmiing
Fish," Perceptual
[Vos84]
No
650.
-
J.
Vosburgh,
"Productivity
and Motor Skills, Vol
B.Curtis,
Factors
65, 1987, pp.
R. Wolverton,
and
B
Programming
Albert,
740
-
742.
H. Malec,
Environments,"
S.
Hoben
Proceedings
International Conference on Software Engineering, Washington D.C.
and
of
Y.Liu,
the
7th
IEEE Computer
Society, 1984, pp.143- 152.
[Weinb71]
G.M.Weinberg.
NY,
[Wolf93]
77ic
Psychology of Computer Programming. Van Nostrand Reinhold Co,
1971.
A.L. Wolf and D.S. Rosenblum,
"A Study
in
Software Process Data Capture and Analysis,"
Proceedings of the 2nd International Conference on the Software Process (ICSP2), Berlin,
Germany, February 25 - 26, 1993.
27Appendix A: Software Developer Self-Report Time Form
Diagnostic Development Process Measurement Record
Tnckcd
Projecl:
Devdopcn
ABCDE
A_ B. Smith
Date: Monday. August 9. 1993
Uk the following
flow chin to detennine the number/letter or
letter
combination
that best
descnbes your
acuvity on and off this project:
(Df\ttapm
>tl»at«/LavMClaat«
p
1
2
3
T
•
II
-
A
-
Use
-
higher pnorlcy/ (r«ikaal0i»«i
your cholc« is kaaigBBMBC
blocked watting on r**oure*
Traiaiog
M
R«vt*w/LaBp«etloD
O
Assiac«ac*/eeuiia«iiao
-
plui/ftrchlt»ctur*
ra^tl r iMi n
high Ivl d^stan
lav Ivl dMloB
writ* t«st plan
-
r •
K»*ciae
Other
(he number/letter or teller combinatioo
h
-
1
w
-
T
-
R
A
-
-
f
0100
-
d
t
-
u
-
cod*/in*p*ct
d«ltv*rabl« caat
tm^tur* c*«t
us«r doc\«M)t«tloo
•
o^pQl.
-
Working prockaa
H*«Clng/pro] aucua
OUiar ia«« Notal
TrsiaUtf
H
R*vl««/t.o«p*ecion
H
AaaiatAacs/eOTuuAliag o
detennmed above
0000
c
to
&U
0200
-
in the following tune chart:
0300
0400
0500
0600
28Appendix B: Completed Software Developer
Self- Report
Time Form
Diagnostic D«v*lopni«nt Process itteasurement Record
TraektdPmJMti ARIIW
A
Drvdupsr:
Ikxc
Ux
R, Silith
Momlt);, Atigist!). ><»^
Ibr
ruBumif
lUiw ftad ki <lctcfninr rlu ironlicrAcn^ or
Wo frnMourion
lioc bL-d (l<saihi.i
>T)iff
oniriiy na atd oft' flifo pci|ixx
:)
.K
_^iy^
ftclA::c co^ss. rcr
>
i-
'.M "Afci":
"r
_
Y
I-....
•
-"it
A
..
a
Ki
C I'.'.oi/lar: ictloi
^
T,V dicimnhcfncnoc Of k«araaiUiBilJii<Jrttniun«d.liiMV wfniin Ihcrollowim drasdnit
(1700
woo
uaii]
0100
ccoo
0300
(>«xi
fiMci
cyimi
ixo
looo
i;o(i
laoii
itiik
nwon
(7«ii
1000
iKio
LMO
ISftO
MOn
2I0II
22HI
230ri
I
(UxvmaUE
licfa
AW^CJ
-
^1i.>7d
4
qimlkan^axuiaeM) hr.
MuikUujut:
LartyVctm:
N«<r
(?<icfMi&J MouiiiMi 4 Ut»,UM<
w
fJ
V>-/>iha''.
r.-osi*/*)..) ysv. UiJpt>Ja.tx.i>Jac
m>Ky7).MiMZ romnJilVDUa
i^i
m^
Tiie
International Center for Research on the Management of Technology
Sloan School of Management
Massachusetts Institute of Technology
Working Paper and Reprint
List
Number
Date
1-89
11/89
2-90
Author(s)
Title
Netgraphs: A Grapliic Representation of Adjacency Tool for Matrices as a
Network Analysis
George
Allen
and the Success of High Technology Companies
Roberts
Strategic Transformation
8/89
3-90
1/90
4-90
Managing
CAD Systems in Mechaniccil Design Engineering
Robertson
Allen
(Rev. 3/91)
The Personality and Motivations
Roberts
of Technological Entrepreneurs
1/90
5-90
Current Status and Future of Structural Panels in the
Wood Products Industry
4/90
6-90R
8/92
Do Nominated Boundary Spanners Become
7-90
The Treble Ladder Revisited:
as They Grow Older?
7/90
8-90
Effective Technological Gatekeepers?
8/90
Allen
Nochur
Why Do Engineers
Lose Interest in the Dual Ladder
Allen
Katz
McCormack
Technological Discontinuities: The Emergence of Fiber Optics
Utterback
8/90
9-90
Montrey
Utterback
Work Environment,
Professionals:
Organizationeil Relationships and
Advancement
of Technical
A Ten Yecir Longitudinal Study in One Organization
Basa
Allen
Katz
10-90
Allen
People and Technology Transfer
8/90
11-90
Exploring the Dynamics of Dual Ladders:
A Longitudinal Study
8/90
Katz
Tushman
Allen
12-90
8/90
13-90
8/90
14-90
Managing the Introduction of
in a Multi-Plant Network
New Process Technology:
International Differences
Task Characteristics and Organizational Problem Solving
Process
in Technological
Tyre
Tyre
Change
The Impact
of "Sticky Data"
on Innovation and Problem-Solving
von Hippel
8/90
15-90
5/90
Underinvestment and Incompetence as Responses to Radical Innovation: Evidence
from the Photolithographic Alignment Equipment Industry
Henderson
Number
Date
Author(s)
Title
Communication Among Mariceting, Engineering and Manufacturing
Comparison Between Two New Product Teams
16-90R
7/90
Patterns of
17-90R
8/92
Age, Education and the Techiucal Ladder
18-90
A Model of Cooperative R&D Among Competitors
A
—
Allen
Katz
Sinha
Cusumano
1/90
19-90
4/90
20-90
Griffin
Hauser
Strategy, Structiu-e,
the
and Performance
in Product
Development: Observations from
Auto Industry
A Model-Based Method for Organizing Tasks in Product Development
Cusumano
Nobeoka
Eppinger
Whitney
Smith
6/90
(Rev. 5/93)
Gebala
21-90
The Emergence
of a
New Supercomputer Architecture
Afuah
7/90
Utterback
22-90
Superceded
23-90
Software Complexity and Software Maintenance Costs
t>y
39-91
Banker
Datar
Kemerer
8/90
(Rev. 1/92)
Zweig
24-90
Rotemberg
Saloner
Leadership Style and Incentives
9/90
25-90
Cusumano
Factory Concepts and Practices in Software Development
11/90
26-90
Going
Roberts
Public: Sell the Sizzle or the Steak
10/90
27-90R
12/90
Evolving Toward Product and Market -Orientation: The Early Years of
Technology-Based Firms
28-90
The Technological Base
of the
New
Roberts
Roberts
Enterprise
11/90
29-90R
3/93
Innovation, Competition, and Industry Structure
30-91
Product Strategy and Corporate Success
Utterback
Suirez
Roberts
Meyer
1/91
31-91
Cognitive Complexity and
CAD Systems: Beyond the Drafting Board
Metaphor
1/91
32-91
Ulrich
Filerman
CAD System Use and Engineering Performance in Mechaniccil
Design
1/91
33-91
6/91
Investigating the Effectiveness of Technology-Based Alliances: Patterns and
Consequences of Inter-Firm Cooperation
34-91
No Paper Issued
35-91
Impacts of Supervisory Promotion and Social Location on Subordinate Promotion in an
RD&E Setting: An Investigation of Dual Ladders
2/91
(Rev. 11/91)
Robertson
Robertson
Allen
George
Katz
Tushman
Allen
Number
Date
Author(s)
Title
36-91R
8/92
Demography and Design: Predictors of New Product Team Performance
37-91
The Changing Role
of
Teams
in Organizations: Strategies for Survival
Ancona
Caldwell
Ancona
2/91
38-91
Schrader
Between Firms
Ir\formal Alliances: Information Trading
3/91
39-91
3/91
40-91
3/91
Supplier Relations and Management:
A Survey of Japanese,
Japanese-Transplant,
and U.S. Auto Plants
Cusumano
Tcikeishi
Maneuvering and Mciss-Market Dynamics: The Triumph
Over Beta
Strategic
of
VHS
Cusumano
Mylonadis
Roser\bloom
41-91
The Software Factory:
An
Entry for the Encyclopedia of Software Engineering
Cusumano
3/91
42-91
4/91
43-91
Dominant Designs and the Survived
Suarez
Utterback
of Firms
(Rev. 7/92)
An Environment for Entrepreneurs
Roberts
Technology Trar^sfer from Corporate Research to Operations: Effects
of Perceptions on Technology Adoption
Nochur
6/91
44-91
7/91
45-91
When Speeding
Concepts to Market Can Be a Mistake
Allen
Utterback
Meyer
3/91
Tuff
Richardson
46-91
6/91
47-91
8/91
48-91
10/91
49-91
Paradigm Shift: From Mass Production
to
Mass Customization
Pine
(Rev. 10/93)
Computer Aided Engineering and Project Performance: Managing
a Double-Edged Sword
The Marketing and
Murotake
Allen
R&D Interface
Griffin
Hauser
(Rev. 2/92)
Systematic' Versus 'Accidental' Reuse in Japanese Software Factories
Cusumano
10/91
50-91
Flexibility
and Performance:
A
Literature Critique
and
Strategic
Framework
Suarez
Cusumano
11/91
Fine
51-91
11/91
52-91
12/91
Shifting Economies:
From
Craft Production to Flexible Systems
An
Analysis of Entry and Persistence
among Scientists
54-91R
Youth and Scientific Innovation: The Role of Yoimg
Development of a New Field
12/91
an Emerging Field
Scientists in
Rappa
Debackere
Institutional Variations in
55-911?
in
(Rev. 9/92)
53-91R
1993
Sp/93
Cusumano
and Software Factories
Problem Choice and Persistence among
Technological Communities and the Diffusion of
Debackere
Rappa
an Emerging Field
Scientists in the
Knowledge
Rappa
Debackere
Rappa
Debackere
Number
Date
56-91R
Author(s)
Title
The Voice
of the
Customer
Griffin
Hauser
10/91
57-92
The Influence
of Inter-Project Strategy
on Market Performance
in the
Auto Industry
58-92
1/92
59-92J?
Linking International Technology Transfer with Strategy and Management:
A
Literature
Commentary
Modeling Contribution-spans of
Cusumano
Elenkov
Scientists in a Field:
The Case of Cochlear Implants
Rappa
Garud
4/92
60-92i?
Nobeoka
Cusumano
1/92
Rappa
Technological Progress and the Duration of Contribution Spans
Debackere
2/92
Garud
61-92
9/91
62-92J?
A Socio-Cognitive Model of Technology Evolution:
The Case
of Cochlear Implants
On the Persistence of Researchers
in Technological
Garud
Development
Rappa
8/92
63-92
Life
on the
Frontier:
An International Comparison
of Scientists in an
Emerging Field
The
Social Construction of Technological Reality
Garud
Rappa
1/92
65-92
Superceded by 77-92
66-92
Windows Of Opportunity: Temporal
3/92
Debackere
Rappa
1/92
64-92
Garud
Rappa
(Rev. 2/93)
Patterns of Technological
Adaptation In Organizations
Tyre
Orlikowski
(Rev. 9/92)
67-92
4/92
68-92
Puritan
-
Bennett
— the Renaissance^*^ Spirometry System: Listening
to the Voice of the
How Consumers Allocate Their Time When Searching for Information
2/92
(Rev. 5/93)
69-92
Moving Ideas
to
Market and Corporate Renewal
7/92
70-92
Hauser
Customer
Hauser
Urban
Weinberg
Meyer
Utterback
Project
Management
in
Technology Innovation, Application and Transfer
Frankel
5/92
71-92
Investments of Uncertain Cost
Pindyck
9/92
72-92
Identifying Controlling Features of Engineering Design Iteration
9/92
73-92
Smith
Eppinger
Objectives and Context of Software Measurement, Analysis and Control
Cusumano
10/92
74-92
An
Empirical Study of Manufacturing Flexibility in Printed-Circuit Board Assembly
11/92
Suarez
Cusumano
Fine
75-92
11 /92
Japanese Technology Management: Innovations, Transferability, and the
Limitations of "Lean" Production
Cusumano
Number
Date
76-92
Author(s)
Title
Hauser
Customer-Satisfaction Based Incentive Systems
Simester
Wernerfelt
11/92
(Rev. 3/93)
77-91
The Product Family and the Dynamics
Meyer
of Core Capability
Utterback
11/92
78-92
11/92
79-92
12/92
80-92
11/92
81-92
and Organizational Coordination
Automobile Product Development
Multi-Project Strategy
Patterns of hidustrial Evolution,
in
Dominant Designs, and Firms' Survival
(Rev. 7/93)
Innovation from Differentiation: Pollution Control Departments and
Innovation in the Printed Circuit Industry
Answer Garden and
Nobeoka
Cusumano
Utterback
Suarez
King
Ackerman
the Organization of Expertise
1/92
82-92
Skip and Scan: Qeaning
Up
Resnick
Virzi
Telephone Interfaces
ll°(l
83-92
Developing
New Products and Services by Listening to the Voice of the Customer
Roberts
8/92
84-92
A
Comparative Analysis of Design Rationale Representatioi\s
3/92
85-93
1/93
86-93
2/93
An Introductory Note for Using
Ancdyze Nodes, Relationships, Partitions and Boundaries
Relational Data in Orgaiiizational Settings:
AGNI and
Netgraphs
An Knowledge
to
Asset-Based View of Technology Transfer in International
Joint Ventures
Lee
Lai
George
Allen
Rebentisch
Ferretti
(Rev. 9/93)
87-93
Managing Technological Leaps:
A Study of DEC's Alpha Design Team
Katz
4/93
(Rev. 6/93)
88-93
Tools for Inventing Organizations:
Toward a Handbook of Organizational Processes
5/93
89-93
5/93
90-93
5/93
The Impact of Knowledge and Technology Complexity on Decision Making
Software Development
Locating Adaptive Learning: The Situated Nature of Adaptive Learning in
Organizations
91-93R
5/93
Exploiting Opportimities for Technological
92-93
Scientific
3/93
93-93
7/93
Improvement
in Organizations
and Commercial Importance and the Source of Scientific Instrument
Inductive System Diagrams:
Meyer
Curley
Tyre
von Hippel
Tyre
Orlikowski
Riggs
von Hippel
Innovations
An
Empirically Based Theory Generation Technique
Burchill
Kim
7/93
94-93
Malone
Crowston
Lee
Pentland
The Hypercube of Innovation
Afuah
Bahram
Number
Date
95-93
Authorrsl
Title
McCord
Managing the Integration Problem in Concurrent Engineering
Eppinger
8/93
96-93
Beyond the Software
Factory:
A Comparison of
'Qassic' and
PC Software
Developers
9/93
97-93
9/93
98-93
Obstacles to Systemic Iimovation:
An Illustration from Semiconductor Manufacturing
Smith
Cusumano
Rappa
Technology
Utterback
Radical Innovation
9/93
99-93
Finding Out What Goes
On in a Software Development Organization
Paper numbers with the suffix
Perry
Staudenmayer
Votta
11/93
R
are reprints
The International Center
for Research
on the Management
of
Technology
Working Paper Order Form
Name:
Title:
Company:
Address:
n
I
would
like to
become a working paper subscriber and
during the current year. ($150
n
I
would
like to
U.S.,
Canada, Mexico; $175
receive all papers published
all
other countries)
order working papers individually. Please send
me
the following
papers:
Total
#
#
#
#
#
#
#
#
#
®
number papers ordered
$9.00 /paper
$_
Additional postage charges
(shipments outside US only)
$_
Subscription rate
$_
Total
Due
$_
Within the US All orders mailed first class.
Outside the US For orders to Canada add $.15 per paper for air delivery. For orders to
other countries, add $.25 per paper for surface delivery or $2.00 per paper for air delivery.
:
:
PAYMENT MUST ACCOMPANY
Make check
or
money
MTT
and send
to:
order (in
/
US fimds)
payable
ICRMOT
ICRMOT Working Papers
Mir, Room E56-390
Cambridge,
MA
02139-4307
to:
THIS
ORDER
hs
Date Due
Lib-26-67
*"ili!l
4974
3 9080 02163