See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/223198582
Comparing freshman and senior engineering design processes: An
in-depth follow-up study
Article in Design Studies · July 2005
DOI: 10.1016/j.destud.2004.09.005
CITATIONS
READS
274
1,324
4 authors, including:
Monica Cardella
Jennifer Turns
Florida International University
University of Washington
223 PUBLICATIONS 3,609 CITATIONS
200 PUBLICATIONS 4,390 CITATIONS
SEE PROFILE
All content following this page was uploaded by Robin Adams on 12 February 2018.
The user has requested enhancement of the downloaded file.
SEE PROFILE
Comparing freshman and senior
engineering design processes: an
in-depth follow-up study
Cynthia J. Atman, Monica E. Cardella, Jennifer Turns and Robin
Adams, Center for Engineering Learning and Teaching, University of
Washington, Seattle, WA 98195, USA
In this paper we report the results of an in-depth study of engineering
student approaches to open-ended design problems. We collected verbal
protocols from 61 senior (fourth year) engineering students and
reanalyzed protocols from 32 freshman (first year) engineering students
as they worked on two design problems. The design processes of these
student groups were compared. Results show that seniors produced higher
quality solutions, spent more time solving the problem, considered more
alternative solutions and made more transitions between design steps than
the freshmen. This dataset also includes protocols for 18 within-subject
participants. These students participated in the study first as freshmen
and later as seniors, affording us the opportunity to compare design
process changes over time on the individual student level. Finally, this
paper includes results for design process differences across the two design
problems.
Ó 2004 Elsevier Ltd. All rights reserved.
Keywords: design behaviour, design education, protocol analysis,
engineering education
D
Corresponding author:
C. J. Atman
atman@engr.washington.edu.
esign is exploratory; design is rhetorical; it is emergent,
opportunistic, reflective, risky, and an important human
endeavour (Cross, 1999). Design is a central activity to all
types of engineering. Mechanical, civil and electrical engineers attempt
to solve very different types of problems, but they all design some
solution to the problem at hand. Though the final products of the design
process are different, each engineer must be equipped with skills in
design in order to complete job functions. As such, there has been
a growing awareness of the importance of equipping engineering
students with design skills through the education experience.
www.elsevier.com/locate/destud
0142-694X $ - see front matter Design Studies 26 (2005) 325e357
doi:10.1016/j.destud.2004.09.005
Ó 2004 Elsevier Ltd. All rights reserved Printed in Great Britain
325
The role of design in engineering education has been recognized in the
United States in the context of national standards for engineering
programmes (ABET, 2000). For institutions to meet these standards, the
institutions must have an understanding of design and how students
learn design (Newstetter and McCracken, 2001). A starting point for
this is to characterize how students naturally go about solving design
problems and to determine what processes and activities distinguish
novice designers from expert designers. With this in mind, we can
characterize the way that freshman engineering students approach
design problems and the way that senior engineering students approach
design problems. By understanding the way that freshman and senior
engineering students practise design we can learn about the engineering
students’ initial design states and learn how an engineering education
affects the way students practice design.
Appropriate research methods and experimental tasks are vital to
accomplishing this task. A prevalent method in current research for
attempting to characterize design is verbal protocol analysis (VPA). It is
a well-documented approach in which subjects are instructed to ‘think
aloud’ while performing a task. We begin with a brief description of
example protocol studies that have revealed attributes of design. We
then describe our previous findings regarding differences between
freshman and senior engineers’ design behaviour and the experimental
tasks that we have used. Our current research extends these findings to
a new study populationd93 protocolsdand a different task set. This
research offers a rich description of freshman design behaviour, senior
design behaviour and differences between freshman and senior design
behaviour and extends previous findings to a new study population and
a new study task. Additionally, the study shows differences in design
behaviour for individual students that solved the problems both as
freshmen and as seniors.
1 Literature review
Researchers have approached the investigation of design in a multitude
of ways. Some have studied expert designers, others have characterized
students’ design behaviour. Many have used verbal protocol analysis.
Ericsson and Simon (1993) have demonstrated the validity of the verbal
protocol method and argue that concurrent reports are a valid method
to obtain data about thinking processes. They also argue that, if done
properly, think-aloud procedures do not influence the sequence of
subjects’ thoughts and that the resulting data can be treated as
objectively as any other data. Information is collected from short-term
memory while subjects are prompted to ‘keep talking’ with minimal
326
Design Studies Vol 26 No. 4 July 2005
interference from the experimenter. Verbal protocol analysis does
require a great amount of time (Ericsson and Smith, 1991); consequently, most of these studies are either case studies or small sample
studies.
In this section, we present some examples of verbal protocol studies of
engineering design. A more thorough review of the applications of
verbal protocol analysis to the study of design can be found in a book
chapter authored by Cross (2001).
Cross and Clayburn Cross (1998) explored the design strategies of two
expert designers by interviewing one and video recording the other as he
thought aloud while designing a backpack carrying device for
a mountain bicycle. While the two designers attended to different types
of design tasks and the researchers studied their expertise using two
different methods, the researchers noticed three parallels in the
designers’ strategies. The designers used a systemic approach to the
problem, invested a good amount of effort into carefully framing
the problem and relied on first principles of design despite their extensive
design experience. Though Cross and Clayburn Cross also had a small
sample size, their use of two different methods to explore two different
domains of design expertise shows that their findings are not specific to
a particular task, designer or method. The second strategy that Cross
and Clayburn Cross noticed, the designers’ careful framing of the
problem, has begun to receive attention in other studies as well. Some
researchers discuss this as task clarification (Lewis and Bonollo, 2002),
specification (Radcliffe and Lee, 1989) and problem analysis (Römer
et al., 2000).
In their overview of twelve years of empirical studies of engineering
design, Pahl et al. (1999) noted several strategies that yielded good
solutions. One of these is ‘thorough goal analysis’, something that starts
early in the process like the problem framing described in the Cross and
Clayburn Cross study. Pahl et al. also offer a discussion of other
elements of the design processdsearching for a solution, solution
analysis and decision making. Their observations are drawn from an
accumulation of 40 laboratory experiments. While it is unknown how
many participants were involved in these 40 experiments, this series of
experiments suggests that these findings are consistent for a number of
designers attending to a number of tasks. While Pahl et al. have studied
engineering design in Germany, other researchers worldwide have also
attempted to characterize the individual steps of the design process
Comparing freshman and senior engineering design processes
327
(Ullman et al., 1988; Radcliffe and Lee, 1989; Gero and McNeill, 1998;
Lewis and Bonollo, 2002).
Research has shown that engineers do not simply progress step by step
through this design process but instead iterate through cycles of
proposal, testing and modification. Smith and Tjandra (1998) offer
a review of several different models of iteration and compare them to
their own experimental results. Their results are based on observations
of nine teams of four engineering students. Smith and Tjandra found
that while no existing model accounted for all of their observations,
features from each of the models matched most of their observations.
They noticed that the student teams began with a short period of noniterative design during which the students shared goals and perceptions
on what were the most important design considerations. During the
following iterative portion, the groups engaged in a combination of
analysis and synthesis activities. Smith and Tjandra suggested that their
observations should be further substantiated through the observation of
design iteration on professional industrial projects. Other researchers
add to our understanding of the role of iteration in design through
studies of both students and practitioners attending to individual and
group tasks (Ennis and Gyeszly, 1991; Ball and Ormerod, 1995;
Brereton et al., 1996; Brockman, 1996; Adams, 2001).
These examples have illustrated both the power of verbal protocol
analysis in the wide variety of insights it has yielded and the
shortcomings of this methodology, such as small sample sizes and
limited replication of results. The deep findings of verbal protocol
analysis have extraordinarily advanced our understanding of design yet
need to be followed up with studies showing the same empirical,
statistical findings. We now present an overview of the previous studies
that we have conducted. A strength of these studies is the large data sets
that allow for statistical analyses. A more detailed review of these studies
is presented elsewhere (Atman and Turns, 2001; Adams et al., 2003).
1.1 Previous protocol studies
In a series of three lab-based studies we have investigated the design
processes of freshman and senior engineering students. One of our
guiding goals has been to determine if the students with more academic
experience used more sophisticated design processes while solving
engineering design problems. The first study, The First Semester
Freshman Study (Mullins et al., 1999), compared 16 freshmen that
had already completed a semester of study to 16 freshmen that had not
yet begun their college studies as they solved two relatively short design
328
Design Studies Vol 26 No. 4 July 2005
problems. In the first problem the students designed a ping-pong ball
launcher (PP), and in the second the students designed a means for
crossing a busy street (SC). A second study, the Design Text Study
(Atman and Bursic, 1996), compared five freshmen that had just finished
reading a short design text to five freshmen that had not yet read the text
solving the same two problems used in the First Semester Freshman
Study. Finally, we compared 24 senior engineering students to 26
freshmen as they designed a playground (PG) for a fictitious neighbourhood in the FreshmaneSenior Comparison, or Playground, Study
(Atman et al., 1999). Table 1 shows the measures used for each study
and the study results.
In particular, the table shows the measures used in each of the earlier
protocol analysis studies and whether significant differences between the
two comparison groups were found for the given measure in the given
study. The results associated with the first two studiesdthe First
Semester Freshman Study and the Design Text Studydinvolved
comparisons among freshmen rather than between freshmen and
seniors. Each of the measures used in a previous study also represents
a potential measure (or dependent variable) for the analysis of our
Table 1 Synthesis of study measures
Categories/Measures
Product measures
Solution quality
First Semester
Freshman
Study
Design Text
Study
Playground Senior
Study
Follow-Up
Study
PPa
SCb
PP
SC
PGc
PP & SC
-
(O)
-
-
O
yes
O
O
O
O
O
O
(O)
(O)
yes
yes
yes
yes
yes
O
(O)
O
O
O
O
-d
Process measures
Time spent
O
Time spent in decision step
Number of transitions
O
Transition rate
Number of alternative solutions
Number of design criteria considered
O
Number of explicit
information requests
Number of assumptions made
Number of information categories covered
Progression to later stages of process
O
O
O
O
yes
O = p ! 0.05, (O) = 0.05 ! p ! 0.10, - = p O 0.10
a
PP: participants designed a ping pong ball launcher
b
SC: participants designed a means for crossing a busy street at the participants’ university
c
PG: participants designed a playground for a fictitious neighbourhood
d
Although the number of alternative solutions considered was not significant for differentiating among
freshmen and seniors, this measure was correlated with solution quality
Comparing freshman and senior engineering design processes
329
current dataset. The measures are described in greater detail in Section
2.4. The final column of the table indicates which of these measures we
used in the current study, the Senior Follow-Up Study, by the presence
of a ‘yes’.
1.2 Research questions
With our current study we intend to explore some of the same questions
from these prior studies and determine if our measures yield consistent
results. Specifically, we seek to answer these questions:
What is the design process of freshmen likedwhat does the process
look like and what measures correlate with quality of solution?
What is the design process of seniors likedwhat does the process
look like and what measures correlate with quality of solution?
Do the freshmen use a different design process than the seniors? In
what ways is the senior design process different from that of the
freshmen?
Do individual students exhibit different design processes when they
are freshmen compared to when they are seniors?
Do engineering students exhibit different design behaviour for
different types of design problems?
In this paper we review the methodology we have used in our studies as
well as the participant group and task set for the current study. The
participant group for this study is particularly notable for its large size.
We present some results that are consistent with our prior findings as
well as some that are not. We also introduce within-subject findings for
a group of students that participated in the study both as freshmen and
as seniors.
2 Methodology
To accomplish the above objectives and answer the guiding questions,
we use verbal protocol analysis to document and describe student design
processes. In this study, students were asked to give a verbal protocol
(think aloud) as they individually solved four short open-ended design
problems. Three of these problems were used previously in the First
Semester Freshman and Design Text Studies (Mullins et al., 1999).
These problems were selected based on their ability to provide insight
into different aspects of engineering design skills.
2.1 Design problems
Problem 1 (ping pong) gives the participants the opportunity to use
mechanical and analytical skills (ECSEL, 1993). Problem 1 is also
330
Design Studies Vol 26 No. 4 July 2005
similar to the types of homework or in class problems students would
have encountered in their coursework. Problem 2 (street crossing) allows
the participants to consider a real world problem in a familiar context.
The text of these two problems is presented in Figure 1.
The experimental procedure consisted of several steps. The participants
solved two practice problems out loud to familiarize themselves with the
process of thinking aloud and then solved the ping pong and street
crossing problems. The participants solved two more design problems
after completing the street crossing problem, but the results of the
analyses of these problems are presented elsewhere (Bogusch et al., 2000;
Rhone et al., 2001; Louie, 2001). Both audio and video tapes were used
to collect subject protocols.
2.2 Participants
In addition to using the problem set from the First Semester Freshman
Study, we also used the pool of freshman data for our freshman sample
for the current study. To complement the 32 freshman protocols, we
collected protocols from 61 senior engineering students during the
freshman participants’ senior year. The original freshman participants
were contacted and asked to participate again; 18 agreed to participate
Figure 1 Problem statements
Comparing freshman and senior engineering design processes
331
during their last semester of school, resulting in 18 within-subject data
points. Additionally, 43 seniors new to the study participated after
completion of their capstone course, for a total of 61 seniors. The
students were each paid US$20. The average age of the freshmen was
18.0 years. Nine female freshmen and 23 male freshmen participated.
Thirty of the freshmen were Caucasian, one was African American and
one was Asian American. The average age of the seniors was 23.2 years.
The senior participants included 46 males and 15 females. The senior
population included 49 Caucasians, 1 African American, 5 Asian
Americans and 6 seniors who did not report ethnicity. The senior group
consisted of 16 chemical engineering, 9 civil engineering, 8 electrical
engineering, 3 engineering physics, 12 industrial engineering, 10
mechanical engineering and 3 materials science and engineering
students. The curriculum in each of these departments, with the
exception of engineering physics, included a capstone design course. The
average freshman age of the within-subject participants was 17.9 and the
average senior age was 21.9 years. The within-subject group consisted of
14 male and 4 female engineering students. One of these participants was
Asian-American and the rest were Caucasian.
2.3 Coding
Transcription, segmenting, and coding of the text from the audio tapes
allows us to describe student design behaviour. In addition, we are able
to determine the amount of time that subjects spent in various steps of
the design process from analysis of the videotapes. A detailed
description of the application of the verbal protocol method is provided
elsewhere (Atman and Bursic, 1998). Here we briefly describe the steps
involved:
Transcription. Each subject’s verbal protocol was transcribed from the
audio tape.
Segmenting. The purpose of segmenting is to break the verbal text into
units (segments) that can be coded with a pre-defined coding scheme.
For this study, a sentence formed the basic unit to be segmented. If
a sentence contained more than one idea, it was segmented into two or
more parts.
Coding. A variable called ‘design step’ was chosen to describe each
student’s design process on the first two problems. It is important to
note that while we use the term ‘step’, the design process does not
necessarily proceed in a linear fashion. Two coders coded each segment
in terms of design step, which identifies the step in the design process in
332
Design Studies Vol 26 No. 4 July 2005
which the subject is working. Although the freshman data had been
coded for the previous study, it was recoded for the current study. The
design process steps that were used as codes in this study (see Table 2)
are based on a content analysis of seven freshmen engineering design
texts (Moore et al., 1995). The coders checked their coding for intercoder reliability and if they agreed on at least 70% of the codes that they
had assigned, they discussed all differences on the transcript until they
reached full agreement. The average inter-coder reliability for all 186
transcripts (two problems for each of the 93 protocols) was 82%.
2.4 Design process measures
Table 1 shows the measures used for the current study, in relation to the
measures used for the previous studies.
In summary, the following independent variables were explored:
- Solution Quality: quality-of-solution score
- Time Spent (Design Time): total amount of time spent in design
activity (this excludes time spent talking to the experiment
administrator)
- Time Spent in Design Steps (e.g., modelling, evaluation, decision):
amount of time spent on individual design activities
- Time Spent in Design Stages: amount of time spent in problem
scoping (combination of time spent in problem definition and time
spent in gathering information), developing alternative solutions
(combination of generating ideas, modelling, feasibility analysis and
evaluation) or project realization (combination of decision and
communication)
Table 2 Coding scheme for verbal protocol data
Design Step (abbreviations used for Figures 2 and 3)
Identify need
Problem definition (PD)
Identify basic needs (purpose, reason for design)
Define what the problem really is, identify the constraints, identify
criteria, reread problem statement or information sheets, question
the problem statement
Gather information (GATH) Search for and collect information
Generate ideas (GEN)
Develop possible ideas for a solution, brainstorm, list different alternatives
Modelling (MOD)
Describe how to build an idea, measurements, dimensions, calculations
Feasibility Analysis (FEAS) Determine workability, does it meet constraints, criteria, etc.
Evaluation (EVAL)
Compare alternatives, judge options, is one better, cheaper, more accurate
Decision (DEC)
Select one idea or solution among alternatives
Communication (COM)
Communicate the design to others, write down a solution or instructions
Implementation
Produce or construct a physical device, product or system
Comparing freshman and senior engineering design processes
333
- Number of Transitions: number of transitions made between design
steps
- Transition Rate: number of transitions between design steps per
minute
- Number of Alternative Solutions: number of potential solutions
considered
- Progression to Later Stages: amount of time in decision, communication and project realization as well as percent of time spent in
project realization; ratio of time spent in the project realization stage
to time spent in the developing alternative solutions stage
2.5 Quality score
After all 93 protocols had been collected and transcribed for analysis,
evaluators assigned a ‘quality of product’ score for problems one and
two based on the final solution for problems one and two. Five
professors initially developed a scoring rubric for all of the solutions
generated during the First Semester Freshman Study. The professors
evaluated each suggested alternative to rank them based on their
abilities to meet design criteria (flight time and accuracy for the ping
pong problem; cost efficiency and accident reduction for the street
crossing problem). For the ping pong problem, these alternatives
included a cannon, catapult, slingshot, tennis ball launcher, spring, or
see-saw. For the street crossing problem the alternatives were a bridge,
crossing guard, tunnel, gate, or sensor. The professors were then asked
to determine which of the design features from a master list were
necessary for the various alternatives to function. Finally, the professors
made pairwise comparisons between all possible pairs of design
solutions, and the constant sum algorithm was then applied to these
judgements in order to arrive at the set of relative weights. Scores were
then based on the sum of the weights of the ‘necessary’ features they
included in the participants’ final solutions (Atman and Bursic, 1996;
Mullins et al., 1999).
Because this rubric was originally developed for the freshman solutions,
some of the seniors’ solutions did not appear on the rubric. Two
researchers created new rubrics to accommodate the new potential
solutions. Using both rubrics, two evaluators independently evaluated
each solution for problem one and for problem two for all participants.
The two evaluators met to check the reliability of their scores and if
they were at least 70% in agreement, they arbitrated these scores until
they reached a consensus. If the reliability rating was less than 70%,
the evaluators returned to the transcripts later to re-score them,
334
Design Studies Vol 26 No. 4 July 2005
independently, and then met again for arbitration. The average interevaluator reliability was 93.5%.
2.6 Coding to determine number of solutions
In addition to the final solution scoring and the design step coding,
a coding scheme to determine number and type of solutions was applied
to each protocol. Two evaluators assessed each protocol using the
solution types described in the quality scoring rubrics. The coding for
number of alternative solutions differed from the quality scoring in that
the evaluators identified and coded every alternative that the participant
considered rather than just the final solution. As was the case with the
design step coding and quality scoring, the two evaluators independently coded the transcripts and then compared coding. If the
evaluators met the 70% agreement criteria, they discussed the coding
until a consensus was reached. If the 70% reliability level was not met,
the evaluators independently recoded the transcripts. The average interevaluator reliability level was 97%.
3 Results
The data yielded rich results. Some are consistent with the findings from
the FreshmaneSenior Comparison (Playground) Study and others are
inconsistent. Some add new insights. In this section we characterize the
design processes for the freshman and the senior groups using the
measures presented in Section 2.4 and discuss the measures that
correlated with quality of solution. We then describe some example
participants. We will also discuss comparisons made between these two
groups and present freshmanesenior differences for our within-subject
participants. Finally, we introduce findings for differences in design
process according to design task.
3.1 Freshmen
Figure 2 shows how the freshmen and seniors distributed their time
amongst the steps of the design process. Tables 3 and 4 present the
average values for the design process measures and final design quality
for both the freshman and senior participants on the ping pong (Table 3)
and street crossing (Table 4) problems. The freshmen spent an average of
6.3 min solving the ping pong problem and an average of 4.8 min solving
the street crossing problem.
Problem definition and modelling solutions dominated the design
processes of the freshmen. The freshmen allocated similar amounts of
time to the problem scoping and developing alternative solutions stages,
spending slightly more time problem scoping on the first problem and
Comparing freshman and senior engineering design processes
335
Design activity
Project Realization Developing Alternative Solutions
Problem Scoping
Ping Pong Problem
PD
GATH
GEN
MOD
FEAS
EVAL
Seniors
Freshmen
DEC
COM
0
15
30
45
60
Percent of total time
Design activity
Figure 2 Average percent of
time spent in each design step
for freshmen and seniors on
both problems (see Table 2
for design step abbreviations)
Project Realizatior Developing Alternative Solutions
Problem Scoping
Street Crossing Problem
PD
GATH
GEN
MOD
FEAS
EVAL
Seniors
DEC
Freshmen
COM
0
15
30
Percent of total time
45
60
slightly more time developing alternative solutions on the second
problem. For both problems, the freshmen spent very little time in the
project realization stage. Spending little time in project realization is
consistent with the findings from the Playground Study. However, in the
Playground Study, the freshmen spent dramatically more time in the
developing alternative solutions stage than in the problem scoping stage.
Also, the results from that study suggest that freshmen spend the most
time in the modelling step, followed by the gather information step and
then the generate ideas step.
336
Design Studies Vol 26 No. 4 July 2005
The results of the Senior Follow-Up Study are mainly consistent with
this, aside from the difference in the problem definition behaviour.
Unlike the freshmen in the Playground Study, the freshmen in the
Senior Follow-Up Study spent a large percent of time in the problem
definition step.
The freshman participants averaged 9.7 transitions between design steps
during the ping pong problem and 11.1 transitions during the total street
crossing problem design process. This resulted in a mean transition rate
of 2.4 transitions per minute for the ping pong problem and 3.4
transitions per minute for the street crossing problem. These transition
rates are both greater than the 1.2 transitions per minute rate for the
freshmen in the Playground Study.
Table 3 Summary statistics for freshmen and seniors: ping pong problem
Design Process Measure
Freshmen (n = 32)
Seniors (n = 61)
Time spent in design process
Avg. (min)
(Std. dev.)
Avg. (%)
(Std. dev.)
Avg. (min)
(Std. dev.)
Avg. (%)
(Std. dev.)
6.2 (6.3)
4.8 (5.7)
2.4 (1.8)
2.1 (1.4)
0.3 (0.7)
2.4 (4.1)
74.9 (17.0)
57.6 (20.0)
51.9 (20.5)
5.7 (6.5)
44.2 (20.0)
11.8 (6.7)
9.1 (5.3)
3.2 (1.3)
2.6 (1.1)
0.5 (0.6)
5.9 (4.7)
77.8 (12.9)
41.0 (17.9)
35.4 (18.2)
5.6 (6.0)
58.3 (18.2)
0.1 (0.2)
2.0 (3.6)
0.3 (0.4)
0.0 (0.1)
0.0 (0.0)
0.0 (0.0)
0.0 (0.0)
1.4 (1.3)
2.4 (4.0)
34.4 (20.2)
5.0 (5.3)
0.5 (1.3)
0.1 (0.2)
0.1 (0.2)
0.0 (0.1)
25.1 (17.0)
0.2 (0.3)
5.2 (4.5)
0.4 (0.6)
0.1 (0.2)
0.1 (0.2)
0.0 (0.1)
0.0 (0.1)
2.7 (2.1)
3.2 (4.1)
50.2 (19.8)
4.3 (5.2)
0.7 (1.5)
0.7 (2.2)
0.5 (1.7)
0.2 (1.0)
22.2 (12.9)
Avg. no. of
(Std. dev.)
9.7 (9.1)
5.3 (6.2)
Avg. no./min
(Std. dev.)
2.3 (1.3)
1.2 (0.8)
Avg. no. of
(Std. dev.)
18.6 (13.0)
11.7 (8.6)
Avg. no./min
(Std. dev.)
2.2 (1.0)
1.4 (0.8)
Total time
Design time
Problem scoping stage
Problem definition step
Gathering information step
Developing alternative
solutions stage
Generating ideas step
Modelling step
Feasibility analysis step
Evaluation step
Project realization stage
Decision step
Communication step
Non-design activity
Transition behaviour
Step transitions
Stage transitions
Alternative solutions
Objects coded
Design quality measures
Quality score (max = 3.618)
Average number of
(Std. dev.)
1.4 (0.7)
Average number of
(Std. dev.)
1.3 (0.7)
Average score
(Std. dev.)
1.0 (0.8)
Average score
(Std. dev.)
1.5 (0.7)
All time dependent variables (transitions per minute, percent of time spent in a step or stage) are calculated using
design time, with the exception of non-design activity. This is calculated as the percent of total time spent in nondesign activity
Comparing freshman and senior engineering design processes
337
Table 4 Summary statistics for freshmen and seniors: street crossing problem
Design Process Measure
Freshmen (n = 32)
Seniors (n = 61)
Time spent in design process
Avg., min
(std. dev.)
Avg., %
(std. dev.)
Avg., min
(std. dev.)
Avg., %
(std. dev.)
4.7 (2.9)
4.0 (2.6)
1.6 (0.7)
1.5 (0.6)
0.1 (0.2)
2.4 (2.1)
0.2 (0.2)
1.6 (1.7)
0.5 (0.6)
0.1 (0.1)
0.0 (0.0)
0.0 (0.04)
0.0 (0.0)
0.7 (0.6)
84.6 (9.1)
48.7 (18.4)
45.5 (18.3)
3.2 (4.8)
50.9 (18.4)
4.4 (3.7)
32.6 (20.5)
10.6 (7.7)
3.3 (6.0)
0.4 (1.1)
0.4 (1.1)
0.0 (0.0)
15.4 (9.1)
12.1 (7.6)
9.6 (6.1)
3.0 (2.0)
2.7 (1.7)
0.3 (0.5)
6.5 (4.5)
0.4 (0.4)
4.5 (3.6)
1.3 (1.4)
0.2 (0.3)
0.1 (0.1)
0.1 (0.1)
0.0 (0.1)
2.5 (2.2)
79.8 (10.7)
34.3 (13.6)
31.5 (13.8)
2.8 (3.8)
64.9 (13.5)
5.4 (5.9)
43.9 (17.2)
12.5 (8.8)
3.1 (4.7)
0.8 (1.3)
0.8 (1.2)
0.1 (0.3)
20.2 (10.7)
Avg. no. of
(std. dev.)
11.1 (7.4)
6.3 (4.3)
Avg. no./min
(std. dev.)
3.1 (2.6)
1.7 (1.0)
Avg. no. of
(std. dev.)
26.1 (19.2)
14.4 (10.5)
Avg. no./min
(std. dev.)
2.9 (1.3)
1.6 (0.9)
Total time
Design time
Problem scoping stage
Problem definition step
Gathering information step
Developing alternative solutions stage
Generating ideas step
Modelling step
Feasibility analysis step
Evaluation step
Project realization stage
Decision step
Communication step
Non-design activity
Transition behaviour
Step transitions
Stage transitions
Alternative solutions
Objects coded
Average number of (std. dev.)
1.4 (0.6)
Average number of (std. dev.)
2.5 (1.3)
Design quality measures
Quality score (max = 1.28)
Average score (std. dev.)
0.3 (0.2)
Average score (std. dev.)
0.5 (0.2)
All time dependent variables (transitions per minute, percent of time spent in a step or stage) are calculated using
design time, with the exception of non-design activity. This is calculated as the percent of total time spent in nondesign activity
The freshmen considered an average of 1.4 alternative solutions to the
ping pong problem before selecting their final solution and an also an
average of 1.4 alternative solutions to the street crossing problem. The
final solutions that the freshmen selected and designed resulted in
average scores of 1.0 (out of 3.618) on the ping pong problem and 0.32
(out of 1.28) on the street crossing problem. Both of these scores are less
than 30% of the total possible score.
Spearman Rank correlations were used to investigate possible relationships among the study variables. In this section, we report only
correlations with a coefficient of 0.5 or greater (absolute value).
Time, number of transitions and quality of solution correlated with each
other in the Playground Study. For the current study, quality did not
338
Design Studies Vol 26 No. 4 July 2005
correlate with transitions or with time for the freshmen for the ping pong
problem, but time and transitions were strongly correlated with each
other (0.70). The correlations for the street crossing problem, however,
are consistent with the Playground Study; there were strong correlations
between time and quality (0.69), transitions and quality (0.53) and
between transitions and time (0.67) for the freshmen. Additionally, the
freshman quality score correlated with many of the step and stage
measures. There were positive correlations with quality for amount and
percent of time in modelling (0.65, 0.56) and amount and percent of time
in developing alternative solutions (0.68, 0.64) and negative correlations
with quality for percent of time in problem definition (ÿ0.65), percent of
time in problem scoping (ÿ0.65) and the ratio of time spent in problem
scoping to developing alternative solutions (ÿ0.64).
3.2 Seniors
The seniors spent an average of 11.8 min on the ping pong problem and
12.7 min on the street crossing problem. As was the case with the
freshmen, modelling and problem definition dominated the design
process of the seniors. Figure 2 shows how the seniors distributed their
time on each problem across the different design activities.
It is important to note that while the seniors represented seven
engineering disciplines, an analysis of variance showed that generally
the seniors’ design behaviour did not vary according to major. There
was a significant difference in performance linked to major for only two
measures: ping pong stage transition rate, F(4,38) = 3.517, p = 0.011,
and street crossing amount of time spent in feasibility analysis, F(4,
38) = 2.642, p = 0.039. In the case of ping pong stage transition rate, the
electrical engineering students had a much higher average stage
transition rate (2.04) than the mechanical (0.95) or chemical (0.98)
engineering students. As for street crossing amount of time spent in
feasibility analysis, the industrial engineering students on average spent
a relatively large amount of time in feasibility analysis (2.17 s) in
comparison to the engineering physics students (0.38 s). The withinsubject participants were excluded from the analysis of variance.
For the ping pong problem, the seniors allocated the most time to
modelling, a large amount of time to problem definition, and then the
third highest amount of time to gathering information. On the street
crossing problem, the seniors also allocated the most time to modelling
and problem definition and then allocated the third highest amount of
time to feasibility analysis. The senior allocation of time differs from the
Comparing freshman and senior engineering design processes
339
freshman allocation only in that the freshmen spent more time in
problem definition than in modelling. This allocation of time is very
different from the way that the seniors who participated in the
Playground Study spent their time. The seniors from the Playground
Study spent more than half their time in modelling and then spent the
next greatest amount of time gathering information followed by
generating ideas with only a small percent of time allocated to problem
definition.
The seniors made an average of 18.6 transitions between design steps on
the ping pong problem and an average of 26.1 transitions on the street
crossing problem. For the seniors, this resulted in average transition
rates of 2.2 transitions per minute for the ping pong problem and 3.4
transitions per minute for the street crossing problem. These transition
rates are also higher than the average transition rate of 1.8 transitions
per minute exhibited by the seniors in the Playground Study.
Similar to the freshmen, the seniors considered few alternative solutions
for the ping pong problemdan average of only 1.3. On the street crossing
problem, however, the seniors considered an average of 2.5 alternative
solutions. Their final design solutions earned higher quality scores than
the freshmen’s (1.5 for the ping pong problem and 0.5 for the street
crossing problem) but these average scores are still low; they are
approximately 40% of the maximum possible scores.
As was the case in the Playground Study, there are fewer interesting
correlations for the seniors than there were for the freshmen. Neither
time, transitions nor transition rate is correlated with quality of solution
for the seniors on the ping pong problem. In fact, there are no measures
correlated with quality with a correlation coefficient greater than 0.5.
However, transitions and time (0.66) and transitions and transition rate
(0.54) are correlated for the seniors on the ping pong problem, and
transitions and time are correlated (0.73) on the street crossing problem.
This is consistent with both the results for the freshmen for the current
study and the results from the Playground Study. Finally, the fact that
the correlations for the seniors are weaker than those for the freshmen is
also consistent with the findings from the Playground Study.
3.3 Example participants
Figure 3 shows timelines representing three example participants’
performance on each problem. The timelines were created using
MacSHAPA (Sanderson et al., 1994). Time is presented from left to
right and each design step is listed along the horizontal axis. As
340
Design Studies Vol 26 No. 4 July 2005
Freshman W-T spent time in a design step, a block is placed on the line
for that step. The width of the block represents the amount of
uninterrupted time that Freshman W-T spent in the step. Wider blocks
suggest that the participant is staying in one step rather than
transitioning between steps.
Freshman W-T represents a typical freshman performance. This
freshman spent near average amounts of time on the ping pong problem
(4.4 min) and street crossing problem (3.5 min). Freshman W-T made
eight transitions between design activities on each problem, and received
average quality scores (1.4 and 0.4). Freshman W-T did not exhibit
progression into the later steps of the design process on either problem.
Senior NW-T represents typical senior design behaviour for the two
problems. Senior NW-T spent an average amount of time on the ping
Freshman W-T*: Ping Pong Problem
Senior NW-T: Ping Pong Problem
00:00:00:00 00:02:15:0 00:04:30:00 00:06:45:0 00:09:00:00
00:00:00:00 00:02:15:00
PD
GATH
GEN
MOD
FEAS
EVAL
DEC
COM
PD
GATH
GEN
MOD
FEAS
EVAL
DEC
COM
Freshman W-T*: Street Crossing Problem
Senior NW-T: Street Crossing Problem
00:00:00:00 00:02:15:00 00:04:30:00 00:06:45:00 00:09:00:00
00:00:00:00 00:02:15:00
PD
GATH
GEN
MOD
FEAS
EVAL
DEC
COM
PD
GATH
GEN
MOD
FEAS
EVAL
DEC
COM
Senior W-E*: Ping Pong Problem
00:00:00:00 00:02:15:00 00:04:30:00 00:06:45:00 00:09:00:00 00:11:15:00 00:13:30:00 00:15:45:00
PD
GATH
GEN
MOD
FEAS
EVAL
DEC
COM
Senior W-E*: Street Crossing Problem
00:00:00:00 00:02:15:00 00:04:30:00 00:06:45:00 00:09:00:00 00:11:15:00 00:13:30:00 00:15:45:00
Figure 3 Timelines for example students: a typical freshman (W-T), a typical senior
(NW-T) and an exceptional
senior (W-E)
PD
GATH
GEN
MOD
FEAS
EVAL
DEC
COM
* Freshman W-T and Senior W-E are an example of within-subject data—these are two data
points for the same student.
Comparing freshman and senior engineering design processes
341
pong problem and a slightly below average amount of time on the street
crossing problem. Senior NW-T made an average number of transitions
between design steps on each problem and received quality scores that
were similar to the average score for the seniors for each problem. This
senior did not exhibit progression to the later steps of the design process
on the ping pong problem, but did spend time in decision on the street
crossing problem (though briefly).
Senior W-E represents an exceptional senior. Additionally, Senior W-E
represents within-subject change; Freshman W-T and SeniorW-E are
the same participant at two different points in time. Senior W-E spent
above average amounts of time solving the problems, made more
transitions between design steps than the average for the seniors,
received higher quality scores and spent more time in the later steps of
the design process (communication on the ping pong problem and
decision on the street crossing problem). While this participant showed
similar design behaviour as a freshman and as a senior for the ping pong
problem, Senior W-E showed more sophisticated design behaviour, or
growth, as a senior on the street crossing problem. Each pair of withinsubject timelines (one pair per participant per problem) was classified as
exhibiting change, more of the same, no change or simplification. A
discussion of the method for classifying the transcripts is presented in
Turns et al. (2002). The transcripts for this participant (Senior W-E and
Freshman W-T) were classified as ‘more of the same’ for the ping pong
problem and ‘change’ for the street crossing problem.
3.4 Across-subject differences
Table 5 shows the measures used for across-subject comparisons for
the current study as well as the previous studies. It shows the measures
which consistently yielded across-subject differences that were significant as well as the measures that yielded significant differences for
some studies but not for others. The final measure, ‘progression to later
stages of process’, encompasses a number of measures: amount of time
in decision, amount of time in communication, amount of time in
project realization, percent of time in project realization and ratio of
time spent in project realization to time spent in developing alternative
solutions. A check in the progression row signifies a significant
difference in one or more of these component measures. Independent
samples t-tests determined the statistical significance of the acrosssubject differences.
The seniors exhibited more sophisticated design behaviour than the
freshmen; generally, the current study’s findings confirm the previous
342
Design Studies Vol 26 No. 4 July 2005
Table 5 Synthesis of study measures
Categories/Measures
Product measures
Solution quality
Process measures
Time spent
Time spent
in decision step
Number of transitions
Transition rate
Number of
alternative solutions
Progression to later
stages of process
First Semester
Freshman Study
Design Text
Study
Playground
Study
Senior Follow-Up
All
PPa
SCb
Within
PP
SC
PGc
PP
SC
PP
SC
-
(O)
-
-
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
-
O
O
O
O
O
O
(O)
(O)
O
O
-d
O
-
O
O
O
-
O
O
O
O
O
(O)
O
O = p ! 0.05, (O) = 0.05 ! p ! 0.10, - = p O 0.10
a
PP: participants designed a ping pong ball launcher
b
SC: participants designed a means for crossing a busy street at the participants’ university
c
PG: participants designed a playground for a fictitious neighbourhood
d
Although the number of alternative solutions considered was not significant for differentiating among
freshmen and seniors, this measure was correlated with solution quality
findings. Though they did not transition at a faster rate than the
freshmen (as was the case in previous studies), the seniors did spend
more time solving the problem (ping pong problem, ppp ! 0.001, street
crossing problem, psc ! 0.001), transition more times between design
steps (ppp ! 0.001, psc ! 0.001), transition more times between design
stages (p ! 0.001, p ! 0.001) and spend more time making decisions
(ppp = 0.042, psc = 0.001) than did the freshmen. In most cases, the
seniors exhibited more ‘progression’ than the freshmen. The seniors
spent more time (ppp = 0.023, psc ! 0.001) and a larger percent of time
in the project realization stage (ppp = 0.021, psc = 0.033) and more time
in the decision step (ppp = 0.042, psc ! 0.001) than did the freshmen.
However, they did not spend significantly more time communicating
their designs (ppp = 0.055, psc = 0.193) on either problem and did not
have a significantly higher ratio of time spent in project realization to
time spent in developing alternative solutions on the street crossing
problem, psc = 0.225 (but did have a higher ratio on the ping pong
problem, ppp = 0.034). The lack of a difference in communication is
consistent with the Playground Study but the ratio of time in the project
realization stage to time in the developing alternatives solutions stage
measure yielded a statistical difference between the freshmen and seniors
on that previous study.
Comparing freshman and senior engineering design processes
343
Beyond the measures that showed significant freshmanesenior differences in previous studies, the seniors in this study also differed from the
freshmen on other measures. In addition to spending more time in the
decision step than the freshmen, the seniors also spent significantly more
time in almost every other step. This is true for amount of time in
problem definition (ppp = 0.015, psc ! 0.001), amount of time generating ideas (ppp = 0.031, psc ! 0.001), amount of time modelling solutions
(ppp ! 0.001, psc ! 0.001) and amount of time in feasibility analysis
(ppp = 0.049, psc ! 0.001) for both problems, and amount of time
gathering information (psc = 0.010) and making evaluations
(psc ! 0.005) on the street crossing problem. The seniors also spent
a larger percent of time in decision (ppp = 0.044, psc = 0.050) and
a larger percent of time in modelling than the freshmen (ppp ! 0.001,
psc ! 0.005) on both problems. Finally, the seniors spent more time in
the problem scoping stage (ppp = 0.015, psc ! 0.001), spent more time
(ppp ! 0.001, psc ! 0.001) and a larger percent of time (ppp ! 0.001,
psc ! 0.001) in the developing alternative solutions stage and had
a higher ratio of time in project realization to time in problem scoping
(ppp = 0.027, psc = 0.020) than the freshmen. The freshmen, however,
spent a higher percent of their time in problem definition (ppp ! 0.001,
psc ! 0.001) and in the problem scoping stage (ppp ! 0.001, psc ! 0.001)
than the seniors.
Another deviation from the Playground Study is unique to the street
crossing problemdhere the seniors considered more alternative solutions than the freshmen (psc ! 0.001). Finally, the seniors created
higher quality products than did the freshmen (ppp ! 0.005, psc ! 0.001).
This last finding is consistent with findings from previous studies.
3.5 Within-subject differences
In addition to demonstrating that the findings from the Playground
Study can be replicated for a different study population attending to
a different study task, the Senior Follow-Up Study shows that the
freshmanesenior differences also exist on the individual student level.
The Senior Follow-Up Study shows possible learning for individual
students using the same time, transition, and quality measures.
Due to the nature of the data, for the within-subjects group we
performed paired samples t-tests rather than independent samples
t-tests. Also, in order to compare the within-subject senior performance
to the freshman performance, we needed to ensure that the senior data
were not affected by the freshman administration of the problem set. To
investigate a possible pre-test effect we compared the within-subject
344
Design Studies Vol 26 No. 4 July 2005
senior data to the data for the non-within-subject seniors. We found that
the two groups performed similarly for most measures. Major
exceptions include: quality of ping pong solution (ppp = 0.008) and time
spent on the street crossing problem (psc = 0.004). The within-subject
seniors earned higher quality scores than the non-within-subject seniors
(within-subject mean = 1.9, non-within-subject mean = 1.3) on the ping
pong problem but did not use a noticeably different design process. The
main difference in design process between the two groups of seniors for
the street crossing problem is the amount of time spent on the problem;
the within-subject seniors spent more time on the second problem
(within-subject mean = 11.3 min) than the other seniors (non-withinsubject mean = 7.0 min). The remaining measures show no differences
between the two groups of seniors.
The results for the within-subject participants who provided protocols
first as freshmen and later as seniors are presented in Tables 5 and 6. The
final two columns of Table 5 indicate whether a measure showed
a significant change between freshman and senior performance for the
within-subject group. Table 6 presents the data for the average
differences between freshman and senior performance for every
measure. Similar to the entire population of seniors and freshmen, the
participants who solved the ping pong problem first as a freshman and
later as a senior scored higher as seniors, ppp = 0.001; on average the
students increased their scores by 0.8 (on a 0e3.618 scale). Likewise, the
students that participated in the study both as a freshman and later as
a senior spent a significantly larger amount of time on the problem
(mean = 6.8 min) as seniors than as freshmen, ppp ! 0.001. For the
within-subject participants, there was also a significant difference in
number of transitions between the senior and the freshman performance
(mean = 12.78), ppp ! 0.001. Likewise, the students made an average of
9.2 more stage transitions as seniors than they did as freshmen
(ppp = 0.001). Just as the entire group of freshmen tended to have
a higher transition rate than the entire group of seniors, the withinsubject freshmen also had a slightly higher transition rate (mean = 0.03)
than the seniors, which also was not statistically significant
(ppp = 0.540). Also, similar to the larger group of freshmen and seniors,
the within-subject participants increased their stage transition rate by an
average of 0.2 transitions per minute; this increase from freshman to
senior performance was also not statistically significant, ppp = 0.161.
These students also tended to spend more time in the modelling step as
seniors than they did as freshmen (mean = 4.5 min), ppp = 0.018 and
they also spent a larger percent of their time in the modelling step as
seniors (mean = 17.2%), ppp = 0.004. The only other change in these
Comparing freshman and senior engineering design processes
345
Table 6 Summary statistics for within-subject performance: difference between freshman and senior
performances
Design Process Measure
Ping Pong (n = 18)
Street Crossing (n = 18)
Time spent in design process
Avg., min
(Std. dev.)
Avg., %
(Std. dev.)
Avg., min
(Std. dev.)
Avg., %
(Std. dev.)
Total time difference
Design time difference
Problem scoping stage difference
Problem definition
step difference
Gathering information
step difference
Developing alternative
solutions stage difference
Generating ideas step difference
Modelling step difference
Feasibility analysis step difference
Evaluation step difference
Project realization stage difference
Decision step difference
Communication step difference
Non-design activity difference
6.8 (6.4)
5.3 (5.7)
0.6 (2.0)
0.1 (1.3)
3.4 (18.3)
ÿ17.3 (19.0)
ÿ18.9 (18.7)
9.5 (7.6)
7.5 (6.4)
1.9 (2.1)
1.7 (1.8)
2.9 (15.7)
ÿ14.3 (19.5)
ÿ14.2 (18.1)
0.4 (1.0)
1.6 (6.8)
0.2 (0.6)
ÿ0.1 (6.9)
4.5 (4.5)
16.6 (18.1)
5.4 (4.9)
13.6 (19.6)
0.0 (0.3)
4.5 (4.1)
0.2 (0.9)
0.0 (0.1)
0.1 (0.1)
0.0 (0.1)
0.2 (0.1)
1.5 (2.8)
0.2 (6.9)
18.5 (21.7)
ÿ2.1 (7.9)
ÿ0.1 (1.7)
0.8 (2.4)
0.7 (2.4)
0.1 (0.4)
ÿ3.4 (18.3)
0.2 (0.3)
3.9 (3.1)
1.1 (1.7)
0.2 (0.3)
0.1 (0.2)
0.1 (0.1)
0.0 (0.1)
2.0 (2.5)
ÿ2.0 (5.2)
15.3 (26.6)
0.6 (14.5)
ÿ0.7 (6.2)
0.7 (1.7)
0.4 (1.5)
0.3 (0.8)
ÿ2.9 (15.7)
Avg. no. of
(Std. dev.)
12.8 (13.5)
9.2 (9.2)
Avg. no./min
(Std. dev.)
ÿ0.1 (1.8)
0.3 (1.2)
Avg. no. of
(Std. dev.)
19.4 (19.8)
11.5 (10.0)
Avg. no./min
(Std. dev.)
ÿ0.3 (1.5)
ÿ0.1 (0.9)
Transition behaviour
Step transitions difference
Stage transitions difference
Alternative solutions
Difference in number
of objects coded
Design quality measures
Quality score difference
(max ping pong
score = 3.618;
street crossing = 1.28)
Average number of (Std. dev.)
ÿ0.2 (1.0)
Average number of (Std.
dev.)
1.1 (1.3)
Average score (Std. dev.)
0.8 (0.9)
Average score (Std. dev.)
0.2 (0.3)
All time dependent variables (transitions per minute, percent of time spent in a step or stage) are calculated using
design time, with the exception of non-design activity. This is calculated as the percent of total time spent in nondesign activity. Differences are calculated by subtracting freshman value from senior value. A negative value
indicates that the freshman value was greater than the senior value
students in terms of activity in individual steps was that the students
spent a larger percent of time in problem definition as freshmen
(mean = 30.4%) than as seniors (mean = 17.0%), ppp = 0.001, though
they spent a similar amount of time.
For the street crossing problem, the participants who solved the problem
first as a freshman and later as a senior again scored higher as seniors,
346
Design Studies Vol 26 No. 4 July 2005
psc = 0.003, this time with a 0.2 increase in quality score (on a 0e1.28
scale). Likewise, among the students that participated in the study both
as a freshman and later as a senior, the students spent an average of 9.5
more minutes on the problem as seniors than as freshmen, psc ! 0.001.
Also, for the within-subject participants, there was a significant gain
(mean = 19.4) in number of transitions between the seniors and the
freshmen, psc ! 0.001. The seniors also showed a significant increase of
11.5 more stage transitions as seniors, psc ! 0.005. The within-subject
participants, like the general population of participants, had a slightly
higher (mean = 0.2) transition rate as freshmen than as seniors, which
statistically was not significantly different, psc = 0.719. On average, the
overall stage transition rate did not change, psc = 0.569. As was the case
for the performance of the larger population of the freshmen and seniors
on the street crossing problem, the within-subject seniors spent
significantly more time in each design step than did the freshmen, still
with the exception of the communication step. Here the seniors tended
to spend more time than they did as freshmen, but the 2.5 s difference is
not statistically significant (psc = 0.227). As was the case on the ping
pong problem for the within-subject students, the students tended to
spend more time (mean = 1.7 min) in problem definition as seniors than
as freshmen, psc = 0.002, but a larger percent of time in problem
definition as freshmen than as seniors (mean = 9.9%) psc = 0.029, on
the street crossing problem. Finally, the seniors both spent an average of
4.0 more minutes in modelling and 13% more time in modelling than
they did as freshmen psc ! 0.0005, psc = 0.013.
3.6 Across-problem differences
The inclusion of multiple design problems in this study afforded us
a new opportunity to compare students’ behaviour across problems. For
some measures, the study participants exhibited different behaviour on
the ping pong problem than on the street crossing problem. Acrossproblem differences were analyzed by comparing across measures in
Tables 5 and 6 for senior, freshmen, and within-subject groups. For ease
of comparison, significant differences for the seniors and freshmen are
summarized in Table 7, and differences for the within-subjects group are
summarized in Table 8. Paired samples t-tests were used to determine
statistical significance.
As illustrated in Table 7, seniors differed in how they approached the
street crossing and ping pong problems. The seniors’ design processes
differed across problem in terms of (1) transition behaviour, (2) problems
scoping behaviour, (3) developing alternative solution behaviour and (4)
number of alternative solutions considered. Using the street crossing
Comparing freshman and senior engineering design processes
347
Table 7 Summary statistics for freshmen and seniors: average across-problem differences
Design Process Measure
Freshmen (n = 32)
Time spent
in design process
Ping
pong
Street
crossing
p value
2.3
1.4
0.006
55.6
46.7
0.023
2.1
1.5
0.003
Problem scoping
stage (min)
Problem scoping
stage (%)
Problem definition (min)
Gathering information (min)
Gathering information, (%)
Developing alternative
solutions stage (%)
Generating ideas (min)
Generating ideas (%)
Feasibility (min)
Feasibility (%)
Evaluation (min)
Evaluation (%)
Modelling (%)
Non-design (min)
Non-design (%)
Stage relationships
Ratio of problem
scoping to developing
alternative solutions
Transition behaviour
Step transition
Stage transition
Step transition
rate: avg. no./min
Alternative solutions
Objects coded
44.2
52.8
Seniors (n = 61)
0.027
0.3
5.0
0.0
0.5
0.5
10.6
0.1
3.3
0.008
0.001
0.018
0.016
1.4
25.1
0.7
15.4
0.002
0.001
Ping pong
Street
crossing
p value
41.0
34.3
0.011
0.5
5.6
58.3
0.3
2.8
64.9
0.017
0.003
0.014
0.2
3.2
0.4
4.3
0.1
0.7
50.2
0.4
5.4
1.3
12.5
0.2
3.1
43.9
0.005
0.015
!0.001
!0.001
!0.001
!0.001
0.019
1.0
0.6
0.024
18.6
11.7
2.2
26.1
14.4
2.9
0.002
0.034
!0.001
2.5
1.3
!0.001
All time dependent variables (transitions per minute, percent of time spent in a step or stage) are calculated using
design time, with the exception of non-design activity. This is calculated as the percent of total time spent in nondesign activity
problem as the point of comparison, seniors were more likely to
transition more between steps and between stages (psteps = 0.002,
pstages = 0.034) and transition at a faster rate (p ! 0.001). Seniors spent
a smaller percent of their time in the problem scoping stage (p = 0.011)
and a larger percent of their time in the developing alternatives stage
(p = 0.014). The problem scoping difference is largely attributable to the
gathering stepdfor the street crossing problem the seniors spent less time
gathering information (p = 0.017) and a smaller percent of time in this
348
Design Studies Vol 26 No. 4 July 2005
Table 8 Summary statistics for within-subject performance: average across-problem differences
Design Process Measure
Ping Pong (n = 18)
Street Crossing (n = 18)
p Value
0.011
0.009
Time spent in design process
Design time difference (min)
Problem definition time difference (min)
5.3
0.1
9.5
1.6
Alternative solutions
Difference in number of objects coded
ÿ0.2
1.1
Differences are calculated by subtracting freshman value from senior value. A negative value indicates that the
freshman value was greater than the senior value
step (p = 0.003). For the developing alternative solutions stage, there are
differences for nearly all the steps that the stage encompasses in terms of
a larger amount time spent and a larger percent of time spent in the
generating ideas step (pamount = 0.005, ppercent = 0.015), feasibility
analysis step (pamount ! 0.001, ppercent ! 0.001), and evaluation step
(p = 0.001, ppercent ! 0.001). However, seniors spent a larger percent of
time in the modelling step on the ping pong than on the street crossing
problem (p = 0.019). Related to the developing alternative solutions
stage differences, the seniors considered a larger number of alternative
solutions on the street crossing problem than on the ping pong problem
(p ! 0.001). Finally, the seniors had a higher ratio of time spent in the
problem scoping stage to time spent in the developing alternative
solutions stage on the ping pong problem than on the street crossing
problem (p = 0.024).
Like the seniors, the freshmen’s problem scoping and developing
alternative solution behaviour differed across the two problems. A
difference unique to the freshmen was the time and percent of time spent
in non-design activities (e.g., activities that could not be coded as a design
step). On the street crossing problem, freshmen spent a smaller percent of
their time in the problem scoping stage (p = 0.023) and a larger percent
of their time in the developing alternative solutions stage (p = 0.027)
than on the ping pong problem. The freshmen also spent more time in the
problem scoping stage for the ping pong problem (p = 0.006), and much
of this can be attributed to differences in problem definition behaviour
rather than gathering information behaviour. The freshmen spent nearly
40% more time in the problem definition step on the ping pong problem
(p = 0.003). For the developing alternative solutions stage, the acrossproblem differences are similar to those of the seniors for feasibility
analysis and evaluation. For the street crossing problem, freshmen spent
more time and a larger percent of time in feasibility (pamount = 0.008,
ppercent = 0.001) and evaluation (pamount = 0.018, ppercent = 0.016).
Comparing freshman and senior engineering design processes
349
Finally, the freshmen spent more time, and a larger percent of time, on
non-design activities on the ping pong problem than on the street crossing
problem (pamount = 0.002, ppercent = 0.001).
Across-problem differences in changes between freshmen and senior
participation for the within-subject participants include: (1) change in
total amount of time spent in design activity, (2) change in amount of
time spent in problem definition, and (3) change in number of
alternative solutions considered. The within-subject participants
showed a higher increase in total amount of time spent in design on
the street crossing problem (9.5 min) than on the ping pong problem
(5.3 min), p = 0.011. The within-subject participants also showed
a much more dramatic increase in time spent in the problem definition
step on the street crossing problem (1.6 min) than on the ping pong
problem (0.1 min), p = 0.009. Finally, the within-subject participants
showed a large positive increase in number of alternative solutions
considered for the street crossing problem (1.1 alternatives), p = 0.005,
while on average they showed a slight decrease in number of alternatives
considered on the ping pong problem (ÿ0.2 alternatives), p = 0.235.
Overall, there were across-problem differences for the number of
transitions, transition rate, number of alternative solutions, and time
spent in particular steps and stages of the design process. For example,
for both the larger set of seniors and the seniors in the within-subjects
groups there were across-problem differences in the number of
alternative solutions generated. Similarly, the seniors transitioned more
and more frequently for the street crossing problem than the ping pong
problem. Finally, both the seniors and freshmen spent more time in the
developing alternative solutions stage and less time in the problem
scoping stage for the street crossing problem than the ping pong problem.
However, we did not see any differences for the project realization stage
or our other measures of progression. The freshmen were more likely to
spend time in non-design activities for the ping pong problem than the
street crossing problem, and an across-problem difference unique to the
within-subjects group is total time spent in design activity.
4 Discussion
4.1 Replication of findings
For this experiment our main goal was to determine if our measures
(presented in Tables 1 and 5 and in Section 2.4) would yield consistent
results with a new study population and different study tasks. We have
350
Design Studies Vol 26 No. 4 July 2005
seen consistent results for the following measures: quality of solution,
number of transitions, progression, and time spent in the decision step.
However, we have also observed that the time and transition rate
measures do not yield consistent results. Rather the findings suggest that
the amount of time spent solving the problem is an important factor for
shorter problems and that transition rate is important for a lengthier
problem. Finally, number of alternative solutions seems to be an
important though slightly inconsistent measure. For the Playground
Study, we expected that the seniors would consider more alternative
solutions than the freshmen, but instead observed only that the seniors
who considered more alternatives earned higher quality scores. For the
Senior Follow-Up Study the opposite is truedthere is no correlation
with quality but there is a freshmanesenior difference.
The results from the Senior Follow-Up Study also show that some
freshmen ‘get stuck’ defining the problem rather than moving on to the
developing alternative solutions and project realization stages. However, the results also show that generally the freshmen and seniors
allocate their time amongst the design steps in a similar manner. Both
the freshmen and seniors spent a larger percent of time in problem
definition during the Senior Follow-Up Study than the Playground
study. This is not too surprising since reading the problem statement is
included in the problem definition step. Students spent approximately
1 min reading the problem statement for each of the ping pong and street
crossing problems and approximately 2 min reading the problem
statement for the playground problem. Spending 1 min reading the
problem statement may amount to 10% of the total design time for the
10 min ping pong problem while spending 2 min on the playground
problem statement would account for only 1.7% of the total design time
for the 120 min playground problem.
4.2 New findings
Similar to the results for the across-subject differences, the withinsubject participants did exhibit more sophisticated design behaviour as
seniors than as freshmen. For the most part, these design process growth
findings suggest that at least most of the findings from the Playground
Study can be replicated not only with a new participant pool and with
different study tasks but also with a within-subject participant pool.
Additionally, the within-subject participants replicated the solution
quality findingdthe participants earned higher quality-of-solution
scores as seniors than as freshmen. The results also indicate that
individual students exhibit changes in design behaviour in a variety of
manners. For example, some students may show a change in the way
Comparing freshman and senior engineering design processes
351
that they allocate their time amongst design steps while others simply
show a change in the amount of time and effort that they invest in
solving design problems. The within-subject findings are discussed in
greater detail in Turns et al. (2002).
The results of this study also suggest further differences between
freshman and senior design behaviour beyond those of the original
study. One of the most notable differences between the studies is the time
measure. While there was no significant difference between freshman
time and senior time for the Playground Study, there was a significant
difference for the Senior Follow-Up study. In light of this, it makes sense
that there are more differences between freshmen and seniors in terms of
amounts of time spent in individual design activities as well, especially
since total time and amount of time in individual activities are generally
strongly correlated.
Another difference between the studies is the percent of time in problem
definition. In particular, problem definition seemed to dominate the
design processes of the freshmen in the Senior Follow-Up Study while it
occupied a very small percent of the freshmen design time in the
Playground Study. A possible explanation may be that there is a critical
amount of time to be spent in problem definition. That is, perhaps
students need to spend at least 1e2 min reading the problem statement
and 1e2 min developing an understanding of the task. For the
Playground Study, 2e4 min would take up a very small percent of
a participant’s 2 h spent on the problem. In contrast, that same 2e4 min
would account for a considerable percent of a participant’s 10 min spent
on one of the problems on the Senior Follow-Up Study.
Within the Senior Follow-Up Study, students’ design behaviour differed
by problem. While the task from the Playground Study differed from
the tasks in the Senior Follow-Up Study in length, the problems within
the Senior Follow-Up Study differed in terms of the familiarity of the
context (the street crossing problem had a more familiar context for the
students) and in terms of instructions (the street crossing problem
included instructions to estimate costs and benefits). It is important to
note the across-problem differences in the Senior Follow-Up Study for
amount of time, number of transitions, transition rate, evaluation
(which is part of progression) and number of alternative solutions.
These are also measures where seniors exhibited significantly different
behaviour from the freshmen. Implications for these across-problem
differences are further discussed in the next section.
352
Design Studies Vol 26 No. 4 July 2005
We can note at this time, however, that a possible explanation for the
instances in which the Senior Follow-Up Study did not replicate the
findings of the Playground Study is the differences in study tasks.
Specifically, the differences in study results (in the measures of time,
transition rate and number of alternative solutions) may be due to the
following differences in problem type: short versus long, single-problem
versus multiple problems, differing levels of task ambiguity, context
specificity and complexity.
5 Conclusions
5.1 Overall findings
The empirical findings from this study show that most of the results
from the previous Playground Study can be replicated. Senior design
behaviour tended to be more sophisticated than freshman design
behaviour and seniors tended to produce higher quality design solutions. Additionally, this study includes data for changes in design
behaviour for individual participants. While the within-subject participants as a group showed differences between freshman and senior
participation that were similar to the findings for the general population
of freshmen and seniors, some participants did not exhibit growth on all
measures. Individual student design behaviour also varied by problem
(Cardella et al., 2002; Turns et al., 2002).
Finally, this study confirms the speculation that the choice of task is very
important. Within this study, design behaviour varied by problem. Both
the freshmen and seniors spent a smaller percent of time in problem
scoping and a larger percent of time in developing alternative solutions
(as well as a larger amount and percent of time in feasibility analysis and
evaluation) on the street crossing problem than on the ping pong
problem. Additionally, across-subject design behaviour varied more on
the street crossing problem, as evidenced by significant differences
between freshmen and seniors on more measures for the street crossing
problem than for the ping pong problem.
5.2 Implications
There are many differences between freshman and senior design
behaviour that are consistent across design problems and for a large
group of participants. It seems that freshmen are not allocating much
time to the problem scoping and developing alternative solutions stages;
this presents an opportunity for further research into the appropriate
teaching strategies to address these challenges. Both freshmen and
seniors spend very little time in the project realization stage as well as the
Comparing freshman and senior engineering design processes
353
evaluation step. While it is clear that students’ skills have improved over
the course of their engineering education, students may need additional
support for progressing into the later steps of the design process.
The across-problem findings suggest that a variety of complex problems
with varying task environments (Goel and Pirolli, 1992) can allow
students to practice and improve different skills. A potential implication
for this finding is that students should be exposed to a wide variety of
problems throughout their undergraduate education to develop these
skills (Woods, 1995; Cardella et al., 2002). Given that the students spent
very little time in evaluation, but the students did tend to engage in
evaluation behaviour more on the street crossing problem than on the
ping pong problem, it seems that students may need more exposure to
problems similar to the street crossing problem that elicit evaluation
behaviour. These problems could include the elements of the street
crossing problem that may have acted as the catalyst for evaluation:
context that is familiar to the students, instructions to estimate the costs
and benefits associated with the design and a ‘real-world’ setting.
Additionally, for the experimenter, the implication is that task matters
when setting up the experiment. The combination of the findings for
across-problem differences between the ping pong problem and the street
crossing problem with the findings for the differences between the Senior
Follow-Up Study and the Playground Study suggests that the
experimenter must carefully consider the length of the task, the number
of problems the participant needs to solve, the context specificity of the
task and the complexity of the task.
Acknowledgements
This research was made possible in part by National Science
Foundation grants DUE-9254271, RED-9358516, DGE-9714459,
EEC-9872498 and REC-0125547 as well as support from the GE Fund.
We would like to gratefully acknowledge the students who participated
in the study, and also thank the students and staff from two universities
who helped us to collect, transcribe, code and analyze the data.
University of Pittsburgh: Heather Nachtmann and Justin Chimka who
ran the data collection and initial analysis of the Senior Follow-Up
Study, as well as Karen Bursic, Mary Besterfield-Sacre, Rona
Colassimo, Stefanie Lozito, Carie Mullins, Jamari Atkinson, Georgette
Diab, Anthony Horton, Jennifer Kreke, Terra Mitchell, Pamela Moore,
Jill Nagel, Jason Saleem, Gwen Staples, Nick Tettey, Chris Yarsky, Lisa
Younger and Maryland Vick. From the University of Washington:
Jacob Burghardt, Louise Cheung, Jennifer Chin, Julie Christianson,
354
Design Studies Vol 26 No. 4 July 2005
Yimin Chen, Ashley Lam, Jana Littleton, Ian Louie, Alison Schwerzler,
Cathie Scott, Roy Sunarso, Rober Tai, Jennifer Temple, and Bettina
Vuong. Finally, we would like to thank Theresa Barker and Susan
Mosborg for their careful reading of the paper.
References
ABET (2000) Engineering criteria 2000: criteria for accrediting programs in
engineering in the United States, 3rd edn 2000 See. http://www.abet.org/
downloads.htm download EAC Criteria for 2000-01 (Includes EC2000)
pp 32e34
Adams, R (2001) Cognitive processes in iterative design behavior, dissertation, University of Washington
Adams, R, Turns, J and Atman, C J (2003) Educating effective engineering
designers: the role of reflective practice Design Studies Vol 24 No 3
pp 275e294
Atman, C J and Bursic, K M (1996) Teaching engineering design: can
reading a textbook make a difference? Research in Engineering Design Vol 7
No 7 pp 240e250
Atman, C J and Bursic, K M (1998) Verbal protocol analysis as a method to
document engineering student design processes Journal of Engineering
Education Vol 88 No 2 pp 121e132
Atman, C J, Chimka, J R, Bursic, K M and Nachtmann, H N (1999) A
comparison of freshman and senior engineering design processes Design
Studies Vol 20 No 2 pp 131e152
Atman, C J and Turns, J (2001) Studying engineering design learning: four
verbal protocol studies in C M Eastman, W M McCracken and W
C Newstetter (eds) Design knowing and learning: cognition in design
education, Elsevier Science Ltd, Oxford, UK pp 37e60
Ball, L J and Ormerod, T C (1995) Structured and opportunistic processing
in design: a critical discussion International Journal of HumaneComputer
Studies Vol 43 No 1 pp 131e151
Bogusch, L L, Turns, J and Atman, C J (2000) Engineering design factors:
what do students think is important? Frontiers in Education Conference
Proceedings
Brereton, M F, Cannon, D M, Mabogunje, A and Leifer, L J (1996)
Collaboration in team designs: how social interaction shapes the product in
H Christiaans, N Cross and K Dorst (eds) Analysing design activity, John
Wiley & Sons, Chichester, UK
Brockman, J B (1996) Evaluation of student design process Frontiers in
Education Conference Proceedings
Cardella, M E, Atman, C J, Adams, R and Turns, J (2002) Engineering
student design processes: looking at evaluation practices across problems
American Society of Engineering Education Conference Proceedings
Cross, N (1999) Natural intelligence in design Design Studies Vol 20 No 1
pp 25e39
Cross, N (2001) Design cognition: results from protocol and other empirical
studies of design in C M Eastman, W M McCracken and W C Newstetter
(eds) Design knowing and learning: cognition in design education, Elsevier
Science Ltd, Oxford, UK
Comparing freshman and senior engineering design processes
355
Cross, N and Clayburn Cross, A (1998) Expertise in engineering design
Research in Engineering Design Vol 10 No 3 pp 141e149
ECSEL (1993) Introduction to design engineering active learning modules:
a catalogue ECSEL National Office, Washington, D.C.
Ennis, C W and Gyeszly, S W (1991) Protocol analysis of the engineering
systems design process Research in Engineering Design Vol 3 No 1
pp 15e22
Ericsson, K A and Simon, H A (1993) The MIT Press, Cambridge,
Massachusetts
Ericsson, K A and Smith, J (1991) Prospects and limits of the empirical
study of expertise: an introduction in K A Ericsson and J Smith (eds)
Toward a general theory of expertise, Cambridge University Press,
New York pp 1e38
Gero, J S and McNeill, T (1998) An approach to the analysis of design
protocols Design Studies Vol 19 No 1 pp 21e61
Goel, V and Pirolli, P (1992) The structure of design problem spaces
Cognitive Science Vol 16 No 3 pp 395e429
Lewis, W P and Bonollo (2002) An analysis of professional skills in design:
implications for education in research Design Studies Vol 23 No 4
pp 385e406
Louie, I (2001) Analysis of senior follow-up data: designing a playground
CELT Technical Report CELT-01-05, Center for Engineering Learning
and Teaching, University of Washington, Seattle, WA
Moore, P L, Atman, C J, Bursic, K M, Shuman, L J and Gottfried, B S
(1995) Do freshmen design texts adequately define the engineering design
process? American Society for Engineering Education Conference Proceedings 164e172
Mullins, C A, Atman, C J and Shuman, L J (1999) Freshman engineers’
performance when solving design problems IEEE Transactions on
Education Vol 42 No 4 pp 281e287
Newstetter, W C and McCracken, W M (2001) Novice conceptions of
design: implications for the design of learning environments in
C M Eastman, W M McCracken and W C Newstetter (eds) Design knowing
and learning: cognition in design education, Elsevier Science Ltd,
Oxford, UK
Pahl, G, Badke-Schaub, P and Frankenberger, E (1999) Résumé of 12 years
interdisciplinary empirical studies of engineering design in Germany Design
Studies Vol 20 No 5 pp 481e494
Radcliffe, D F and Lee, T Y (1989) Design methods used by undergraduate
engineering students Design Studies Vol 10 No 4 pp 199e209
Rhone, E, Turns, J, Atman C J, Adams, R, Chen, Y and Bogusch, L (2001)
Analysis of senior follow-up data: the Midwest floods problem-addressing
redundancies in coding CELT Technical Report CELT-01-03, Center
for Engineering Learning and Teaching, University of Washington,
Seattle, WA
Römer, A, Leinart, S and Sachse, P (2000) External support of problem
analysis in design problem solving Research in Engineering Design Vol 12
No 3 pp 144e151
Sanderson, P M, Scott, J J P, Johnston, T, Mainzer, J, Watanabe, L M and
James, J M (1994) MacSHAPA and the enterprise of Exploratory
356
Design Studies Vol 26 No. 4 July 2005
Sequential Data Analysis (ESDA) International Journal of HumanComputer Studies Vol 41 No 5 pp 633e681
Smith, R P and Tjandra, P (1998) Experimental observation of iteration in
engineering design Research in Engineering Design Vol 10 No 3 pp 107e117
Turns, J, Atman, C J, Cardella, M and Adams, R (2002) Do we see withinsubject change? Four cases of engineering student design processes
presented at Common Grounddthe Design Research Society Conference,
Runnymede, United Kingdom
Ullman, D G, Dietterich, T G and Stauffer, L A (1988) A model of the
mechanical design process based on empirical data AI EDAM Vol 2 No 1
pp 33e52
Woods, D R (1995) Problem-based learning: helping your students gain the
most from PBL D.R. Woods Publishing, Waterdown, ON
Comparing freshman and senior engineering design processes
View publication stats
357