DesignWebs to Support Engineering
Design Student Projects
Sharad V. Oberoi
2
Overview
• Motivation Research Contributions
Related
•• An
automatedresearch
mechanism to visualize
engineering design project
Motivation
Future
Work
documentation
Research
•• How
do studentstestbed
learn in engineering
design project teams?
Related
Research
•• Analysis
of
graphical
and
verbal
communication
A surrogate measure to track the emergence of the shared solution
•
Research
questions
• Collaborative
How
can additional
information
be provided
to instructors to assess
learning
innoun
design
Research
Testbed
•• Real-time
monitoring
of
phrase
usage
Astudent
research
test bed to validate different prior research findings
learning?
•
Research
findings
•
Organizational
memory
Rapid
Prototyping
of Computer
Class 2008
Research
Questions
• Functional and operational
issuesSystems
Contributions
•• Kiva
Computational
linguistics
How
can the state
of the artifact
be summarized
and presented visually to
Research
Findings
students and instructors from the project documents and Kiva
•
Future
work
Information
structuringcan
andbevisualization
discussions?
•• Design
documentation
visualized asapproaches
concept-map based graphs
Whatphrases
objective
can measures
be used tofor
assess
the student
design
•• Noun
actmeasures
as surrogate
tracking
design team
dynamics
process and gain insights into student design collaboration?
• DesignWebs can act as research platform to validate prior research
• How can prior research in design learning be validated?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
3
Motivation
• How do students learn from each other in engineering
design project teams?
• How can additional information be provided to
instructors to assess student learning?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
4
Students in Design Projects
•
•
•
•
Create most of the knowledge outside the classroom
Have little experience in team-work or in design
Consider final product as the most important take-away
Have little or no support infrastructure for information
management
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
5
Instructors of Design Projects
• Intend to assess student learning outcomes, but have
more ready access to product outcomes
• Often have to rely on the self-reported functioning of the
teams to assess their progress
• Cannot always detect instances when students deviate
from objectives
• Need a mechanism to monitor student progress in realtime
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
6
Research Issues
• Assessing evidence of student learning requires
monitoring of student activities
• Unobtrusive approaches are necessary:
– Students alter behavior in front of instructors
– Embedded observations are tedious
• Evidence of student collaboration process exists, but is
difficult to summarize and use
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
7
Overview
•
•
•
••
••
••
••
Motivation
Related research
Research testbedRelated Research
Researchlearning
questions
Collaborative
in design
Researchmemory
findings
Organizational
Contributions
Computational
linguistics
Future work
Information
structuring and visualization approaches
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
8
Collaborative Learning in Design
• Collaborative learning allows students to hone their
communication and negotiation skills (Dillenbourg 1999)
• When students with different strengths work together,
they solve problems that would be beyond their reach if
they were working alone (Vygotsky 1978)
• The depth of student explanations is correlated with how
much students learn (Webb et al. 2002)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
9
Organizational Memory
• Reuse of design information in organizations often
requires designers to recreate the rationale
(Subrahmanian et al. 1997)
• Easy access to archives can help future teams be
successful with lower transactional costs (Roth et al.
1998)
• Organizational memory systems are usually domain
specific and cannot be readily adpated in project classes
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
10
Automatic Topic Segmentation
1. Lexical cohesion Models:
–
–
Lexical co-occurrence of thematically-related terms
indicates continuity in topic
Introduction of new vocabulary refers to a new topic
2. Content-oriented Models:
–
12/13/2011
Combine lexical cohesion with other indicators of topic
shift such as cue phrases using probabilistic models
(Reynar 1998)
Sharad V. Oberoi: PhD Final Public Oral Exam
11
Latent Dirichlet Allocation (LDA)
• Widely used to cluster document fragments and
automatically assign them topic labels (Blei et al. 2003)
• The analysis is done at the semantic level by considering
probabilities for each of the words occurring in the
corpus
• The context of the word can be examined, making word
disambiguation possible
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
12
Co-word Analysis
• Co-word analysis reduces a space of descriptors to a set
of network graphs with the strongest associations
between descriptors (Coulter et al. 1998)
• Co-word graphs construct multiple networks that
highlight associations between keywords (Coulter et al.
1998)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
13
Information Structuring
• As student expertise increases, the corresponding
concept interrelatedness in concept maps resembles
tightly integrated structures (Royer et al. 1993)
• Scientometrics identifies emerging research areas and
the evolution of research areas (Callon et al. 1991; van
Raan 1992)
• Library sciences have done research on organizing
information stored in books and journals and providing
structures for locating relevant information
(Bhattacharya et al. 1998)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
14
Information Visualization
• Domain visualization “aims to reveal realms of scientific
communication as reflected through scientific literature
and citation paths interwoven by individual scientists in
their publications” (Börner et al. 2003)
• Longitudinal maps have been used to detect the evolving
nature of scientific fields by scientometrists (Garfield
1994)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
15
Overview
•
•
•
•
•
•
•
•
•
Motivation
Related research
Research testbed
Research questions
Research Testbed
Research findings
Rapid Prototyping of Computer Systems Class 2008
Contributions
Kiva
Future work
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
16
Research Testbed
• Rapid Prototyping of Computer Systems (RPCS) 2008
• The intermediate and final deliverables and
conversations captured through the Kiva
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
17
RPCS 2008
• 25 students; 3 phases; 4 major teams
• Taught by two instructors and one teaching assistant
• Goal: To design a virtual coach called Guru for new power
wheelchair users
• Clients are actively involved with specific teams,
providing relevant expertise, project advice and any
clarifications that are needed.
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
18
Kiva
• Light-weight collaboration tool
• Combines functions of e-mail and bboards
• Widely accepted and liked by student teams; it feels likes
chat and meets their needs
• Each year’s Kiva has hundreds of threads and thousands
of posts and files
• The posts are publicly visible
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
19
Overview
•
•
•
•
•
1.•
•
Motivation
Related research
Research testbed
Research questions
Research findingsResearch Questions
Contributions
Can concept maps of a design artifact be automatically extracted from the group
conversation and documents for ongoing and archived project corpora?
Future
work
2.
Can the noun phrases used by students in a design project be used as a surrogate
measure for assessing design team dynamics?
3.
Can DesignWebs be used as a research framework to facilitate the validation of prior
research findings?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
20
Research Questions
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
21
Research Questions
1. Can the concept maps of a design artifact be
automatically extracted from the group
conversation and documents for ongoing and
archived project corpora?
2. Can the noun phrases used by students in a design
project be used as a surrogate measure for
assessing design team dynamics?
3. Can DesignWebs be used as a research framework
to facilitate the validation of prior research findings
about the functioning and assessment of design
project teams?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
22
DesignWebs
• This question addresses the problem of information
management in engineering design projects
• Evidence of student design collaboration is mined
through computational linguistics and machine learning
• A concept-map based graphical representation is created
that can be used to navigate the project corpus
• DesignWebs assist as navigation aids in contextualizing
and viewing project archives by structuring the design
information as graphs of related entities
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
23
Implementation of DesignWebs
• Extract the LDA model and apply hierarchical clustering
to extract a hierarchical topic structure
• Collapsing of nodes
• Calculate topic-term connections
• Vector space models for all documents in a corpus are
combined to form a word-by-document matrix
• Retrieval through a combination of navigation and query
by the student
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
DesignWebs as Representation of
Design Knowledge
• DesignWebs reflect the shared structure of the artifact
from design documentation
• DesignWebs assimilate information from multiple
sources that would be beyond the comprehension of any
single member of the project
• DesignWebs have the potential to reveal missing links
between concepts or show seemingly unexpected
connections
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
24
25
Research Questions
1. Can the concept maps of a design artifact be
automatically extracted from the group
conversation and documents for ongoing and
archived project corpora?
2. Can the noun phrases used by students in a design
project be used as a surrogate measure for
assessing design team dynamics?
3. Can DesignWebs be used as a research framework
to facilitate the validation of prior research findings
about the functioning and assessment of design
project teams?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
26
Research Question 2
• Presents noun phrases as a surrogate measure for design
team dynamics
• Shows that tracking the noun phrases over time and
across team boundaries can reveal insights about teams
collaboration
• Shows that shared vocabulary mirrors the work-flow
between teams
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
27
Research Questions
1. Can the concept maps of a design artifact be
automatically extracted from the group conversation
and documents for ongoing and archived project
corpora?
2. Can the noun phrases used by students in a design
project be used as a surrogate measure for assessing
design team dynamics?
a)
b)
12/13/2011
Can the noun phrase counts in the design vocabulary of
student design teams at different project milestones be used
to assess their design process?
Can tracking the noun phrases, as they are introduced into the
design vocabulary and shared across team boundaries over
time, reveal insights into design team collaboration?
Sharad V. Oberoi: PhD Final Public Oral Exam
28
Noun Phrase Trends over Time
• Noun phrases have been used as a predictor of
design performance (Mabogunje 1997)
• Noun phrases were extracted using Stanford
part-of-speech tagger
• Challenges with RPCS 2008 data:
1. Issues with task-specific groups
2. Shared authorship of documents
3. Role of references in structuring project knowledge
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
29
First Analysis
• Only the Kiva documentation posted in the
Kiva groups of the four main teams in the
project were used
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
30
Second Analysis
• Individual team members were followed
throughout RPCS 2008 Kiva
• All postings and files were attributed to the
parent teams, regardless of the group in which
they were posted
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
31
Third Analysis
• Time-based trends of noun phrases in the
external references used by students were
compared with the documents and
discussions they had with each other
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
32
Research Questions
1. Can the concept maps of a design artifact be
automatically extracted from the group conversation
and documents for ongoing and archived project
corpora?
2. Can the noun phrases used by students in a design
project be used as a surrogate measure for assessing
design team dynamics?
a)
b)
12/13/2011
Can the noun phrase counts in the design vocabulary of
student design teams at different project milestones be used
to assess their design process?
Can tracking noun phrases, as they are introduced into the
design vocabulary and shared across team boundaries over
time, reveal insights into design team collaboration?
Sharad V. Oberoi: PhD Final Public Oral Exam
33
Connectivity of Key Ideas across Teams
• Noun phrases, used as design concepts, can
be classified as:
1.
2.
3.
4.
12/13/2011
Self-looping concepts
Shared design concepts
Transient design concepts
Design escape concepts
Sharad V. Oberoi: PhD Final Public Oral Exam
34
Self-looping Concepts
• Some design concepts are introduced by a
team, used only by them alone over time and
become a part of the final artifact design
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
35
Shared Design Concepts
• Some design concepts are introduced by one
team, get picked up by other teams, and are
adopted in the shared vocabulary.
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
36
Transient Design Concepts
• Some design concepts are introduced by a
team but disappear from its vocabulary with
time
• They either represent abandoned design
approaches, or are transformed into a related
concept in the final design
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
37
Design Escape Concepts
• Some design concepts are introduced by one
team and are important for the tasks assigned
to other teams, but fail to attain ownership
• Transient design concepts and design escape
concepts cannot be automatically
differentiated
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
Noun Phrases mapped on System
Architecture Diagram
• When design vocabulary is mapped on the system
architecture diagram, it can assist the instructor in
assessing team activities
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
38
39
System Architecture Diagram
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
Reflection of the System Architecture
in the Design Team Vocabulary
Self-looping concepts
Shared design concepts
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
40
Student Activities reflected in Timebased Transformation of DesignWebs
Sensors Team in Phase 2
13
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
41
42
Review of Tracking Noun Phrases
• Tracking the design vocabulary reveals:
–
–
–
–
–
Expansion and contraction of vocabulary over time
Insights into team collaboration
Missed or abandoned concepts
Work-flow between teams
Current state of the design
• Instructors can observe the changes in a team’s design
knowledge unobtrusively by monitoring its design
vocabulary
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
43
Research Questions
1. Can the concept maps of a design artifact be
automatically extracted from the group
conversation and documents for ongoing and
archived project corpora?
2. Can the noun phrases used by students in a design
project be used as a surrogate measure for
assessing design team dynamics?
3. Can DesignWebs be used as a research framework
to facilitate the validation of prior research findings
about the functioning and assessment of design
project teams?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
44
DesignWebs as a Research Testbed
• Three diverse research studies are examined:
1. Can using the semantic coherence of student
communications in team-based project courses act
as an indicator of the design process? (Song et al.
2003)
2. Can applying metrics on the “communication
artifacts” generated by computer supported
collaborative tools can provide insights into the
design process that created them? (Dutoit 1996)
3. Can various individual and group work processes
when made more explicit to instructors assist them
in carrying out the assessment processes? (Gweon
2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
45
DesignWebs as a Research Testbed
• Three diverse research studies are examined:
1. Can using the semantic coherence of student
communications in team-based project courses act
as an indicator of the design process? (Song et al.
2003)
2. Can applying metrics on the “communication
artifacts” generated by computer supported
collaborative tools can provide insights into the
design process that created them? (Dutoit 1996)
3. Can various individual and group work processes
when made more explicit to instructors assist them
in carrying out the assessment processes? (Gweon
2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
Semantic Coherence of Student
Communications
• Semantic coherence measures the similarity with which
texts in a project corpus represent the variation in voices
referred to in forming the concept
• A text analysis module called AgoraParse was used to
extract communication metrics
• A language analysis module (AgoraProbe) was used
to extract the semantic coherence metrics using LSA
• A graph of the semantic coherence was created using
MATLAB
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
46
47
Semantic Coherence of RPCS 2008
Semantic
coherence
trendtrend
for RPCS
2008 2008
(Kiva discussions
and documents)
Semantic
coherence
for RPCS
(Kiva discussions
only)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
48
Research Question 3
• Three diverse research studies are examined:
1. Can using the semantic coherence of student
communications in team-based project courses act
as an indicator of the design process? (Song et al.
2003)
2. Can applying metrics on the “communication
artifacts” generated by computer supported
collaborative tools can provide insights into the
design process that created them? (Dutoit 1996)
3. Can various individual and group work processes
when made more explicit to instructors assist them
in carrying out the assessment processes? (Gweon
2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
Communication Metrics to provide
Insights into the Design Process
• Bruegge et al. (1997) used quantitative measures of bboard traffic as surrogates for inter-and intra-team
communication metrics
• A software product can be measured through a number
of measures such as code length, code complexity
• For comparability of results across projects, a formative
approach was used
• Two structural equation models were employed for
testing hypotheses on multiple projects
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
49
50
Research Question 3
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
51
Research Question 3
• Three diverse research studies are examined to
demonstrate the wide applicability of DesignWebs as a
research platform:
1. Can using the semantic coherence of student
communications in team-based project courses act as an
indicator of the design process? (Song et al. 2003)
2. Can applying metrics on the “communication artifacts”
generated by computer supported collaborative tools can
provide insights into the design process that created
them? (Dutoit 1996)
3. Can various individual and group work processes when
made more explicit to instructors assist them in carrying
out the assessment processes? (Gweon 2008)
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
52
Process Assessment of Design Teams
• Gweon (2008) used the mixed methods approach to
support instructors in identifying difficulties in project
teams
• Interviewed instructors to identify the group processes
that instructors observe and desire in evaluating group
work
• Five process assessment categories were subsequently
identified
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
53
Personal Goal Setting
• Students in RPCS 2008 were required to submit a worklog every week
• Students answered the following questions:
1. What did you accomplish this week?
2. What problems did you encounter this week?
3. What do you plan to work on next week?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
54
Personal Progress
• Estimated in RPCS 2008 from student work logs
• Involves subjective evaluation
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
55
Participation
• Tracks whether students actively attended
team meetings or regularly posted on the Kiva
• Cannot distinguish between diligent students
and fakes
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
56
Overview
• Motivation
Research Contributions
Related
•• An
automatedresearch
mechanism to visualize engineering design project
documentation
• Research testbed
• A surrogate measure to track the emergence of the shared solution
• Research questions
• A research test bed to validate different prior research findings
• Research findings
• Contributions
• Future work
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
57
Research Contributions
• An automated mechanism to integrate, visualize
and navigate the evolving content of documents
created by teams in engineering design projects
• A visual interface that summarizes the team
communications to track the emergence of the
shared solution using noun phrases as a
surrogate measure
• A research test bed to validate different prior
research findings about design project teams
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
58
Future Work
• Analysis of graphical and verbal
communication
• Real-time monitoring of noun phrase usage
• Functional and operational issues
– Version mapping
– Summarization and aggregation
– Allowing complex queries
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
59
Questions?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam
60
Research Focus
1. How can the student activities, as reflected in
student project documents and discussions, be
monitored by instructors actively and
unobtrusively?
2. How can students visualize the state of the artifact
as reflected in the project documents?
3. What objective measures can be used to assess the
student design process and gain insights into
student design collaboration?
12/13/2011
Sharad V. Oberoi: PhD Final Public Oral Exam