Tracking Design Knowledge in Engineering
Student Projects around Course Milestones
Sharad Oberoi and Susan Finger
Carnegie Mellon University

• Goal
– Develop tools that allow instructors to monitor the progress of project-based courses and assess student interactions outside the classroom

• Assertions
– Most learning in design classes takes place in team meetings and in individual activities undertaken to help meet team goals
– Prior research has shown that noun phrases can be used as surrogates for design concepts
– For successful design teams, the number of distinct noun phrases expands at the initial stages of the project and contracted as the project progresses
Mabogunje, A., Measuring Conceptual Design Performance in Mechanical Engineering: A Question
Based Approach, Doctoral Dissertation, Stanford, CA: Stanford University, 1997.

• Setting
– Engineering design capstone course
– Ongoing project to understand collaborative learning by student design teams

• Characteristics
– Students bring together knowledge from different sources:
• collaborate among themselves,
• share design knowledge
• negotiate with each other, faculty members and the client, in order to create engineering artifacts.
– Students reuse previous knowledge and create new knowledge within the context of the problem.

• For 7 years, RPCS has used the Kiva for team collaboration
– Light-weight collaboration tool
– Combines functions of e-mail and bboards
– Widely accepted and liked by student teams; it feels likes chat and meets their needs
– Each year’s Kiva has hundreds of threads and thousands of posts and files
 We have 7 years of data of all the team conversations and files that would normally go through email or chat

• Takes advantage of students’ willingness to send email, use IM, post on newsgroups, send text messages
• Allows students to cross-post and to post to teams in which they are not members
• Design goal: Create an interface that students perceive to be equivalent to their preferred communication modes; that is:

• RPCS: Rapid prototyping of computer systems class of Spring 2008
– Interdisciplinary, capstone design course
– Class of 25 students taught by 2 instructors, 1 TA
– Project on the design of a virtual coach called
Guru for new power wheelchair users
– The aim was to monitor wheelchair users and assist the clinician in assessing if any revisions were required in the prescriptions
– Major teams in the class were: Chair, Clinician,
Infrastructure and Sensors

• Phase 1: Requirements exploration and definition, benchmarking, and a rough conceptual prototype
• Phase 2: Creation of a functional prototype that satisfies most of the feasibility constraints
• Phase 3: System integration and more design iterations to make final refinements

• Discussion posts and attachments of the class were converted to text
• Noun phrases were extracted using the
Stanford part-of-speech tagger
• Number of distinct noun phrases was counted for each group

• Advantages of using noun phrases
– Noun phrases act as an objective design document metric for predicting design process performance
– The pattern of distinct noun phrases at every stage of a design artifact can reveal insights about the state of the project

• Three analyses were done to attribute nouns to students
– First analysis: Includes only the documentation posted to the four main teams in the project
– Second analysis: Includes only the documentation made by members of the teams regardless of the thread in which it was posted
– Third analysis: Compares the time-based trends of noun phrases in the editing/presentation and design categories within the documentation
• Transformation of design concepts across phases was tracked

• Extracts noun phrases only from the documentation posted to the four main teams in the project
• The peaks decrease with time
• Instructor documents included in the analysis, so design contributions made by the individual teams are not adequately reflected in their vocabulary

Mid-term break
Phase-1 Phase-2 Phase-3
Aggregated noun phrase counts by week

Distinct noun phrase counts by week

• The activities of the team members were tracked throughout the semester, instead of analyzing only the material posted in the teams’ threads
• This process attributes only the noun phrases used by members of a team to the team’s knowledge
• Peaks of distinct noun phrases expanded at the initial stages of the project and contracted as the project progressed

Aggregated noun phrase counts by week

Distinct noun phrase counts by week

• The editing/presentation and design activities were tracked separately to understand the externalization done by students around project milestones
• The noun phrases in the editing/presentation documents vis-à-vis the design documents and discussions were compared
• Student design activities were found to not necessarily follow the class deadlines, unlike those including editing of the project reports or presentations

Noun phrase counts in design activities by week

Noun phrase counts in editing tasks by week

• Some design concepts are introduced by teams in project-based courses but wane with time, i.e., they are either abandoned or transformed into a related concept in the final design
• Reflects how student understanding about the artifact is evolving through the project

Noun phrases used by the Sensors Team to refer to the tilt of the wheelchair
Teams Phase 1
Sensors different tilt sensors, tilting angle
Phase 2 accelerometer, general tilt, tilt measurements, tilt range, tilt recline feet elevation, tilt recommendation, tilt sensors, tilt, wheelchair tilting angle
Phase 3 accelerometer, nac tilt recline, pressure tilt, tilt data, tilt measurements, tilt position, tilt range, tilt recommendation, tilt sensors, wheelchair tilting angle

• The paradigm of using noun phrases from student documentation to assess design team progress has been accepted for the past two decades.
• This research takes the paradigm forward and shows how noun phrases in a design project can help to measure how the class has performed from the perspective of prior research models of expansion and contraction of vocabulary for successful projects.
• The paper also shows what activities the students are engaged in outside the instructors’ purview

• When the instructors have access to such information in real-time, they can actively monitor the progress being made by the teams
• They can also intervene in a timely manner in case they detect a potential teachable moment.