A Case Study in Acquiring Robotic Capabilities
Inherently a System of Systems Activity?
DeWitt T. Latimer IV, dlatimer@usc.edu
USC Center for Systems & Software Engineering
http://robotics.usc.edu/~dlatimer
http://csse.usc.edu
USC-CSSE Convocation 25 Oct 2006
The views expressed in this presentation are those of the author and
do not reflect the official policy or position of the United States Air
Force, Department of Defense, or the U.S. Government.
Outline
•
•
•
•
•
•
Overview
Case Study Environment
Acquisition Program Goals
Surface Assessment Robot
Root Cause of Acquisition Failure
Summary
Overview
Overview
• Describe the acquisition environment
• Examine the acquisition goals for a straightforward robotic system
• Describe the system as it was developed
• Identify a root cause of the acquisition failure at
the transition to operations stage
• Discuss if there are inherent system of systems
aspects of straight-forward robotic systems and
need to generate case studies for application of
knowledge
Case Study Environment
Case Study Environment
• Based on completed robot built by Carnegie
Mellon University graduate students and staff
• Acquiring commercial company was very familiar
with subcontracting, sponsoring research projects,
and a had long history of successful acquisitions
with CMU
Case Study Environment
Diffuse Management
The business units
involved in client
organization
No senior level
involvement, business
units essentially
autonomous
QA office provides
mechanism for
handling deviations
GMs manage delivery of
roadways and utilize
inspectors
Quality
Assurance
Office
(TQM/6Sigma
Champion)
Individual
Project
General
Managers
VP R&D
(Acquirer)
Acquisition Program Goals
Acquisition Program Goals
• Proposed system to:
– Assess suitability of roadway subsystem by
identifying deviations from required profile
– Automate measurement function performed by
aging senior field engineer
– Ensure Return on Investment (ROI) achieved in
single roadway construction project
– Ensure system can be delivered in time to utilize
on upcoming project
– Provide system that increases the technical
prestige of the company in delivery of systems
that include roadway surfaces
– Continue successful relationship between
company and CMU
Surface Assessment Robot
System History
• Developed by CMU team (I was a member)
• Met all technical requirements
– Autonomously detected and marked high/low
variations of a road exceeding 1/8” in 10 feet
(~1 part per 1000)
• Met all business objectives
– ROI: 100 to 1
– Paid for R&D investment in demo project
• Currently sits on a shelf (in a bay actually)
Surface Assessment Robot
Development Activities
• Requirement to mark all deviations against profile
specification handled by 2-pass autonomous
detector/marking system
– Driven by human operator for safety concerns
Surface Assessment Robot
Development Activities
• Developer engaged significantly with the client's
six-sigma organization to ensure appropriate
deviation handling and recording
• Recast field engineer's role to be to review
collected data to disposition deviations
Desired
Profile
Upper Limit
of Deviation
Lower Limit
of Deviation
Actual
Profile
“Low” Region
of Deviation
Surface Assessment Robot
Development Activities
• Developers noted system requirement for
roadway was in terms of ride quality, not road
roughness
• Proposed quarter-car model to characterize
deviations before marking as either impacting
system requirements or being insignificant
– All deviations to roughness specification would
be automatically catalogued and those not
impacting ride quality would be classified as no
rework needed
– All deviations could be audited and reviewed
Surface Assessment Robot
Development Activities
• Client's staff engineers rejected,
indicating that all engineering
decisions must be made by
humans – field engineer to
classify all deviations
• Development continued and
delivered system on-time
• Robotic system demonstrated
on actual program track with
client and roadway
subcontractor representatives
Surface Assessment Robot
Final Roll-Out
• System marked several orders
of magnitude more deviations
than expected, surprising as
subcontractor was the bestqualified and had a track
record of low/zero defect
roadways
– All deviations were verified by senior engineer,
however the engineer indicated that most
deviations were so minor as to have no system
impacts and would have been ignored – counter
to the QA culture!
Surface Assessment Robot
End Game
• After final system demonstration, system was
transitioned to customer
• No maintenance role for CMU was desired by
either party (this was an upfront requirement)
• We later learned that the system was never
transitioned to regular usage, indeed was not
used on any future project
Root Cause of Acquisition Failure
Candidate Root Causes
(but not the actual ones)
Acquisition Requirements Development (RD)
– All top level requirements were present, roughness was the right
technical requirement for contract enforcement reasons.
Developer Error
– System met and exceeded all technical performance measures,
cost targets, and return on investment parameters
Human-Robot Interaction Design (HRI) factors
– Operators in various interface walk-through events and field tests
were very satisfied
Acquisition Verification or Validation (VER/VAL)
– How to perform VER/VAL on requirements at an acquisition level
if no engineer has experience to indicate what analysis would be
needed? See next slide...
Root Cause of Acquisition Failure
Proposed Root Cause
Constraints did not conceive of the robot acting as
a senior engineer agent (Acquisition TS SG1)
– Acquisition constraints levelled on the project, while
valid, implied that no automated analysis tools could
classify defects in this project.
– Discovered real deviation rate by roadway roughness
being larger than previously believed would have
required the client to entirely rethink both their
roadway standard and their quality management
practice – both of which were outside the scope of the
project's management. This is the essence of a
system of systems issue.
Summary
Summary
This provides a compelling story for the study of
acquisition and acquisition engineering and
provides a concrete example of how an
acquisition organization can fail to acquire “the
right system” without appropriate attention paid
to system of systems issues.
Therefore, observing an organization's acquisition and
employment of robotics provides a microcosm to
explore system of systems acquisition issues.
Summary
Summary
•
•
•
•
•
Overview - Environment
Acquisition Program Goals
Surface Assessment Robot
Root Cause of Acquisition Failure
Summary
Backup
Acronyms
•
•
•
•
•
•
•
•
•
•
CMU – Carnegie Mellon University
GM – General Manager (construction project)
HRI – Human-Robot Interaction
QA – Quality Assurance
RD – Requirements Development Process Area
SG – Specific Goal (of a CMMI process area)
TS – Technical Solution Process Area
USC – University of Southern California
VAL – Validation Process Area
VER – Verification Process Area
References
• “Adapting CMMI for Acquisition Organizations: A
Preliminary Report”, Dodson, et. al., CMU/SEI2006-SR-005
• "Running Surface Assessment Technology
Review", Latimer, et. al., CMU/ICES-04-03-02
• Research project notes, meeting minutes, and
project plan documents from CMU for Roadway
Assessment System
Bio
1Lt DeWitt Latimer IV, USAF is currently assigned as a
PhD Student to the Air Force Institute of Technology and
working towards a PhD in Computer Science at the
University of Southern California. He is advised by Prof
Barry Boehm and Prof Gaurav Sukhatme. His research
focuses on investigating the nature of acquiring
autonomous robotic systems. He earned his MS
degrees in Robotics (2001) and Civil Engineering (2002)
at Carnegie Mellon University. He is a senior member of
the IEEE and a member of ACM, ASCE, and AFCEA and
was awarded the CSDP credentials from the Computer
Society.