Achieving maximum benefits from formal reviews/inspections strategies and case studies
Fran O’Hara
Insight Consulting
114 Granitefield, Dun Laoghaire,
Co. Dublin, Ireland
Phone/fax: +353-1-2854510
E-mail: fran.ohara@insight.ie
Abstract
Formal reviews/inspections are currently a ‘hot topic’ with many references and case studies
detailing, in quantitative terms, the benefits that can be achieved. They have been practiced for
over twenty years and yet there are many organisations not using them to anywhere near their full
potential. Indeed, many examples exist where introducing a review/inspection process and/or
maintaining an effective process have failed. This paper analyses some case studies from the real
world of dubious success (and sometimes even failure) and focuses on some of the key issues
which caused problems. The aim is point out the pitfalls and highlight the key elements in
achieving maximum benefits from introducing and maintaining a formal review/inspection
process. A strategy to deploy the process is presented which encapsulates some of the lessons
learnt.
1. Introduction
What are the key elements of a cost-effective formal process?
There are many implementations of review and inspection from quite informal approaches
through to the most formal and rigorous. ‘Formal’ for the purpose of this paper is a process
which meets or exceeds the requirements of Peer Reviews as defined in the SEI’s Capability
Maturity Model [CMM, 1993]. As such, processes based on Fagan’s Inspection technique
[Fagan, 1976] and the Gilb/Graham [Gilb, 1993] approach most certainly meet and in most cases
greatly exceed these requirements. In simple terms, ‘formal’ therefore implies, at a minimum, the
following:
 the goal of the process is to rigorously identify and fix defects (especially
major/critical defects)
 a secondary goal is to prevent defects
 reviews/inspections are planned explicitly in the project schedule
 review leaders/facilitators and reviewers receive specific training
 a shared understanding of the work products under review is obtained
In terms of the process activities, the following is implied:


an entry check is performed to ensure the material is ready for review
reviewers/inspectors spend time individually checking the work product using
appropriate parent or ‘source’ documents to help find defects. These work products
may be code, design documents, requirements documents, project plans, test plans,
1





procedures, etc. An overview meeting has usually been held to discuss optimum
checking rates, to assign individual reviewers additional checking roles, to focus the
reviewers on criteria/rules to be checked, etc.
a meeting is held to log the issues that have been found. Discussion is kept to a
minimum. New issues are identified.
the author investigates issues and fixes any problems.
data on the review process, the work products, etc. is collected and used for example
to determine the effectiveness of the review/inspection process, the optimum
checking rates, the quality of work product, etc.
if defect prevention is a main goal then some activity relating to defect causal
analysis should take place (see for example the process brainstorming meeting
concept [Gilb, 1993]). If defect prevention is a secondary goal then capturing
improvement suggestions as related issues during logging meetings may suffice.
etc.
Notice the above description advocates some discussion rather than zero discussion at logging
meetings. This is so in order to help achieve the process goal of establishing a shared
understanding of the work product under review (and to train/educate). However, any discussion
should be strictly limited and controlled to facilitate clarification of issues only. Solutions should
not be discussed. If this goal were not required then zero discussion would be advocated. The
point is to link the process activities to the required process goals.
There is still much confusion around as to the difference between peer reviews, inspections,
walkthroughs, technical reviews, etc.. The main difference between these approaches lies in the
purpose, level of formality and the amount of effort required. Formal inspections [Fagan, 1976,
1986], [Gilb, 1993] have greatest formality and incorporate some statistical quality improvement
and defect prevention. Informal reviews and walkthroughs, on the other hand, may focus more on
consensus/buy-in and training respectively rather than rigorously trying to identify and fix
defects. An in-depth discussion is beyond this paper (and has been provided elsewhere
[Gilb,1993],[Knight, 1993]). A key point to remember is that there should be an explicit
relationship between the process activities and the goals of the process. For example, if defect
prevention is not an initial goal of the process then the related defect causal analysis activities are
not necessary, resulting in a simplified process.
What are the benefits of such a process?
The main benefits are time saving, improved quality and reduced costs. This is obtained:
 by finding more defects
 by finding them early in the development lifecycle
 by preventing defects
The improvement in quality as a result of finding more defects before your customer does is
obvious. The fact that they are found early is the key point associated with the prevention/early
detection principle of modern best practices1. It is well accepted that the earlier one finds a defect
in the development lifecycle, the cheaper it will be to fix. Figure 1 shows an example of the
relative cost of fixing defects in the development lifecycle2. Costs associated with the
For example, the Systematic Test and Evaluation Process (STEP), the testing approach from Bill Hetzel’s
company SQE - note the importance of evaluation and that formal reviews/inspections are the obvious
vehicle for Evaluation
2
These were minimum figures with the maximum ratio being 1000 to 1 rather than 40 to 1. The IBM
average was 82 to 1 [Gilb, 1993]
1
2
maintenance phase are also reduced because of a higher quality product with fewer defects and
better documentation to support it. [Gilb, 1993] provides many specific examples and quotes the
following as typical results from organisations which have introduced inspection:
 net productivity increase of 30% to 100% (less rework so productivity improves)
 net timescale reductions of 10% to 30%
 test execution costs and timescales reduced by a factor of 5 to 10 since there are
fewer defects left to find
 reduction in maintenance costs by one order of magnitude
 early or on-time delivery of systems (no defect-correction backlash at systems
integration time, since most defects are treated earlier)
The time savings are obvious from the above discussion in terms of reduced rework, etc. but
ultimately the prevention of defects by designing quality in to the work products will yield the
greatest time saving.
Another benefit is to help build effective technical teams with an associated enhancement in
the flexibility of individuals to take over each other’s work when required. This point is often as
significant a point as the direct benefits described above. Formal reviews/inspections are a form
of on-the-job training where a broad knowledge of the work products and associated quality
issues is rapidly learned by less experienced participants. They are invaluable as a form of
communication especially when different functional areas such as development and test are
brought together to review a work product such as a requirements specification. Work products
improve as a result of a review but they also tend to improve before review simply because they
are going to be reviewed. The development of a quality culture in general is often strongly linked
to the introduction of formal reviews/inspections.
Capers Jones [Jones, 1996] recently reported that formal reviews/inspections had the highest
empirical evidence of success with almost no failures. They ranked top position in structured
methods along with JAD (Joint Application Development) and QFD (Quality Function
Deployment) in terms of Return on Investment (RoI). The industry data collected showed an RoI
of 10 at four years! This figure is backed up by many examples including Thorn EMI in [Gilb,
1993]. Even informal reviews had a RoI of 4 at four years according to Jones’ data.
Cost of fixing defects
40
40
35
30
30
25
20
15
15
O peration
Accept.
D ev. Test
3
C oding
1
D esign
0
R eqs.
5
R elative C o st
10
10
From an analysis of 63 projects (Boehm 1981)
Figure 1. The relative cost of fixing defects depending on where they are found during the
development lifecycle.
3
2. What are the practical experiences?
Four case studies are outlined below. The first two relate to the introduction of the process and
the second two are concerned with issues and improvements surrounding an existing process.
3. Case study - Schaffner Ltd.
Background
This company is a market leader in the manufacture of Power Supply Automated Test equipment
and EMC Systems and Instruments. The Irish division has eight full-time software engineers and
a further twelve that are involved in both software and hardware. As with many traditionally
hardware-oriented companies, they are becoming increasingly oriented towards developing
software, in this case Windows applications. A business goal therefore was to gain more business
through the expansion of their software development activities. To achieve this it was recognised
that more structure was required in their software development process and Peer Reviews was
chosen as one of the process areas on which to focus.
A prior initiative in the area of code reviews failed. This was reported to have been due to a lack
of training and experience with reviews, which led to informality, and a general lack of review
leadership. Reviews degenerated into trivial comments and soon were deemed a waste of time.
They re-launched their improvement initiative at the start of 1995 and focused on a number of
areas most notably project management3. It was estimated at the time that more than half a person
year was being lost each year to fixing operational defects. In January 1996 they received
funding for a focused process improvement project under the ESPITI TRI-SPIN program (a case
study is available [Schaffner, 1996]). The topic of the 6 month project was peer reviews in the
areas of code, design, requirements and test documentation. The purpose was to pilot a peer
review process with an aim of reducing operational defects by 25%. It was also hoped that
reviews would help drive documentation improvements in the areas such as testing which would
be the subject of review.
Implementation
The process improvement was project managed by the quality manager with the Peer Reviews
project a sub-project in the overall improvement program. Peer Review implementation took the
form of a 1-day training workshop followed by seven days consultancy support over the six
months to help with the implementation of the process and improve its effectiveness.
Results
The pilot project was viewed as a success. Despite some cultural resistance that had to be
overcome, the quality manager noted ‘One of our very first reviews saved us considerable grief
and money’. In quantitative terms the following was achieved at the end of the 6 month pilot:
 they achieved a review efficiency of 1 defect/hr i.e. an average of one defect found
for every hour spent either preparing for or attending the review meeting.
 an overall benefit to cost of 2:1 was obtained i.e. for every hour spent with reviews,
two hours was saved on subsequent rework
3
A consultancy ISO9001 audit was performed around this time to help identify opportunities for
improvement and to assess the current costs associated with poor quality.
4
Lessons Learnt
The following were the key lessons learnt:
 project management improvement is a pre-requisite to engineering improvement (see
for example the Level 2 processes which come before Peer Reviews on Level 3 of
the CMM - figure 2.). The progress with the improvement in project management
had been less than anticipated so that when the externally funded Peer Review
process was being piloted, its implementation was often hampered by
resource/schedule issues. This point was made earlier but despite the non-ideal
situation, a benefit to cost of 2:1 was achieved.
 management commitment and support at all levels is crucial.
 experienced practitioner support and just in time training are key issues when
introducing the Peer Review process. The process is very much a people-centered
process and requires implementation support after formal training (especially to help
review leaders do their job effectively). Also, a time lag between the formal training
and the actual piloting was not ideal and reduced the effectiveness of the training.
 ‘with reviews the cultural issues are probably a bigger part of things than pure
techniques’. All of the cultural related activities embedded in an SPI lifecycle model
such as the SEI’s IDEAL lifecycle model [IDEAL, 1997] need to be applied to
overcome the inherent resistance that such a people-centered process can create.
 short improvement cycles such as this 6 month pilot can be quite effective at getting
some tangible improvements underway.
CMM: The key Process Areas
assigned to Process categories
Processes
Categories
Management
Organizational
Engineering
Software project planning,
management, etc.
Senior management
review, etc.
Requirements analysis,
design, code, test, etc.
Levels
5. Optimizing
Technology Change
Management
Process Change
Management
4. Managed
Quantitative Process Management
3. Defined
Integrated Software Management
Intergroup Coordination
2. Repeatable
Requirements Management
Software Project Planning
Software Project Tracking &
Oversight
Software Subcontract Management
Software Quality Assurance
Software Configuration
Management
Ad Hoc Processes
1. Initial
Defect Prevention
Software Quality
Management
Organization Process
Focus
Organization Process
Definition
Training Program
Software Product
Engineering
Peer Reviews
Figure 2. The SEI’s Capability Maturity Model: Key Process areas assigned to process
categories. Note Peer Reviews has been elevated to a Key Process Area in the engineering
category at level 3.
4. Case Study - I.T. Department Bank X
A review process closely based on Fagan Inspections was introduced into the I.T. department of
an Irish Bank. The department consisted of about 200 people. The Software Quality Assurance
manager pioneered the process introduction and personally performed internal presentations on
inspections. Supporting material including process descriptions/definition, templates, etc. was
provided to all the individuals who attended the presentation.
5
Results
The following points summarise what happened:
 overall the process introduction was a failure although there were a few pockets of isolated
success where the people involved adopted the process. Elsewhere the process fell into
disuse.
 Only trivial issues were found at these inspections, data on inspections was not returned to
the process owner (which was SQA), buy-in was generally poor, the process was viewed as
‘too formal with too many forms’, etc.
 The I.T. department are now looking to introduce a more practical approach to formal
reviews/inspections
Lessons learnt





The process which was introduced was too formal for the culture and level of process
maturity which existed at the time in the department
There was insufficient initial training (especially practical exercises) and a lack of on-going
experienced practitioner support. The initial presentations only lasted a few hours.
A number of generic software process improvement principles were not addressed when
trying to introduce the process
 Improvement needs to be managed as a project with clear goals, detailed planning,
sufficient tracking, etc. This was not the case.
 There was no improvement lifecycle of phases and activities (see for example the
‘IDEAL’ approach from the SEI [IDEAL, 1997]. The pilot phase with a set of clear
measures was especially lacking.
There was a lack of involvement of the I.T. staff in general in the process definition. As a
result no tailoring/customisation occurred and there was a lack of ‘ownership’ of the process
by the staff. It was essentially viewed as an ‘SQA process’. There was also a fear of the
metrics being introduced – again buy-in had not been achieved re their use.
There was a lack of focus on major defects (i.e. those with significant downstream costs),
optimum checking rates, etc.
5. Case Study - Telectronics Pacing Systems
This case study is about how an existing peer review process can become ineffective and what
can be done to improve the situation.
Background
Telectronics Pacing Systems were a manufacturer of heart pacemakers and implantable
defibrillators. This case study relates to the Sydney division who specialise in the implantable
defibrillators. During the period to which this case study relates (‘94/’95) there was about 270
people in Research and Development with about 35 in the software department.
The implantable defibrillator communicated with an external instrument which had a user
interface to allow physicians to customise device parameters to individual patients and to read
data logs. The software department was correspondingly divided into implant software,
programmer software and test.
6
Formal Reviews were used firstly in the software department and quickly spread throughout the
division - they soon became the cornerstone of the quality system. The process followed was
similar to that used in Schaffner Ltd. - a major difference however was that no data was collected
from the reviews and as such it would not have met CMM requirements. Inefficiencies in the
process could not be detected from any data. Neither could the uniformity with which the process
was being applied across departments. It became evident however, that some significant
problems were present in certain departments and that the effectiveness of the process was being
questioned in others. A self-assessment focusing on the review process in the different
departments was carried out by the software department.
Problems and Solutions
The self-assessment identified a number of problems in different departments. As the company
wanted a common effective process, a training course was developed to address the problems and
improve the company-wide process. Some solutions however were already present in the process
but had not been well implemented in certain areas. Others were common sense solutions to new
problems that had arisen. The key problems, each reported in at least one department, are
outlined below:
 Cultural issues:
 reviews were viewed as bureaucratic
 ‘it’s not my job’
 reviews seen as a gate or impediment
 reviewing the author or forcing the author to do it ‘my way’
 poor material submitted for review by authors
 etc.
 Process issues
 inadequate preparation be reviewers
 conflict in meetings
 purpose of review unclear
 reviewers find faults but not omissions
 focus on style not content
 meetings were sometimes petty, trivial, slow
 etc.
The training incorporated solutions to all these issues. Some solutions were simply about
enforcing aspects of the existing process, others were improvements to the process and/or useful
improvement guidelines/tips and others were simply discussions to help achieve more buy-in and
address the cultural resistance which was increasing rather than decreasing in some departments.
A key point running through many of the problems was the quality of the written material to start
with. The importance of this issue was recognised and an effective writing course was also
developed and delivered with the help of external consultants.
Some of the improvement guidelines and process improvements to address the ineffectiveness
issues described above are outlined below. They are grouped by phase of the review process:
 Preparation:
 use risk analysis to help select items to review. Everything cannot be reviewed so
select items with the greatest risk e.g. the most complex modules of code, the
modules supporting the most important function required by the customer, etc.
 use a pre-writing conference (there is nothing more inefficient than writing the
wrong document)
7



author and leader should discuss if the review criteria and the reviewers chosen are
appropriate in terms of the number and their skills
 use same review audiences as the earlier phase e.g. code review audience same as
design review
 leader should proof read document before it is distributed as a readiness for review
check
 distribute supporting material with the item under review e.g. design document and
code checklist with code
Review meeting:
 leader should ask how long people spent preparing and defer meeting if it’s not
enough
 leader should constantly monitor and comment on the level of issues being raised
(for typos, most style issues, convention deviations, etc. reviewers can simply pass
on their marked up copy of the document)
 raise issues don’t solve them
 reviewers should be constructive in their criticism
 don’t waste time in the meeting - this is everyone’s responsibility but ultimately the
leaders. Conversations between two individuals for example which are clearly of a
nature which can be discussed outside the review (and don’t involve other reviewers)
should be noted down as such and the meeting moved along.
 remember the useful phrase ‘Consider if......’ when noting down an issue which is
becoming a source of conflict
Follow-up:
 leader checks decisions and edits of author using original document, minutes and
updated version (1 in 6 edits will be incomplete or incorrect according to [Fagan,
1986])
 complete edits and checking ASAP
 re-reviews should use same audience
Results and lessons learnt
The training was viewed as a success by practitioners and managers alike. As no quantitative
data was available however, it really was a case of hoping for the best! The company did receive
ISO9001 certification (which places emphasis the importance of reviews) in this time. From a
software point of view the bottom line result which may be attributable at least in part to
effective reviews was that there was only 10 field detected bugs and no field failures due to
software in thousands of years of run time operation. The key lessons learnt were as follows:
 without metrics the effectiveness of the Peer Review process can only be a
subjective judgement. This reduces your ability to respond in a timely manner when
inefficiencies arise, it is difficult to control the process, it is difficult to overcome
cultural resistance without being able to point to quantifiable benefits and it leaves
one with only a subjective opinion of the quality of the work products.
 formal (re-)training is necessary for effective Peer Reviews.
 effective writing is a key underlying skill which can be a great asset to an
organisation and the individual
 many ‘solutions’ to the problems above are already embedded in more formal
approaches to Peer Reviews. So use a process that is as formal as your current
culture can support.
8
6. Case study - Telecommunications Company Y
This case study is about a telecommunications software development unit in Ireland consisting of
about 40 people. It was recognised that the existing process, which had been loosely based on
Fagan Inspections, had become inefficient. It was decided to perform a series of combined
assessment/training workshops with all the staff. The purpose was to examine the current
process and the associated problems, to provide best practice training on formal
reviews/inspections and to identify key improvements (i.e. to effectively refine the existing
process) to address the problems and achieve the process goals. A further objective was to
achieve buy-in for the process and the improvements.
The workshops were very successful generating considerable buy-in and ownership for the
improvements that were identified. Interestingly, many of the issues identified were similar to
those outlined in the Telectronics case study. The following are some of the actual key
improvement suggestions from these workshops that are currently underway:
a) Project Management
Probably the most significant issue raised by each group was that project management need to
allocate resources for reviews in order for them to be effective. Indeed management needed to
identify what reviews are to be performed and to schedule this time in the project plan. One idea
was to allocate a fixed % of a person-month solely for reviews. This planning process needed to
be explicit and defined at the start of the project.
b) What to review?
 Identify which documents were mandatory to review, which were optional and put this
into the procedure/lifecycle description
 functional specifications supplied by the parent organisation should be reviewed by the
Irish unit but this usually doesn't happen and indeed the specifications are often not
validated till late in development with high rates of requirements creep. This latter issue
is a requirements management problem but reviews would help.
 currently, changes to code/documents arising from test bug fixes, enhancements, patches,
customer issues, etc. are not reviewed. Until now only the first pass through the
development lifecycle involved some review. After the code was written there was never
a review of changes to functional specs, designs or code. There is a need to start
reviewing these changes.
 testers felt that they should review scripts and that resources should be allocated for this.
c) When to review?
Rather than reviewing code in one big lump at the end, it was thought that chunking the
code/spreading it out/reviewing it incrementally along the way would be more effective. The
question of how much code to review depends on risk - risk analysis can help here as can
complexity analysis tools.
d) How to review?
The following are number of specific recommendations provided by the groups re the review
procedure:
 use the same team to review functionality, design, code and associated tests i.e. a
function thread team. This is much more effective than bring people in 'cold' to
review say code and having to educate them on the associated design.
9









keep the reviews as formal as possible with less unnecessary discussion (leader
training was identified as crucial here)
put a time limit of say 2hrs on reviews
develop a better template for recording reviews and/or use immediate on-line entry.
METRICS was identified as a key improvement point. Improve the collection,
analysis and feedback on data (preparation time, meeting time, fix time, defects) and
provide tighter definitions for defects and their classifications. Suggest rules of
thumb for preparation rates, for maximum sizes of code/documents to review in one
review, etc.
use round robin at the review when asking for comments (leader responsibility)
improve the criteria/checklists for use when writing the code/document as well as
when reviewing it. Checklists need to be brought up to date and new checklists for
different types of documents created (and even for patches).
if the reviewers are unfamiliar with the parent document (e.g. functional spec. when
reviewing the design) the author should give a presentation on the parent document
to bring them up to speed before preparation starts.
formalise code presentation for reviews e.g. line numbering, develop a code ‘diff’
tool for use when reviewing changes to code to highlight new, old and modified
code.
ask QA to attend reviews and verify consistent process implementation
7. What do these case studies tell us in terms of deployment strategies?
The two main concepts that one can deduce are as follows:
 implementing a full-blown very formal process to begin with is a questionable strategy,
especially if the maturity of your organisation is too low (e.g. CMM level 1 or level 2)
 a full-blown formal process is very desirable in terms of the additional benefits it can bring
and especially because it may help to minimise the ineffectiveness which tends to creep into
existing less formal approaches
In practical advice terms, one could suggest that an organisation should adopt as formal an
approach as the existing culture and level of process maturity will allow. The plan should be to
increasing the formality and scope of what the process is trying to achieve once acceptance of the
initial deployment has been achieved. Customisation of ‘off-the-shelf’ approaches to
reviews/inspections to match the process goals and culture/maturity of the organisation is
therefore required. This strategy is also more likely to keep the process ‘alive’ i.e. continuously
improving. This is one significant element of what the SEI call ‘institutionalisation’ of a process
[CMM, 1993].
The following two stage deployment model may be of use in this customisation exercise:
Basic process:
 focus on identifying major/critical defects in the work products
 defect prevention (see for example Gilb/Graham) not aggressively pursued - for example
process improvement suggestions which inevitably arise during the review meeting are raised
as related issues - they are a side-effect rather than a focus
 use simple and few forms e.g. an entry check to the process should occur but it may not
require a separate form to be completed
10






minimise the number of meetings in the process e.g. an entry review or check can occur
without a formal meeting, an overview/kick-off meeting can be replaced by informal contacts
supplemented by distributing a review summary form containing additional roles4, etc.
data collection and analysis is simple, minimal and focused
concentrate more on the big questions (like does this design address all the requirements) and
the more experienced reviewers ability to find the important problems rather than
sophisticated criteria/rules and standards - obviously an increasing use of criteria, checklists,
etc. should be aggressively planned.
informal use of additional reviewing roles. Again, plan to improve. An example would be to
better define what the ‘tester’ reviewing role should focus upon when reviewing say a
functional specification. Initially, it would be left to individual testers to provide their
perspective before these criteria/checklists become available.
facilitators and reviewers involved in the process definition and trained
etc.
Advanced process:
 basically adopt more of the elements defined in approaches such as Gilb/Graham e.g.
specific meetings/organisational structures to aggressively focus on defect prevention
 formalise more of the process e.g. through specified documented decision points/meetings
 increase use of data in decision making, analysis and general quantitative control of the
process and work products e.g. in process entry/exit criteria
 more sophisticated code/document sampling strategies
 increased use of criteria/rules, checklists, etc. for each type of document to be reviewed
 formalised use of additional reviewing roles for different types of documents
 facilitators trained and subject to initial and on-going certification
 etc.
This deployment model may be of use to those who understand and believe in the benefits of a
process such as that defined by Gilb/Graham but who know from experience or intuition that
they actually require something which is more practical and effective for their particular
organisation, at least to begin with. In the initial deployment, the model attempts to retain the
essence or key elements of the more formalised approaches in order to maximise the benefits
achievable from this powerful engineering technique.
8. What therefore are the critical success factors in achieving maximum
benefit from formal reviews/inspections?



Customise the process to suit your goals and to match your culture/maturity – start simple
but formal.
Provide initial training and ongoing experienced practitioner support. This support can be in
the form of experienced practitioners leading review meetings during initial deployment,
holding clinics to assess the process and the level of acceptance, mentoring, etc.
Foster continuous process improvement by continually analysing the existing process
quantitatively, piloting improvements, expanding the process goals, etc.
4
Note however, that during the initial introduction of a process, the overview meeting is a practical way to
reinforce objectives and approach required.
11





Determine and enforce your the optimum checking rate for key parts of documents (i.e. nontrivial parts where Majors can be found). This will involve initially estimating and then
calibrating the rate based on real data from your process.
Focus on Major defects (not minors) as they are what give the real return on investment5
Focus initially on upstream documents such as contracts, requirements and functional
specifications rather than code - the return is greater.
Ensure the results and benefits of reviews are quantified and published to all staff.
Place increasing emphasis on improving documentation and the development process is
general as a result of the improvement suggestions arising from reviews. In relation to the
review/inspection process, the documentation of review criteria/rules and checklists is a key
factor.
8. References
[Boehm, 1981] B. Boehm, Software Engineering Economics, Englewood Cliffs, NJ: PrenticeHall
[CMM, 1993] M. Paulk, et. al., Key Practices of the Capability Maturity Model, Version 1.1,
Software Engineering Institute SEI-93-TR-025, 1993
[Fagan, 1976] M.E. Fagan, Design and Code Inspections to Reduce Errors in Program
Development, IBM Systems Journal, 15(3), 182-211, 1976.
[Fagan, 1986] M.E. Fagan, Advances in Software Inspections, IEEE Transactions on Software
Engineering, SE-12(7), July, 744-751
[Gilb, 1993] T. Gilb, D. Graham, Software Inspection, Addison-Wesley, 1993, ISBN 0-20163181-4
[Jones, 1996] C. Jones, Quantifying Software Process Improvements, Keynote address, SP’96,
(SPI track), Brighton, 1996
[Knight, 1993] J.C. Knight, E. Ann Myers, An improved inspection technique, Communications
of the ACM, Nov. 1993.
[Schaffner, 1996] TRI-SPIN Case Study on Schaffner pilot available from http://www.cse.dcu.ie
[IDEAL, 1997] See http://www.sei.cmu.edu for details
[Gilb, 1996] See http://www.stsc.hill.af.mil/SWTesting/gilb.html
5
See [Gilb, 1996] for a more detailed discussion on how to focus on majors, optimum checking rates, etc.
12