INFO 631
Prof. Glenn Booker
Week 5 – Quality Assessment &
Process Improvement
INFO631 Week 5
1
www.ischool.drexel.edu
Time for a Sanity Check
• We have defined many ways to measure a
project, and establish various guides for
quality
• Now, as a project is happening, we want
to determine if we are reaching our goals
INFO631 Week 5
2
www.ischool.drexel.edu
Assessment versus Audit
• Here, per Kan, distinguish between a
quality assessment and a quality audit
– A quality assessment is an evaluation of the
project to see if desired goals are being met,
and identify problem areas
– A quality audit is used to refer to formal
external activities, such as an ISO 9000 audit
or CMMI audit
INFO631 Week 5
3
www.ischool.drexel.edu
Quality Assessment
• A quality assessment could be done by an
independent team, or by people within the
project’s QA organization
• Assessments can be done at various
points in a project
– A typical project might schedule 4-6 major
quality reviews (Kan, p. 398), evenly spaced
during the project
INFO631 Week 5
4
www.ischool.drexel.edu
Quality Assessment
• Each quality assessment typically has
three phases
– Preparation phase
– Evaluation phase
– Summarization phase
• Recommendations and Risk Mitigation
• The scope of data and measures are key
aspects of each phase
INFO631 Week 5
5
www.ischool.drexel.edu
Preparation Phase
• Given the development process used on
a project, and the project schedule,
determine where the project stands
relative to them
– Need to focus on both qualitative and
quantitative data to determine development
progress, find problems, and predict final
product quality
INFO631 Week 5
6
www.ischool.drexel.edu
Preparation Phase
• Qualitative data can be obtained from
interviews (one-on-one or small group)
or questionnaires
• May focus on identifying
– Whose input is critical right now?
– Are affected all areas represented?
– Who needs to know what’s happening right
now?
INFO631 Week 5
7
www.ischool.drexel.edu
Preparation Phase
– What are the key risks, and our mitigation
strategy for each?
– How does our status compare to similar
previous projects?
INFO631 Week 5
8
www.ischool.drexel.edu
Evaluation Phase
• Now that you have the data on hand, the
evaluation phase is the act of using
various techniques to analyze the data,
and extract the desired information
– Data might be presented using the indicators
discussed with GQ(I)M
INFO631 Week 5
9
www.ischool.drexel.edu
Evaluation Phase
• Part of the challenge is understanding
when you have enough data, and how to
put the data in perspective of the project
as a whole
– Here it’s critical to choose the right measures
to evaluate the data, and pick good evaluation
criteria to judge what’s ‘good’ and ‘bad’
INFO631 Week 5
10
www.ischool.drexel.edu
Summarization phase
• Now we want to blend the qualitative and
quantitative results into an overall
summary of the state of the project
• Summary is typically broken into major
sections, depending on the types of
activities currently occurring
– Design reviews, code inspections, system
testing, regression testing, etc.
INFO631 Week 5
11
www.ischool.drexel.edu
Summarization phase
– Each section’s current status might be
summarized with a basic stoplight scale
(red/yellow/green) to indicate how severe
problems are in that area
– These could be tracked over time to show
trends in quality
INFO631 Week 5
12
www.ischool.drexel.edu
Recommendations and Risk
Mitigation
• Part of the summary is to give specific
recommendations
– Look at current risky areas, and describe what
activities are needed to bring them under
better control?
• This often links to your project’s risk
management strategy, where you
identified significant risks, and defined a
risk mitigation strategy for each
INFO631 Week 5
13
www.ischool.drexel.edu
Risk Mitigation
• Risk strategies can include
– Contain the risk, using specific activities to
reduce the chance of it happening (risk
prevention) and reduce its impact if it does
occur (risk mitigation)
– Develop a contingency plan to manage the
risk if it occurs
INFO631 Week 5
14
www.ischool.drexel.edu
Risk Mitigation
– Transfer to risk to another organization or part
of the system (let a subcontractor develop a
part you know poorly, for example)
– Ignore the risk, if it has low impact
– Avoid the risk, by using a different process or
eliminate a risky feature from the product
requirements
INFO631 Week 5
15
www.ischool.drexel.edu
Conducting Assessments
• In planning an assessment, the scope is a
critical aspect
– Often an assessment will be limited
to part of a project, to address areas most
likely to be critical
– The emphasis is still on determining whether
the project is meeting its goals for quality
INFO631 Week 5
16
www.ischool.drexel.edu
Conducting Assessments
• Audits are performed in a manner similar
to assessments
– The difference in focus is that audits focus on
comparing processes to a process model
(CMMI, ISO 9000), whereas assessments
compare project outcomes to its own goals
– An audit’s scope could cover several projects
within one organization
INFO631 Week 5
17
www.ischool.drexel.edu
Conducting Assessments
• In addition to the usual models, audits
could be based on other models
– Bootstrap
– Trillium
– ISO 12207
– ISO 15504
– DOD-STD-2167a (obsolete)
– MIL-STD-498 (obsolete)
INFO631 Week 5
18
www.ischool.drexel.edu
Conducting Assessments
• Likewise, the scope of an audit could differ
based on the model selected
– ISO 9000 is facility-based, since it
was created for a manufacturing environment
– CMMI is organization-based, regardless of
where the organization is located
INFO631 Week 5
19
www.ischool.drexel.edu
Process Assessment Cycle
• The conduct of a process assessment or
audit typically has six major steps
– Preplanning – before the audit, get
sponsorship for the audit and define its scope
and purpose
– Planning – Select who will conduct the audit,
when, and provide training if needed
INFO631 Week 5
20
www.ischool.drexel.edu
Process Assessment Cycle
– Fact gathering – determine how data will be
collected for the audit; interviews, documents,
demonstrations, questionnaires, etc.
– Fact analysis – analyze the data, and get
additional data if needed to fill holes in
understanding the subject
– Reporting – prepare a presentation to give the
results of the audit
INFO631 Week 5
21
www.ischool.drexel.edu
Process Assessment Cycle
– Finally, after the audit, determine how its
results will impact the project’s process
improvement activities
• Assessments follow the same activities
described above; and are typically
conducted every few months
• Audits have to be repeated every 6-18
months to remain valid
INFO631 Week 5
22
www.ischool.drexel.edu
A Bigger Picture
• Assessment or audit results can be
tracked over many events to look for
chronic problem areas, or help refine
process improvement plans
INFO631 Week 5
23
www.ischool.drexel.edu
More Details on Audits
• Kan’s chapter 17 focuses on more details
of how process audits are conducted
• We’ll hit the highlights here
• For more information, go to the SEI
website, and look for information
on SCAMPI
INFO631 Week 5
24
www.ischool.drexel.edu
CMMI Structure
• The CMMI is broken into five cumulative
levels of maturity (#1-5), plus sometimes
level 0 is possible
– Level 0 means your project doesn’t do
everything necessary for, e.g. software
development
– Level 1 means your project does stuff, but no
one knows how (“ad hoc” processes)
INFO631 Week 5
25
www.ischool.drexel.edu
CMMI Structure
– Level 2 is the first level earned by an
assessment; it means a project performs
basic project and process management
activities
– Level 3 means that an organization (note the
shift from project) has processes which are
consistently tailored across several projects
INFO631 Week 5
26
www.ischool.drexel.edu
CMMI Structure
– Level 4 means the organization has
measured and tracked key activities
consistently for so long that statistical process
control is in place
– Level 5 means the organization actively
prevents defects from occurring, and is
experimenting with even better processes
(continuous process improvement)
INFO631 Week 5
27
www.ischool.drexel.edu
Staged versus Continuous
• CMMI can be used in staged or
continuous representation (form)
– Staged is the approach just described – the
organization must perform certain types of
activities to earn an overall maturity rating
– Continuous form means that each type of
activity is rated separately, so there’s no
overall rating
INFO631 Week 5
28
www.ischool.drexel.edu
Staged versus Continuous
• In practice, most organizations get audited
using the staged approach
– That CMMI Level 5 plaque on the
wall is impressive!
• But the work of preparing for audits often
follows the continuous approach
– Assess what kind of activities are most in
need of help (process, project, technical, etc.),
and work on them first
INFO631 Week 5
29
www.ischool.drexel.edu
The Bottom Line
• The overall goal for a CMMI-based
assessment is to determine if the project
or organization fulfills the goals of the
model
– Are the goals of each type of activity being
met consistently across the organization?
– Often called institutionalization of the activity –
it’s a way of life for them
INFO631 Week 5
30
www.ischool.drexel.edu
More Structure
• CMMI is broken into about 23 Process
Areas (PA) – each is a ‘type of activity’
mentioned earlier
– Each PA has a few Specific Goals, which are
the heart of that activity
• Then there are more general activities
defined for each PA – doing all of them
constitutes ‘institutionalization’
INFO631 Week 5
31
www.ischool.drexel.edu
More Structure
• The general process area structure
includes
– Commitment to Perform – means that
program management defines a policy to do
the stuff in this PA
– Ability to Perform – means that program
management provides the resources and
training needed to do these activities, and
plans to do them
INFO631 Week 5
32
www.ischool.drexel.edu
More Structure
– Directing Implementation – means that
program management manages the
processes involved in this activity, ensures
quality, and coordinates with other
stakeholders on the project
– Verifying Implementation – covers review of
activities with higher management, and
conducting independent process audits
INFO631 Week 5
33
www.ischool.drexel.edu
More Structure
– Then the specific goals of each PA are
defined in more detail by its Specific Practices
• These describe the types of activities which, done
together, will fulfill the goals of the process area
• In addition, based on the desired maturity
level, there are Generic Goals which apply
to every PA
INFO631 Week 5
34
www.ischool.drexel.edu
CMMI and Measurement
• The measurement focus for CMMI is
– 1) Define a plan of activities,
– 2) Track actual progress, then
– 3) Measure actual and planned performance,
and make decisions based on it
• At higher levels of maturity, analysis of
data and prediction occur
INFO631 Week 5
35
www.ischool.drexel.edu
CMMI and Measurement
• For example, at Level 2, the Measurement
and Analysis process area has specific
goals of
– Measurement objectives and activities are
aligned with identified information needs and
objectives.
– Measurement results that address identified
information needs and objectives are
provided.
From here
INFO631 Week 5
36
www.ischool.drexel.edu
CMMI and Measurement
• At level 4, the Quantitative Project Management
process area has specific goals of
– The project is quantitatively managed using
quality and process-performance objectives.
– The performance of selected subprocesses
within the project's defined process is
statistically managed.
INFO631 Week 5
37
www.ischool.drexel.edu
CMMI in Brief
• CMMI is mostly looking to see if you say
what you’re going to do, make it possible
to do so, do it, and measure what you did
• Following a process improvement path
takes lots of time and commitment from all
levels of management
INFO631 Week 5
38
www.ischool.drexel.edu