Examiner’s Initials:
Applicant Number:
Tennessee Center for Performance Excellence
www.tncpe.org
(800) 453-6474
2008 Pre-Work
Independent Review
Scorebook
Examiner’s
Name
Number of Hours
Worked
Applicant
Number
Criteria, Score Summary Worksheet, and Scoring Guidelines Used:
Business/Nonprofit
Health Care
1
Education
Applicant Number:
Examiner’s Initials:
NOVEL CONNECT KEY FACTORS
For Pre-work the Key Factors in P.1a and P.1b have been provided. Please review and then complete the
Key Factors for P.2a, P.2b and P.2c.
Key Factors Worksheet
To begin the evaluation process, review the applicant’s Organizational Profile and the Eligibility Certification
Form. List the key business/organization factors for this applicant, using the Areas to Address (Organizational
Environment, Organizational Relationships, Competitive Environment, Strategic Context, and Performance
Improvement System) in the order presented in the Preface: Organizational Profile section of the appropriate
Criteria for Performance Excellence booklet.
P.1a Organizational Environment
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Midsized manufacturer of cell phone hardware, software (including ringtones), cell phone
accessories, and other communication devices that integrate audio, text, and Global Positioning
System (GPS) features. Five major product lines: Novel Complete, Novel Secure 1, Novel Free,
Novel Bug, and NovelAid. Basic manufacturing (components and hardware) is done by offshore
suppliers.
Founded in 1994 as a private company; it went public in 2002.
Products are distributed through “big box” retailers, carrier retail outlets, company-branded outlets,
kiosks in shopping malls, and Internet sales
Purpose: Facilitates a world on the move. Vision: the most innovative company for mobile
communication in the world. Mission: leverages new and existing technology to advance mobile
communication. Core values: agility, valuing employees/partners, innovation, and sustainability
Culture promotes core competencies of agility and communication: working out of the home,
flexible work schedules, and maximizing technology (cell phones, virtual meetings,
teleconferencing) to minimize travel.
4,188 employees make up a “virtual,” distributed workforce: 1,200 in innovation (sales, R&D,
marketing, IT, and product engineering), 2,738 in operations, and 250 in administration and support.
There is no employee union. Highly educated workforce: 25% with a postgraduate degree, 40% with
an undergraduate degree, 25% with some technical college, and 99% with a high school diploma.
Women compose 65% of the workforce; 50% of employees are under age 40; for 20%, English is a
second language; and 15% are disabled.
Workforce requirements for all: knowing what is expected, having the right materials and training,
timely and appropriate feedback and recognition, opportunities for growth and development,
organizational flexibility to accommodate diverse lifestyles, co-workers committed to excellence,
connection with the company’s values, and the ability to contribute to its success.
Additional requirements for office and home workers: appropriated work space ergonomics,
personal safety, and security. Additional requirements for manufacturing workers: appropriate
ergonomics, machine operation safety, environmental safety, emergency preparedness, personal
safety, and security
Facilities: Headquarters in Rochester, NY (1,622 employees, or about 39%) includes offices, the
only company-owned manufacturing plant, a research laboratory, and a distribution center. Also,
throughout the U.S., 11 “pods”- leased office spaces that serve as hubs for home-based employees,
including call center and customer briefing center (2,566 employees, or about 61%). Pods are
located in university cities and near components manufacturers and transportation. Technologies
include telecommunications and offshore production and assembly of cell phone housings, key pads,
internal circuit boards, and product packaging materials.
The regulatory environment includes the Wireless Telecommunications Bureau (WTB) of the
Federal Communication Commission (FCC), the Occupational Safety and Health Administration
2
Examiner’s Initials:
Applicant Number:
(OSHA), the Environmental Protection Agency (EPA), the Securities and Exchange Commission
(SEC), ICP-A-610, the International Organization for Standardization (ISO), the Help Desk Institute
(HDI), and the TL9000/QuEST Forum.
P.1b Organizational Relationships
11. Organizational structure: A nine-member Board of Directors (BOD) composed of eight independent
members and the CEO, four standing BOD committees, and a five-member Senior Leader Team
(SLT). The relatively flat organization has one rotating ethics officer, and the 11 pods have team
leaders.
12. Key customer segments: personal consumers (students in Gen-Y, celebrities and sports stars,
preteens, single adult females, the elderly, the disabled); personal/business consumers (“outdoors
people”); business consumers (truckers, taxi drivers); business/government consumers (emergency
services workers); and government consumers (the Department of Homeland Security)
Emphasis on repeat purchases to build relationships and maintain market share
14. Key customer requirements: all-ease of use, reliability; personal consumers-trendiness, convenience,
secure/encrypted data and transmission, personal/home safety and security, low cost, ruggedness;
business consumers-ruggedness, personal safety and security, data and voice capability, sustained
signal/strength across distances, secure/encrypted data and transmission; business/government
consumers-security, data and voice capability, secure/encrypted data and transmission, sustained
signal/strength across distances
15. Key suppliers/partners: two offshore manufacturing suppliers (in China and India), a cell carrier,
retailers, transportation companies, integrated component/software manufacturers, universities, IT
support, a security company, and a law firm
13.
P.2a Competitive Environment
16.
P.2b Strategic Context
P.2c Performance Improvement System
3
Examiner’s Initials:
Applicant Number:
Independent Review Worksheet – Sample
Item 3.1
4–6 most significant key factors (KFs) for this Item
1. Customer segments – personal consumers (students, celebrities,
preteens, elderly disabled), personal/business consumers
(“outdoors people”), business consumers (truckers, taxi drivers),
business/government consumers (emergency service workers),
and government consumers (US Dept. of Homeland Security).
Employees are also customers
4. Five products lines –Novel Complete (core device includes
Phashion phones). Novel Secure (focus on home and personal
security), Novel Free (audio centric), Novel Bug (small, full
wireless, hands free device), Novel Aid (for visual and hearing
impaired customers), but phones can be customized with features
from each line.
2. Customer requirements – reliability (all); ease of use (all personal,
business, and business/gov’t), data and voice capability (gov’t,
business/gov’t), (students: celebrities:, secure/encrypted data and
transmission (celebrities, business/gov’t, gov’t), low cost
(preteens), personal safety and security (personal/business),
personal/home safety and security (preteens), ruggedness
(business, personal/business), sustained signal strength across
distances (business/gov’t), trendiness (students), convenience
(students), trend setting (celebrities), and security (gov’t).
5. Strategic Advantages – Product/feature design innovation,
business model innovation, supplier partnerships/lowered costs
a. Distribution through “big box retailers, carrier retail
outlets, limited number of small company-branded retail
outlets & kiosks, direct internet sales
6. Strategic Challenges - Communication, logistics, volatility in
niche markets, Market forces driving the cost of cell phones and
market penetration
3. Success factors - relationship with carriers, ability to rapidly
respond to marketplace changes with new products, features
and/or superior quality; maintaining strong margins by controlling
costs and/or optimizing process performance, supply chain
management, and collaborations with key suppliers/partners.
Item ref.
3.1a(1)
List only those challenges, success
factors, etc., that are relevant to this
Item.
Brief description of approach
Relevant
KF
Evidence that approach
is systematic
Evidence of extent of
deployment of approach
Evidence of systematic
improvements to
approach
Evidence of alignment
and integration
How identify customers and
market segments
Uses a niche focus to identify
key sub-segments of broad key
customer groups (Fig 3.1-1)
1, 2, 4,
5
– Steps performed to
identify new market
segments & product
features from
reviews not evident
+ Reviewed each
Feb. as part of SPP
+ Nearly all products
and services are
represented by
segmentations in Fig.
3.1-1
+ after 2000,
customer input
obtained before new
products and product
features are added
+ Integrated with
SPP
– Information from
customers of
competitors & other
potential customers
– no evidence
processes have been
systematically
reviewed and
Marketing & Public Relations
team reviews market segments,
product features and price points
to identify new market segments
Show evidence
including steps, time
frames, inputs &
outputs to indicate
approach is repeatable/
systematic.
4
+ Aligned w/
strategic advantages
of product/feature
design innovation
and business model
innovation through
distributed virtual
Examiner’s Initials:
Applicant Number:
to target & product features to
add
Seven customer types in five
market segments identified.
3.1a(2)
How determine key customer
requirements
Using italics, record brief
description of Item
requirements you are
addressing.
not available
Process observations should be
brief, concise phrases.
1, 2, 3,
4, 6
Bold those
observations
you view as
particularly
favorable or
unfavorable.
Five VOC processes are used
(Figure 3.1-2) ) as primary
means to determine key
customer requirements, needs,
and changing expectations
+ repeatable
approach
– Not clear how
data gathered are
used to determine
customer
requirements.
+ VOC approaches
have varying targets,
locations, content,
and frequencies
improved
Use + and  to indicate to
other Examiners which
observations you view as
favorable or unfavorable.
+ VOC data made
available to all
employees in MAP
and sent to retailers
and carrier on a
quarterly basis
+ Data fed into MAP
for process
improvement projects
workforce
+ Integrated with
Product, Feature, and
Process Development
Process (PFPD)
(Figure 6.1-4)
+ Integrated with
SPP, PFPD, and
MAP database
+ All employees also
considered customers
and salespeople
+ Appear to be
deployed to all
customer segments
and sub-segments
– VOC methods do
not appear to vary for
the different
customer groups

3 customer surveys

customer complaints

CAGs
+ administered at
specifics times
+ Partners with
carrier on customer
surveys and
complaint
management
+ conducted
quarterly
+ CAGs deployed to
all pod communities
by employees
5
Examiner’s Initials:
Applicant Number:


3.1a(3)
market research
customer relationship
management process
Four blogs used to promote
products and bounce off new
ideas
Use VOC information/feedback
to become customer-focused and
satisfy needs/desires
+/- ongoing
+ includes carrier and
its call center
2, 3
Formal Improvement Day using
Allegiance Survey data
3.1a(4)
How keep listening learning
methods current
Question on Allegiance Survey
is used to determine
opportunities to improve
listening and learning methods
2, 4, 6
+ held annually by
Marketing and
Public Relations
Team
+ Data fed into SPP
+ Survey process
owners and other
internal stakeholders
+ Integrated with
SPP, PFPD, and
MAP database
+ Data reviewed by
process owners and
int. stakeholders at
Annual
Improvement Day
 Responses limited
to the those surveyed
Do not leave boxes blank if
there is no evidence to record.
Write “n/a” or “no evidence
to record” to tell other
Examiners you did not skip
this area.
 No evidence of
improvements made
+ Integrated with
SPP
 Unclear if question
on Allegiance Survey
and Improvement
Day systematically
apply to all Five
Voices and to all
customer types such
as those in Fig. 3.1-1
Were any key Criteria requirements or relevant KFs not addressed?
Item ref.
Requirement or KF not addressed
Why is this gap significant for the applicant?
a(1)
How applicant includes customers of competitors and
other potential customers
Applicant may miss opportunities to identify new niche markets
6
Examiner’s Initials:
Applicant Number:
a(2)
Unclear how applicant uses relevant information and
feedback from former customers
a(2)
How information from current and former customers used
for making work system and work process improvements
Keeping listening and learning methods current with
business needs and directions
a(4)
Gathering information from former customers about why they no longer use
their products may help the applicant to understand how it may better meet
customer requirements.
Business model innovation is defined by the applicant as a Strategic
Advantage.
Lack of a systematic process may make it difficult to ensure the applicant is
getting up to the minute feedback from customers, and will be aware of
changes in marketplace
Use the “Notes” box to jot down reminders about
things you want to check in another Item or to provide
further explanation/thoughts about an observation
you’ve made.
Optional: Overall Item Notes
Here you can capture a critical thought that relates to the assessment of the Item. It is not intended to be used to capture the full set of comments for the Item or key themes. You
will prepare your comments for this Item for the first time during the Consensus Review—not during the Independent Review.
a(1) Learning – not sure this improvement to approach was systematic or just happened due to acquisition
a(2) I see a description of the methods used to gather VOC data, but not how they use it to determine key requirements – this may be an OFI comment
a(4) look for results of Allegiance Survey question “How can we better listen and learn of your needs and future directions?” in 7.2
a(4) not sure how using this single point from the customers’ view can be effective without additional data e.g., process performance, outcome
metrics.
7
Examiner’s Initials:
Applicant Number:
Item 3.1 Scoring
Factor
Approach
0–5%
No systematic approach to
Item requirements is evident;
information is anecdotal.
10–25%
30–45%
The beginning of a systematic
approach to the basic
requirements of the Item is
evident.
An effective, systematic
approach, responsive to the
basic requirements of the Item,
is evident.
50–65%
70–85%
90–100%
An effective, systematic
approach, responsive to the
overall requirements of the
Item, is evident.
An effective, systematic
approach, responsive to the
multiple requirements of the
Item, is evident.
An effective, systematic
approach, fully responsive to
the multiple requirements of
the Item, is evident.
The approach is well
deployed, although
deployment may vary in some
areas or work units.
The approach is well
deployed, with no significant
gaps.
The approach is fully
deployed without significant
weaknesses or gaps in any
areas or work units.
A fact-based, systematic
evaluation and improvement
process and some
organizational learning,
including innovation, are in
place for improving the
efficiency and effectiveness of
key processes.
Fact-based, systematic
evaluation and improvement
and organizational learning,
including innovation, are key
management tools; there is
clear evidence of refinement
as a result of organizationallevel analysis and sharing.
Fact-based, systematic
evaluation and improvement
and organizational learning
through innovation are key
organization-wide tools;
refinement and innovation,
backed by analysis and
sharing, are evident
throughout the organization.
The approach is aligned with
organizational needs identified
in response to the
Organizational Profile and
other Process Items.
The approach is integrated
with organizational needs
identified in response to the
Organizational Profile and
other Process Items.
The approach is well
integrated with organizational
needs identified in response to
the Organizational Profile and
other Process Items.
X
Deployment
Little or no deployment of any
systematic approach is
evident.
The approach is in the early
stages of deployment in most
areas or work units, inhibiting
progress in achieving the basic
requirements of the Item.
The approach is deployed,
although some areas or work
units are in the early stages of
deployment.
X
Learning
An improvement orientation is
not evident; improvement is
achieved through reacting to
problems.
Early stages of a transition
from reacting to problems to a
general improvement
orientation are evident.
The beginning of a systematic
approach to evaluation and
improvement of key processes
is evident.
X
Integration
No organizational alignment is
evident; individual areas or
work units operate
independently.
The approach is aligned with
other areas or work units
largely through joint problem
solving.
The approach is in the early
stages of alignment with basic
organizational needs identified
in response to the
Organizational Profile and
other Process Items
X
Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most
descriptive of the organization’s achievement level for the Item.
Item 3.1—Overall Score
0–5%
10–25%
30–45%
X 50–65%
70–85%
90–100%
Score each evaluation factor, considering all approaches for the Item, deployment for all of the approaches, etc. Then view
all of the factors holistically. DO NOT AVERAGE the scores. Read the Scoring Guidelines and select the range most
descriptive of the applicant’s level of acheivement. Check the ranges above and below your selected range to ensure
appropriate placement. Finally, decide on a specific score.
Item 3.1 Score
8
50
%
Examiner’s Initials:
Applicant Number:
Independent Review Worksheet
Item 3.2
4–6 most significant key factors (KFs) for this Item
Item ref.
Brief description of approach
Relevant
KF
Evidence that approach
is systematic
Evidence of extent of
deployment of approach
Evidence of systematic
improvements to
approach
Evidence of alignment
and integration
Were any key Criteria requirements or relevant KFs not addressed?
Item ref.
Requirement or KF not addressed
Why is this gap significant for the applicant?
Optional: Overall Item Notes
Here you can capture a critical thought that relates to the assessment of the Item. It is not intended to be used to capture the full set of comments for the Item or key themes. You
will prepare your comments for this Item for the first time during the Consensus Review—not during the Independent Review.
9
Examiner’s Initials:
Applicant Number:
Item 3.2 Scoring
Factor
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Approach
No systematic approach to
Item requirements is evident;
information is anecdotal.
The beginning of a systematic
approach to the basic
requirements of the Item is
evident.
An effective, systematic
approach, responsive to the
basic requirements of the Item,
is evident.
An effective, systematic
approach, responsive to the
overall requirements of the
Item, is evident.
An effective, systematic
approach, responsive to the
multiple requirements of the
Item, is evident.
An effective, systematic
approach, fully responsive to
the multiple requirements of
the Item, is evident.
Deployment
Little or no deployment of any
systematic approach is
evident.
The approach is in the early
stages of deployment in most
areas or work units, inhibiting
progress in achieving the basic
requirements of the Item.
The approach is deployed,
although some areas or work
units are in the early stages of
deployment.
The approach is well
deployed, although
deployment may vary in some
areas or work units.
The approach is well
deployed, with no significant
gaps.
The approach is fully
deployed without significant
weaknesses or gaps in any
areas or work units.
An improvement orientation is
not evident; improvement is
achieved through reacting to
problems.
Early stages of a transition
from reacting to problems to a
general improvement
orientation are evident.
The beginning of a systematic
approach to evaluation and
improvement of key processes
is evident.
A fact-based, systematic
evaluation and improvement
process and some
organizational learning,
including innovation, are in
place for improving the
efficiency and effectiveness of
key processes.
Fact-based, systematic
evaluation and improvement
and organizational learning,
including innovation, are key
management tools; there is
clear evidence of refinement
as a result of organizationallevel analysis and sharing.
Fact-based, systematic
evaluation and improvement
and organizational learning
through innovation are key
organization-wide tools;
refinement and innovation,
backed by analysis and
sharing, are evident
throughout the organization.
No organizational alignment is
evident; individual areas or
work units operate
independently.
The approach is aligned with
other areas or work units
largely through joint problem
solving.
The approach is in the early
stages of alignment with basic
organizational needs identified
in response to the
Organizational Profile and
other Process Items.
The approach is aligned with
organizational needs identified
in response to the
Organizational Profile and
other Process Items.
The approach is integrated
with organizational needs
identified in response to the
Organizational Profile and
other Process Items.
The approach is well
integrated with organizational
needs identified in response to
the Organizational Profile and
other Process Items.
Learning
Integration
Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most
descriptive of the organization’s achievement level for the Item.
Item 3.2—Overall Score
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Item 3.2 Score
10
%
Examiner’s Initials:
Applicant Number:
Independent Review Worksheet
Item 5.1
4–6 most significant key factors (KFs) for this Item
Item ref.
Brief description of approach
Relevant
KF
Evidence that approach
is systematic
Evidence of extent of
deployment of approach
Evidence of systematic
improvements to
approach
Evidence of alignment
and integration
Were any key Criteria requirements or relevant KFs not addressed?
Item ref.
Requirement or KF not addressed
Why is this gap significant for the applicant?
Optional: Overall Item Notes
Here you can capture a critical thought that relates to the assessment of the Item. It is not intended to be used to capture the full set of comments for the Item or key themes. You
will prepare your comments for this Item for the first time during the Consensus Review—not during the Independent Review.
11
Examiner’s Initials:
Applicant Number:
Item 5.1 Scoring
Factor
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Approach
No systematic approach to
Item requirements is evident;
information is anecdotal.
The beginning of a systematic
approach to the basic
requirements of the Item is
evident.
An effective, systematic
approach, responsive to the
basic requirements of the Item,
is evident.
An effective, systematic
approach, responsive to the
overall requirements of the
Item, is evident.
An effective, systematic
approach, responsive to the
multiple requirements of the
Item, is evident.
An effective, systematic
approach, fully responsive to
the multiple requirements of
the Item, is evident.
Deployment
Little or no deployment of any
systematic approach is
evident.
The approach is in the early
stages of deployment in most
areas or work units, inhibiting
progress in achieving the basic
requirements of the Item.
The approach is deployed,
although some areas or work
units are in the early stages of
deployment.
The approach is well
deployed, although
deployment may vary in some
areas or work units.
The approach is well
deployed, with no significant
gaps.
The approach is fully
deployed without significant
weaknesses or gaps in any
areas or work units.
An improvement orientation is
not evident; improvement is
achieved through reacting to
problems.
Early stages of a transition
from reacting to problems to a
general improvement
orientation are evident.
The beginning of a systematic
approach to evaluation and
improvement of key processes
is evident.
A fact-based, systematic
evaluation and improvement
process and some
organizational learning,
including innovation, are in
place for improving the
efficiency and effectiveness of
key processes.
Fact-based, systematic
evaluation and improvement
and organizational learning,
including innovation, are key
management tools; there is
clear evidence of refinement
as a result of organizationallevel analysis and sharing.
Fact-based, systematic
evaluation and improvement
and organizational learning
through innovation are key
organization-wide tools;
refinement and innovation,
backed by analysis and
sharing, are evident
throughout the organization.
No organizational alignment is
evident; individual areas or
work units operate
independently.
The approach is aligned with
other areas or work units
largely through joint problem
solving.
The approach is in the early
stages of alignment with basic
organizational needs identified
in response to the
Organizational Profile and
other Process Items.
The approach is aligned with
organizational needs identified
in response to the
Organizational Profile and
other Process Items.
The approach is integrated
with organizational needs
identified in response to the
Organizational Profile and
other Process Items.
The approach is well
integrated with organizational
needs identified in response to
the Organizational Profile and
other Process Items.
Learning
Integration
Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most
descriptive of the organization’s achievement level for the Item.
Item 5.1—Overall Score
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Item 5.1 Score
12
%
Examiner’s Initials:
Applicant Number:
Independent Review Worksheet
Item 6.1
4–6 most significant key factors (KFs) for this Item
Item ref.
Brief description of approach
Relevant
KF
Evidence that approach
is systematic
Evidence of extent of
deployment of approach
Evidence of systematic
improvements to
approach
Evidence of alignment
and integration
Were any key Criteria requirements or relevant KFs not addressed?
Item ref.
Requirement or KF not addressed
Why is this gap significant for the applicant?
Optional: Overall Item Notes
Here you can capture a critical thought that relates to the assessment of the Item. It is not intended to be used to capture the full set of comments for the Item or key themes. You
will prepare your comments for this Item for the first time during the Consensus Review—not during the Independent Review.
13
Examiner’s Initials:
Applicant Number:
Item 6.1 Scoring
Factor
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Approach
No systematic approach to
Item requirements is evident;
information is anecdotal.
The beginning of a systematic
approach to the basic
requirements of the Item is
evident.
An effective, systematic
approach, responsive to the
basic requirements of the Item,
is evident.
An effective, systematic
approach, responsive to the
overall requirements of the
Item, is evident.
An effective, systematic
approach, responsive to the
multiple requirements of the
Item, is evident.
An effective, systematic
approach, fully responsive to
the multiple requirements of
the Item, is evident.
Deployment
Little or no deployment of any
systematic approach is
evident.
The approach is in the early
stages of deployment in most
areas or work units, inhibiting
progress in achieving the basic
requirements of the Item.
The approach is deployed,
although some areas or work
units are in the early stages of
deployment.
The approach is well
deployed, although
deployment may vary in some
areas or work units.
The approach is well
deployed, with no significant
gaps.
The approach is fully
deployed without significant
weaknesses or gaps in any
areas or work units.
An improvement orientation is
not evident; improvement is
achieved through reacting to
problems.
Early stages of a transition
from reacting to problems to a
general improvement
orientation are evident.
The beginning of a systematic
approach to evaluation and
improvement of key processes
is evident.
A fact-based, systematic
evaluation and improvement
process and some
organizational learning,
including innovation, are in
place for improving the
efficiency and effectiveness of
key processes.
Fact-based, systematic
evaluation and improvement
and organizational learning,
including innovation, are key
management tools; there is
clear evidence of refinement
as a result of organizationallevel analysis and sharing.
Fact-based, systematic
evaluation and improvement
and organizational learning
through innovation are key
organization-wide tools;
refinement and innovation,
backed by analysis and
sharing, are evident
throughout the organization.
No organizational alignment is
evident; individual areas or
work units operate
independently.
The approach is aligned with
other areas or work units
largely through joint problem
solving.
The approach is in the early
stages of alignment with basic
organizational needs identified
in response to the
Organizational Profile and
other Process Items.
The approach is aligned with
organizational needs identified
in response to the
Organizational Profile and
other Process Items.
The approach is integrated
with organizational needs
identified in response to the
Organizational Profile and
other Process Items.
The approach is well
integrated with organizational
needs identified in response to
the Organizational Profile and
other Process Items.
Learning
Integration
Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most
descriptive of the organization’s achievement level for the Item.
Item 6.1—Overall Score
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Item 6.1 Score
14
%
Examiner’s Initials:
Applicant Number:
Independent Review Worksheet – Sample
Item 7.2
4–6 most significant key factors (KFs) for this Item
1.
Key customers: Personal consumers (students, celebrities, preteens),
personal/business consumers (“outdoors people”), business consumers
(truckers, taxi drivers), business/government consumers (emergency service
workers) and government consumers (US Dept of Homeland Security).
Employees are also customers.
Customer requirements: Ease of use, Trendiness, Convenience, Reliability
(no dropped calls), low cost, safety and security, Ruggedness, Data and
voice capability, encrypted data and transmission. Requirements vary
among customer segments
3. Comparative data sources: Allegiance survey data, HDI, and
SooperdooperSoft.
4.
Primary competitors are the five largest cell phone manufacturers, several
manufacturers of integrated communication devices, and several dozen
competitors in the fragmented cell phone component and ringtone markets.
5.
Key success factors: Strong relationship with carriers, Rapid response to
marketplace, maintaining strong margins, managing the supply chain,
collaboration with suppliers/partners for innovation
6.
Main products/services: cell phone communications hardware and software,
including accessories, as well as cell phones, communications devices, and
ringtone software. Products and services use niche foci such as integration,
security, audio-centric interface, maximum mobility, and disabled
consumers.
2.
Item ref.
Key results—include figure references
(group the results as appropriate)
7.2a(1)
Customer Satisfaction &
1, 2, 3, + approaching + Improving or
+ Performs well
Dissatisfaction Fig. 7.2-1 to 7.2-6
4, 6
or at goal on
sustained positive against best
7.2-1 Allegiance Survey Overall
all figures
trends since 2003 competitor where
Satisfaction
presented
in all segments
presented
7.2-2 Allegiance Survey
+ most levels
 No comparisons
Agility/Response Time
> 4 on 5-point
on customer
7.2-3 Pulse Survey Results: Overall
Likert scale
complaints
Satisfaction
Group a set of
Summarize data in each evaluation factor
7.2-4 Customer Complaints
results, as
column – do not record exact
7.2-5 Ratio of Problem Calls to appropriate, rather
ratings/percentages for every result.
Positive Calls/Inquiries
than listing each one
7.2-6 Allegiance Survey
individually.
Ruggedness
Customer Perceived Value including 1, 2, 3, + all segments + improving or
+ better than best
Loyalty and other aspects of
4, 5, 6 at or above
sustained trends
competitor in all
Building Relationships
4.5 on 5-point in all segments
customer segments
Fig. 7.2-7 to 7.2-12
Likert scale in from either 2003
on Value and
7.2a(2)
Relevan
t KF
Levels
Trends (time frame,
direction, rate of
change)
15
Comparisons used and
appropriateness
Integration of key results
harmonized across processes (linked
to important segments/groups,
products/services, processes, action
plan requirements)
+ Data segmented by
customer
 Employees not broken out
as segment
 Complaints segmented by
customer not product; may
miss issues relative to one
product
+ Fig. 7.2-2 results aligned to
applicant definition of loyalty
+ Fig. 7.2-6 aligns to
customer requirement of
ruggedness
Examiner’s Initials:
Applicant Number:
7.2-7 Value
7.2-8 Allegiance Survey: Likelihood
to Refer
2007 for
Value and
Likelihood to
Refer
or 2004-2007
Likelihood to Refer
7.2-9 Percentage of Repeat
Customers
+ 96% repeat
customers in
2007
+ improving or
sustained trends
from 2003-2007
+ consistently
exceeds
performance of best
competitor from
2003-2007
 Not segmented by
customer group or product
line
7.2-10 Willing to be Contacted
+ >70%
customers
willing to be
contacted in
2007
+ improving
trends from 20032007
 No comparisons
presented
 Not segmented by customer
group or product line
7.2-11 Allegiance Survey Overall
Results 2007
7.2-12 Allegiance Survey Results –
Relationships and Competitive
Position
+ performing
at level > 4.5
in 10 of 11
satisfaction
dimensions
on Allegiance
survey
 No trending
data against
closest
competitor
provided for
either result
+ better than closest
competitor on 10 of
11 dimensions.
Equal to competitor
on 11th (battery)
+ better than closest
competitor for all
customer segments
Line up your observations across the
columns, as appropriate, so that other
Examiners can follow your evaluation.
Were any key requirements, relevant KFs, or key metrics identified by the applicant not addressed?
Item ref.
Relevant requirement, KF, or key metric not addressed
7.2a(1)
No results for the Won Business Survey.
7.2a(1)
Why is this gap significant for the applicant?
Determines why customers choose applicant, determines customer
expectations, examines customer satisfaction with the buying experience – all
indicators of future satisfaction and loyalty.
No results presented for Allegiance Survey question, “How This was reported as the approach to making improvements to the Allegiance
16
Examiner’s Initials:
Applicant Number:
can we better listen and learn of your needs and future
directions?”
Survey in Item 3.1. Results are reportedly reviewed during Improvement
Day.
Important to check back in the appropriate process
Items for expected results the applicant described.
7.2a (1)
and (2)
7.2a (1)
and (2)
Results not segmented by product line.
No results reported relative to ringtone business line.
Failure to evaluate results by segment may obscure problems that won’t
appear in aggregated results.
This segment represents $200 million in sales for the applicant.
Optional: Overall Item Notes
Here you can capture a critical thought that relates to the assessment of the Item. It is not intended to be used to capture the full set of comments for the Item or key themes. You
will prepare your comments for this Item for the first time during the Consensus Review—not during the Independent Review.
Although many results are segmented by customer groups, some are not (e.g., repeat customers & “willing to be contacted”)
17
Examiner’s Initials:
Applicant Number:
Item 7.2 Scoring
Guidelines
Levels
0–5%
There are no organizational
performance results and/or
poor results in areas
reported.
10–25%
A few organizational
performance results are
reported, and early good
performance levels are
evident in a few areas.
30–45%
Good organizational
performance levels are
reported for some areas of
importance to the Item
requirements.
50–65%
Good organizational
performance levels are
reported for most areas of
importance to the Item
requirements.
70–85%
Good to excellent
organizational performance
levels are reported for most
areas of importance to the
Item requirements.
90–100%
Excellent organizational
performance levels are
reported for most areas of
importance to the Item
requirements.
X
Trends
Trend data either are not
reported or show mainly
adverse trends.
Some trend data are reported,
with some adverse trends
evident.
Some trend data are reported,
and a majority of the trends
presented are beneficial.
Beneficial trends are evident
in areas of importance to the
accomplishment of the
organization’s mission.
Beneficial trends have been
sustained over time in most
areas of importance to the
accomplishment of the
organization’s mission.
Beneficial trends have been
sustained over time in all
areas of importance to the
accomplishment of the
organization’s mission.
X
Comparisons
Comparative information is
not reported.
Little or no comparative
information is reported.
Early stages of obtaining
comparative information are
evident.
Some current performance
levels have been evaluated
against relevant comparisons
and/or benchmarks and show
areas of good relative
performance.
Many to most trends and
current performance levels
have been evaluated against
relevant comparisons and/or
benchmarks and show areas
of leadership and very good
relative performance.
Evidence of industry and
benchmark leadership is
demonstrated in many areas.
Organizational performance
results are reported for most
key customer/patient/student,
market, process, and action
plan requirements, and they
include some projections of
future performance.
Organizational performance
results fully address key
customer/patient/student,
market, process, and action
plan requirements, and they
include projections of future
performance.
X
Integration
Results are not reported for
any areas of importance to
the accomplishment of the
organization’s mission.
Results are reported for a few
areas of importance to the
accomplishment of the
organization’s mission.
Results are reported for
many areas of importance to
the accomplishment of the
organization’s mission.
Organizational performance
results are reported for most
key customer/patient/student,
market, and process
requirements.
X
Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most
descriptive of the organization’s achievement level for the Item.
Item 7.2—Overall Score
0–5%
10–25%
30–45%
50–65%
X 70–85%
90–100%
Item 7.2 Score
18
70 %
Examiner’s Initials:
Applicant Number:
Independent Review Worksheet
Item 7.4
4–6 most significant key factors (KFs) for this Item
Item ref.
Key results—include figure references
(group the results as appropriate)
Relevant
KF
Trends (time frame,
direction, rate of change)
Levels
Comparisons used and
appropriateness
Integration of key results
harmonized across
processes (linked to
important
segments/groups,
products/services,
processes, action plan
requirements)
Were any key requirements, relevant KFs, or key metrics identified by the applicant not addressed?
Item ref.
Relevant requirement, KF, or key metric not addressed
Why is this gap significant for the applicant?
Optional: Overall Item Notes
Here you can capture a critical thought that relates to the assessment of the Item. It is not intended to be used to capture the full set of comments for the Item or key themes. You
will prepare your comments for this Item for the first time during the Consensus Review—not during the Independent Review.
19
Examiner’s Initials:
Applicant Number:
Item 7.4 Scoring
Guidelines
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
There are no organizational
performance results and/or
poor results in areas
reported.
A few organizational
performance results are
reported, and early good
performance levels are
evident in a few areas.
Good organizational
performance levels are
reported for some areas of
importance to the Item
requirements.
Good organizational
performance levels are
reported for most areas of
importance to the Item
requirements.
Good to excellent
organizational performance
levels are reported for most
areas of importance to the
Item requirements.
Excellent organizational
performance levels are
reported for most areas of
importance to the Item
requirements.
Trends
Trend data either are not
reported or show mainly
adverse trends.
Some trend data are reported,
with some adverse trends
evident.
Some trend data are reported,
and a majority of the trends
presented are beneficial.
Beneficial trends are evident
in areas of importance to the
accomplishment of the
organization’s mission.
Beneficial trends have been
sustained over time in most
areas of importance to the
accomplishment of the
organization’s mission.
Beneficial trends have been
sustained over time in all
areas of importance to the
accomplishment of the
organization’s mission.
Comparisons
Comparative information is
not reported.
Little or no comparative
information is reported.
Early stages of obtaining
comparative information are
evident.
Some current performance
levels have been evaluated
against relevant comparisons
and/or benchmarks and show
areas of good relative
performance
Many to most trends and
current performance levels
have been evaluated against
relevant comparisons and/or
benchmarks and show areas
of leadership and very good
relative performance.
Evidence of industry and
benchmark leadership is
demonstrated in many areas.
Results are not reported for
any areas of importance to
the accomplishment of the
organization’s mission.
Results are reported for a few
areas of importance to the
accomplishment of the
organization’s mission.
Results are reported for
many areas of importance to
the accomplishment of the
organization’s mission.
Organizational performance
results are reported for most
key customer/patient/student,
market, and process
requirements.
Organizational performance
results are reported for most
key customer/patient/student,
market, process, and action
plan requirements, and they
include some projections of
future performance.
Organizational performance
results fully address key
customer/patient/student,
market, process, and action
plan requirements, and they
include projections of future
performance.
Levels
Integration
Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most
descriptive of the organization’s achievement level for the Item.
Item 7.4—Overall Score
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Item 7.4 Score
20
%
Examiner’s Initials:
Applicant Number:
Independent Review Worksheet
Item 7.5
4–6 most significant key factors (KFs) for this Item
Item ref.
Key results—include figure references
(group the results as appropriate)
Relevant
KF
Trends (time frame,
direction, rate of change)
Levels
Comparisons used and
appropriateness
Integration of key results
harmonized across
processes (linked to
important
segments/groups,
products/services,
processes, action plan
requirements)
Were any key requirements, relevant KFs, or key metrics identified by the applicant not addressed?
Item ref.
Relevant requirement, KF, or key metric not addressed
Why is this gap significant for the applicant?
Optional: Overall Item Notes
Here you can capture a critical thought that relates to the assessment of the Item. It is not intended to be used to capture the full set of comments for the Item or key themes. You
will prepare your comments for this Item for the first time during the Consensus Review—not during the Independent Review.
21
Examiner’s Initials:
Application Number:
Item 7.5 Scoring
Guidelines
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
There are no organizational
performance results and/or
poor results in areas
reported.
A few organizational
performance results are
reported, and early good
performance levels are
evident in a few areas.
Good organizational
performance levels are
reported for some areas of
importance to the Item
requirements.
Good organizational
performance levels are
reported for most areas of
importance to the Item
requirements.
Good to excellent
organizational performance
levels are reported for most
areas of importance to the
Item requirements.
Excellent organizational
performance levels are
reported for most areas of
importance to the Item
requirements.
Trends
Trend data either are not
reported or show mainly
adverse trends.
Some trend data are reported,
with some adverse trends
evident.
Some trend data are reported,
and a majority of the trends
presented are beneficial.
Beneficial trends are evident
in areas of importance to the
accomplishment of the
organization’s mission.
Beneficial trends have been
sustained over time in most
areas of importance to the
accomplishment of the
organization’s mission.
Beneficial trends have been
sustained over time in all
areas of importance to the
accomplishment of the
organization’s mission.
Comparisons
Comparative information is
not reported.
Little or no comparative
information is reported.
Early stages of obtaining
comparative information are
evident.
Some current performance
levels have been evaluated
against relevant comparisons
and/or benchmarks and show
areas of good relative
performance.
Many to most trends and
current performance levels
have been evaluated against
relevant comparisons and/or
benchmarks and show areas
of leadership and very good
relative performance.
Evidence of industry and
benchmark leadership is
demonstrated in many areas.
Results are not reported for
any areas of importance to
the accomplishment of the
organization’s mission.
Results are reported for a few
areas of importance to the
accomplishment of the
organization’s mission.
Results are reported for
many areas of importance to
the accomplishment of the
organization’s mission.
Organizational performance
results are reported for most
key customer/patient/student,
market, and process
requirements.
Organizational performance
results are reported for most
key customer/patient/student,
market, process, and action
plan requirements, and they
include some projections of
future performance.
Organizational performance
results fully address key
customer/patient/student,
market, process, and action
plan requirements, and they
include projections of future
performance.
Levels
Integration
Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most
descriptive of the organization’s achievement level for the Item.
Item 7.5—Overall Score
0–5%
10–25%
30–45%
50–65%
70–85%
90–100%
Item 7.5 Score
22
%
Examiner’s Initials:
Application Number:
SCORE SUMMARY WORKSHEET
To enter data in this form, double click the worksheet. Enter the Item percent scores in column B. Do not enter data in any other
column. The worksheet will automatically calculate the appropriate scores based on the information you enter. When you are finished
entering scores, click outside the table to close.
SCORE SUMMARY WORKSHEET
Total Points
Percent Score
Summary of
Possible
0–100% (Stage 1—Use 5% Units)
Criteria Items
Column A
Column B
Category 1
1.1
70
1.2
50
Category Total
120
Category 2
2.1
2.2
Category Total
40
45
Category 3
3.1
3.2
Category Total
40
45
Category 4
4.1
4.2
Category Total
45
45
Category 5
5.1
5.2
Category Total
45
40
Category 6
6.1
6.2
Category Total
45
40
Category 7
7.1
7.2
7.3
7.4
7.5
7.6
Category Total
100
70
70
70
70
70
GRAND TOTAL (D)
Score
(AxB)
Column C
0
0
0
SUM C
0
0
0
SUM C
85
50%
85
20
0
20
SUM C
0
0
0
SUM C
90
0
0
0
SUM C
85
0
0
0
SUM C
85
70%
450
1,000
0
49
0
0
0
0
49
SUM C
69
D
23