2012
EXAMINER TRAINING
Final Workshop
November 15, 2012
Presented by
The Granite State Quality Council
and
The Northern New England Alliance for Excellence
1
White River Junction
VA Medical Center
Goal of this Training
• Provide each participant with a method and the tools to develop
Baldrige-based feedback for Award Applicants in New Hampshire,
Vermont, Maine, and other State Baldrige-based Programs.
• Provide each of you with a deeper understanding of Baldrige in
support of your organization & career
2
Agenda: Examiner Training – Final Workshop
• Introductions
• Putting faces with voices….
• Quick Review
• Independent Review Process
• The Consensus Process
• Creating Consensus Comments and Rationale
• Feedback (Applicant) Ready Comments
• Practice, practice, practice
• Examiner Roles, Ethics & Conflict of Interest
• The Organizational Profile Assessment
• After Consensus
– Key Themes
– Introduction to Site Visit
3
The Baldrige Application Evaluation Process
- Stages
Site Visit
Comments
Goal: Value Added
Feedback
To the Applicant
Stage 3
Consensus
Comments
Today’s Focus
Stage 2
Independent
Review
Stage 1
4
Review: The Application Evaluation Process
– Independent Review
Check for
Conflict
of Interest
Evaluate Each Criteria Item
Assemble
Materials
and Read
Application
Key Themes
Observations
Draft
Initial
Key Factors
Complete
Checklist and
Assemble
Scorebook
5
Independent Review Worksheet
Supporting Evidence
& Examples
Main Idea or Nugget
A-D-L-I
Or
Le-T-C-I
Strengths / OFIs
Relationship with
Key Factors
(Importance)
Relationship with
Criteria (traceability)
6
Independent Review Worksheets – Comments
& Score
7
Feedback Ready Comments
• Turns Strength/OFI & Evidence into
clear and actionable feedback for the
applicant
–
–
–
–
Polite, Professional, Positive
Not: Judgmental or Prescriptive
Concise: one message per comment
Indicates
• Criteria-based strength or potential OFI
• Potential significance of Strength/OFI
8
Elements of a feedback ready Comment using
N-E-R-D
Comments should include the following:
1. Nugget: Key point based on Criteria
2. Examples for clarity & context
3. Relevance to this applicant
Done!
No need to add additional information what
won’t add value to the applicant
9
Creating a Feedback Ready Comment: 5.1 Strength
2,4,5,6 Systematic
integrated approach
to employee health
& safety
- Exec Director of each facility = risk
management/safety officer, serve on system
wide Safety Committee
- Review all facility/system health safety,
security policies & procedures
- Meet monthly to review key measures;
APEX Goals
- Integrated with daily rounds with
Collaborative Care Teams
- Use of Data to drive improvements
b(1)
b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure workplace
health, safety, and security. These methods include a system-wide Safety Committee that
meets monthly to review progress toward goals, unexpected events, and best practices.
Daily rounding by Collaborative Care Teams helps identify potential safety hazards.
10
b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure
workplace health, safety, and security. These methods include a systemwide Safety Committee that meets monthly to review progress toward
goals, unexpected events, and best practices. Daily rounding by
Collaborative Care Teams helps identify potential safety hazards.
Polite?
Professional?
Positive?
Nugget: Key point based on Criteria
Judgmental
Examples for clarity & context
Indicates Relevance to
this applicant
11
Prescriptive
Creating a Feedback Ready Comment: 5.1 OFI
1,2,3, 4,
6
Approaches to
management of
workforce
capability/capacity & to
maintain a safe, secure
& supportive work
climate do not address
all segments of
workforce
- Little discussion of physicians, volunteers, students
a,b
- -Unclear if security is an issue at some of the locations
(e.g. SLFs)
- Approach for capacity planning is somewhat unclear –
they just post the positions and move people around?
a, b It is unclear how the applicant manages volunteers, physicians, and students to
fully support its work. For example, it is unclear how volunteers are trained and
managed and how approaches are deployed to precepted students and credentialed
physicians. Approaches for managing these workforce groups may help the
applicant deliver high quality care and services and maintain a safe, secure
environment.
12
a, b It is unclear how the applicant manages volunteers, physicians, and students
to fully support its work. For example, it is unclear how volunteers are
trained and managed and how approaches are deployed to precepted
students and credentialed physicians. Approaches for managing these
workforce groups may help the applicant deliver high quality care and services
and maintain a safe, secure environment.
Polite?
Nugget: Key point based on Criteria
Professional?
Examples for clarity & context
Positive?
Indicates Relevance to
this applicant
Judgmental
Prescriptive
Avoid starting OFI with the “While” or “Although” phrase
13
Agenda Review
 Quick Review
 Independent Review Process
 The Consensus Process
• Creating Consensus Comments and Rationale
• Feedback (Applicant) Ready Comments
• Practice, practice, practice
• The Organizational Profile Assessment
• Examiner Roles, Ethics & Conflict of Interest
• After Consensus
– Key Themes
– Introduction to Site Visit
14
The Consensus Process
Incorporating Team Feedback into
Comments
15
16
The Consensus Process
Creating Consensus Comments &
Rationale
17
What is Consensus?
The Gospel according to Baldrige states…
Consensus is a form of agreement reached by
a group of people resulting from discussion
and consideration of everyone’s ideas,
ultimately bringing the group to agreement
on a particular decision all can live with.
18
Why have Consensus?
• Resolve diversity in individual evaluation
–
–
–
–
Key Factors
Strengths and OFIs
Score
Key Themes
• Pooling of information adding value to overall
knowledge:
– All members exposed to pool of knowledge
– Achieving basic agreement on Strengths & OFIs
– Weigh overall findings to assign score
19
Why have Consensus? (cont.)
• Consensus comments used
• As basis for site visits
• For feedback report if no site visit
• Used by Judges in some cases to determine site
visit or site visit purpose
20
What Changes in Consensus?
• Item Leads:
– ~3 Criteria Items lead by each Examiner
– Incorporate the input of other team members
• 4-6 Key Factors
• Most important ideas/evidence
• Output
– Feedback Ready Comments & Rationale
– Score based on Consensus Comments
21
Creating Consensus Feedback
Examiner A
Independent Review
Consensus
Examiner B
Independent Review
Examiner C
Item Lead
Examiner C
Independent Review
Comments
Rationale
Examiner D
Independent Review
22
The Consensus Process
Consensus
Key Factors
Independent
Review
Item Lead creates
V2 Draft
Team Review
& Feedback
Examiners “send”
Independent Review
to Item Lead
Item Lead &
Back-up Identified
Back-up Review
& Feedback
(V3 Draft)
Consensus
Meeting
Item Lead creates
V1 draft
Consensus
Output
Consolidated
Consensus Review
Feedback Report
“Virtual” Consensus
23
24
“Virtual” Consensus Process
V1 Draft
1. Re-familiarize yourself with Criteria Requirements for
assigned item
2. Develop consensus 4-6 Key Factors
3. Re-familiarize yourself with Applicant Response
4. Aggregate all team member ideas/evidence
–
Determine most important for feedback
5. Draft ~6 Comments for each Item
–
Include Rationale for each comment
6. Propose Score based on Comments
25
“Virtual” Consensus Process
V2 & V3
1. Receive input from teammates
–
–
2.
3.
4.
5.
V2: Backup (and Team Leader)
V3: Full Team
Review Criteria Requirements
Verify/update 4-6 Key Factors
Review Applicant Response
Make Changes, Refine for Clarity
–
–
Read from the Applicant perspective
Update Rationale
6. Verify/update Score
26
Would you be
comfortable giving that
feedback to the applicant?
Consensus Meeting
• Entire Examiner Team participates
• Discuss all items
– Focus on topics Virtual Consensus did not close out
•
•
•
•
Key Factors
Key Themes
Strengths & OFIs by Item
Scores
• Team Leader sets agenda
– Item “Owner” leads discussion
27
Agenda Review
 Quick Review
 Independent Review Process
 The Consensus Process
 Creating Consensus Comments and Rationale
• Feedback (Applicant) Ready Comments
 Practice, practice, practice
• The Organizational Profile Assessment
• Examiner Roles, Ethics & Conflict of Interest
• After Consensus
– Key Themes
– Introduction to Site Visit
28
29
Same content as in Independent Review
Item lead reviews input of all Independent Reviews and
decides on Consensus Key Factors
30
Significant
?
“Double”
Criteria
Reference
Feedback Ready
Comment
Notes:
• Unused Strengths/OFIs & Rationale
•Enough detail to make a Strength/OFI
Capture all ideas from all
Independent Reviews on
Consensus worksheets
Rationale: Why is this a Strength/OFI?
•Examiners having same/similar Strength/OFI
•Examiners having conflicting Strength/OFI
•Indication of ADLI/LeTCI
•Notes to team members
•Not provided to Applicant
31
Score Range: Which of the 6
Scoring Bands best reflects
feedback?
Rationale: Use ADLI or LeTCI to
explain Range selected
Score Value: % Score within
the Band – 5% intervals
32
Round 1 Practice: Consensus
(Items 1.1, 1.2, 7.4)
As a TEAM:
1.
2.
3.
4.
Review Criteria Requirements
Verify/update 4-6 Key Factors
Review Applicant Response
Identify ~6 Total Strengths & OFIs
–
–
Use Blank Item Worksheet
Develop Rationale for each
5. Verify/update Score
Summarize your findings and share with group
33
Results vs. Process Items
Process Items: Strengths and gaps in “How”
• Approach, Deployment, Learning, Integration
Results: charts, graphs, tables, and text on
outcomes/outputs of processes
–
–
–
–
Importance to the applicant, alignment with key factors
Beneficial vs. adverse trends, inconclusive pattern
Performance relative to comparisons
Breadth of results: address key customer, market, and process
requirements; address appropriate segmentation for the item.
Comment on what results tell you that can be of benefit to
applicant’s improvement
– Levels, Trends, Comparisons, Segmentation, Importance
– What is missing??
34
Practice One - Process
Instructions
• Using a round robin approach, capture all the strengths and
opportunities that you noted or recall from your reading.
• Test each observation for its relationship to major ideas in
the Criteria, the key factors you identified, the value added
content, and its potential impact on the applicant’s desired
results.
• Draft around six strength and OFI “nuggets”.
• Prioritize your strengths according to their importance to the
applicant; repeat with your opportunities.
• Remember to take a holistic view of the applicant’s
responses.
35
Agenda Review
 Quick Review
 Independent Review Process
 The Consensus Process
 Creating Consensus Comments and Rationale
 Feedback (Applicant) Ready Comments
 Practice, practice, practice
• The Organizational Profile Assessment
• Examiner Roles, Ethics & Conflict of Interest
• After Consensus
– Key Themes
– Introduction to Site Visit
36
Writing Value-Added Comments
“Beyond Criteria”
ORGANIZATION
“Parroting”
ENVIRONMENT
CRITERIA
“Value-Added Comments
37
“Prescriptive”
Comment Fundamentals
• Comment Purpose: Communication
– Clearly identify most important Strengths and Opportunities
for Improvement (OFIs)
– Provide feedback in a non-prescriptive, non-judgmental
manner
• Comments are based on what the Criteria is asking and
what is important to the Applicant (Hint: Key Factors)
• Comment Components: clearly & concisely state
– What is Strength or Opportunity For Improvement?
– Why is Strength or OFI important to the Applicant?
38
NEW: Comment Components
Applicant Requirements
• Traceability back to Criteria
– Criteria references now listed in feedback
• A concise opening sentence
– Expresses a single thought—the “nugget” (or
essence, or main point)
• Some examples
No prescribed order
• The relevance or importance of the nugget to
the application
Can Applicant use each part of my comment?
39
Writing a Comment
Application
Criteria
Requirements
Organizational Profile
Item Responses
ADLI
LeTCI
your best
judgment
Comment
40
Call to
Action
Strengths and OFIs defined
• A strength is a
process, approach, or
result that supports
the applicant’s
achievement of its
desired results.
• An OFI is a process,
approach, or result
(or lack of one of
these) that creates a
vulnerability in
achieving the
applicant’s desired
results.
Give the applicant the benefit of the doubt when appropriate
(Your best judgment)
41
Comment Components:
The “Nugget”
Application
Criteria
Requirements
Organizational Profile
Item Responses
your best
judgment
The characteristics of a well-written comment
Precise
Concise
42
Shows insights
Sample “Nuggets”
Item 5.2
• a(3) The alignment of individual performance goals with strategic
goals contributes to the achievement of those goals.
• b(1) The applicant’s approach to assessing workforce engagement
is aligned with performance measurement and strategic planning.
• a, b, c It is unclear how the applicant applies workforce
performance management, engagement, and professional
development processes to volunteers, students, and credentialed
physicians/nurse practitioners.
• c(2) The applicant’s evaluation of the learning and development
system, which appears to be limited to a review of posteducation
performance, may not allow the organization to fully leverage its
investment in learning.
43
Comment Components:
Examples
Application
Criteria
Requirements
Organizational Profile
Item Responses
your best
judgment
44
Sample “Examples”
Strengths Item 5.2
• a(3) The alignment of individual performance goals with strategic
goals contributes to the achievement of those goals. The applicant
uses resident outcome data to drive high workforce performance by
linking outcomes to the performance evaluation and compensation
structure, which is unique in the industry.
• b(1) The applicant’s approach to assessing workforce engagement is
aligned with performance measurement and strategic planning.
APEX scorecard results that are considered indicators of workforce
engagement are reviewed regularly, and the organization addresses
key workforce engagement findings within strategic planning.
45
Sample “Examples”
OFIs Item 2.1
• a, b Tillingate Living’s strategic planning and objectives do not appear to
address all strategic challenges or balance all stakeholder needs. For
example, it is not clear how the strategic objectives (Figure 2.1-3)
address the challenge to integrate existing practices with ACOs or how
physician partners and suppliers are systematically included in the SPP.
Such gaps may prevent Tillingate Living from being a top choice for care.
•
a(1) It is not clear how Tillingate Living systematically identifies and
refines its core competencies. For example, how the core competency of
designing and delivering rehabilitation services was systematically
determined during step 3 of the SPP is unclear. Without a defined
approach in this area, Tillingate Living may miss the opportunity to
leverage these strategically important capabilities for future success.
46
5.2 Workforce Engagement – Tillingate OFIs
• a, b, c It is unclear how the applicant applies workforce performance
management, engagement, and professional development processes to
volunteers, students, and credentialed physicians/nurse practitioners.
Without engaging the entire workforce, the applicant may miss
opportunities to maintain its excellent reputation, especially related to the
key family requirement of an attentive staff.
• c(2) The applicant’s evaluation of the learning and development system,
which appears to be limited to a review of posteducation performance, may
not allow the organization to fully leverage its investment in learning. A
systematic approach in this area may help the applicant maintain its core
competency of developing clinical and service competencies for a caring and
exceptional staff.
OFI Examples help to clarify
exactly what the gap is
47
Comment Components:
Relevance/Importance
Application
Criteria
Requirements
Organizational Profile
Item Responses
your best
judgment
Comment Relevance demonstrates how well we know the applicant
48
Sample Comments
Item 2.1
• a(1) The five-month, 14-step strategic planning process (SPP) enables
Tillingate Living to learn about residents’ needs and expectations and
determine its strategic challenges and advantages. The SPP considers
resident needs and minimizes blind spots through an environmental
scan and SWOT analysis (Figure 2.1-2). The short- and near-term
horizons allow responses to senior living trends and corporate/facility
performance. Resident and Family Councils now give input to the
process.
Note: ADLI is not called out but evident
Relevance/Importance
49
Sample Comments
Item 2.1
• a, b Tillingate Living’s strategic planning and objectives do
not appear to address all strategic challenges or balance all
stakeholder needs. For example, it is not clear how the
strategic objectives (Figure 2.1-3) address the challenge to
integrate existing practices with ACOs or how physician
partners and suppliers are systematically included in the SPP.
Such gaps may prevent Tillingate Living from being a top
choice for care.
How can we make this relevance more motivating to act on??
50
Comments: Other Considerations
your best
judgment
Comment Guidelines
Comment
Can Applicant use each part of my comment?
51
Call to
Action
Sample Comments
Item 3.1
Note: no “while” or “although” - consider
“Can Applicant use each part of my comment?”
(is “while” or “although” value added?)
52
Remember: Results Items
•
•
•
•
Charts, graphs, tables, and text describing outcomes/outputs of
process
Importance to the applicant, alignment with key factors
Beneficial vs. adverse trends, inconclusive pattern
Performance relative to comparisons
Breadth of results: address key customer, market, and process
requirements; address appropriate segmentation for the item.
Comment on what results tell you that can be of benefit to applicant’s
improvement
– Levels, Trends, Comparisons, Segmentation, Importance
– What is missing??
53
Item 7.3 Workforce-Focused Outcomes Strengths
•
a(3) Top-decile performance on employee satisfaction and engagement
survey measures, as well as low vacancy and turnover rates (Figure
7.3-7), help the organization sustain its strategic advantage of high
employee retention. Overall satisfaction has been at or better than the
top-decile level since 2008 (Figure 7.3-4). Engagement results for "I
am proud to work [here]" and "I would recommend [applicant] to
family members“ survey items were at top-decile levels in 2010 and
2011 (Figure 7.3-6).
•
a(2) Most of Tillingate Living's reported workforce climate results have
improved over the past five or six years, which helps the organization
support an engaged workforce. Employee back injury results (Figure 7.32), which approached top-decile levels in 2010 and 2011, reflect
reductions in back injuries and workers’ compensation costs over the past
four years.
Bolded
Means
“- -”
What is the difference from a Process Comment??
54
Item 7.3 Workforce-Focused Outcomes OFIs
•
a(3) Tillingate Living does not present workforce engagement results by
service offering, facility, and state, and engagement results for volunteers,
credentialed physicians, and students are missing. Without results for all
segments of the workforce, the organization may be unable to improve
engagement and achieve its vision to be among the top 10% of facilities.
•
a(2) Tillingate Living does not report most workforce health, safety, and
security results. For example, results are missing for tuberculosis screenings,
injuries unrelated to resident care, and injuries from agitated residents (Figure
5.1-2). Without such results, Tillingate Living may limit its ability to ensure a
safe and secure environment.
•
a(1) Tillingate Living has limited results for workforce capability and capacity. For
example, results on competencies or skill levels are not reported. Also, results for
capacity are limited to ratios of nursing care time to health care outcomes. Without
measures in this area, Tillingate Living may be challenged in achieving its strategic
objectives and goals.
What is the difference from
a Process Comment??
55
Comment Guidelines
Dos and Don’ts
Do
 Keep the customer in mind.
 “Can the applicant act based on my comment?”
 Ensure relevancy by drawing linkages across items and to the
applicant’s Organizational Profile
 Ensure that the comment does not contradict other comments.
 Address your comments to the basic, overall, or multiple
requirements that are most important to the applicant.
 Use the evaluation factors (A-D-L-I or Le-T-C-I) to articulate the
areas of strength or opportunities for improvement.
. Write a unified, coherent, well-developed comment. Express the
comment’s single, main point in a topic sentence, followed by
concisely written evidence
56
Comment Guidelines
Dos and Don’ts
Do Not
☻Construct an exhaustive list of every method described
by the applicant that is related to your main point.
☻Go beyond the requirements of the Criteria or assert
your personal opinions.
☻Be prescriptive by using “must,” “should,” and
“would.”
☻Be judgmental by using terms such as “good,” “bad,” or
“inadequate.”
☻Comment on the applicant’s style of writing or data
presentation.
☻Parrot the Applicant/Application
57
Comment Style Guidelines
Do
• Use a polite, professional, and
positive tone.
• Use active voice and present
tense
– ‘completes’ rather than ‘is
completed
• Use “the applicant” or “the
organization”
• Until site visit tell what is
missing if something “is not
clear.” After site visit clarify
all “not clear” statements.
58
Do not
• Use jargon or acronyms
unless they are used by the
applicant.
Comment Litmus Test
•Traceable?
• Single “nugget”?
•Examples?
•Relevance/Importance?
•Follows Content
and Style
Guidelines?
•Call to Action?
Can Applicant use each part of my comment?
Ask yourself, “Can the applicant use each part of my comment?”
If the answer is no, consider some deletions or editing.
59
Round 2 Practice: Consensus – Feedback Ready
Comments (Items 1.1, 1.2, 7.4)
Remaining in your Team; swap items
1. Agree on most important Strength and OFI
2. Develop a feedback ready comment for each
–
–
Make any changes to Rationale
Identify “Item Ref” and if “++” or “- -”
3. Time permitting; repeat steps 1 & 2 with next Strength
& OFI
4. When finished, review comments of other Teams
Summarize your findings and share with everyone
60
Agenda Review
 Quick Review
 Independent Review Process
 Consensus
 Creating Consensus Comments and Rationale
 Feedback (Applicant) Ready Comments
 Practice, practice, practice
 Examiner Roles, Ethics & Conflict of Interest
• The Organizational Profile Assessment
• After Consensus
– Key Themes
– Introduction to Site Visit
61
The Role of a Baldrige Examiner:
Ethics & Code of Conduct
Four Guiding Principles:
– Integrity
• No conflicts of interest or anything that could skew
your evaluation
– Professional Conduct
• Truth, accuracy, fairness, respect, responsibility
– Maintain confidentiality
• Anything that could disclose applicant identity or
information
– Respect for Intellectual Property
62
Examiner Roles: Lead and Back-up
• Lead Examiner: Team Leader
• Back-up Team Leader
• Category/Item Leader
• Team Member: Everyone
63
Examiner Roles: Lead and Back-up
• Lead Examiner: Team Leader
– Work with the State/Local Program
• Develop timeline, site visit plan & manage site visit
• Finalize and Deliver feedback to applicant
– Guides examiner team
• Establishes category leads and due dates
• Leads consensus process, maintains focus
– Liaison with Applicant and Judges
• Back-up Team Leader
– Assists with Lead Examiner duties
– OTJ training to become future lead examiner
64
Examiner Roles: Category/Item Lead
• Typically assigned a Process Category and
corresponding Results Item
– Cat 1/Item 7.4; Cat 2/Item 7.5/7.1; Cat 3/Item 7.2; Cat
4/Item 7.1; Cat 5/Item 7.3; Cat 6/Item 7.1
• Team “expert” for assigned Items
– Collects team member IR for Virtual Consensus
– Drafts and finalizes comments
– Drafts & leads investigation of Site Visit Items
• Acts as back-up for additional category
– Typical pairings: Cats 1 & 2; 3 & 5; 4 & 6
65
Examiner Roles: Team member
• Review all assigned material
– Verify no conflict of interest & availability
– Complete individual review
• Serves as Category Lead & Backup
• Provides input and feedback
– Site visit plan
– Development and closure of all SVIs
• Conducts site visit
• Everyone is a team member
66
Agenda Review
 Quick Review
 Independent Review Process
 Consensus
 Creating Consensus Comments and Rationale
 Feedback (Applicant) Ready Comments
 Practice, practice, practice
 Examiner Roles, Ethics & Conflict of Interest
 The Organizational Profile Assessment
• After Consensus
– Key Themes
– Introduction to Site Visit
67
The Profile Level Recognition Process
Application Submission:
Baldrige Organizational
Profile plus 2-page
Action Plan
Phase 1 Analysis: 2-3 Weeks
•Examiners Independently Develop Clarifying Questions
•Consensus meeting generates request to Applicant
Phase 1
Applicant provides
requested information
to support Examiner
evaluation (2 Weeks)
Phase 2 Analysis: 2-3 Weeks
Examiners independently review Application, including responses to Phase 1
questions, to identify Strengths and potential Opportunities for Improvement
Final Consensus:
Team meets to develop final feedback
•Strengths and OFIs
•Key Themes
Feedback Review Meeting: Applicant and GSQC/NNEAE (optional)
68
Phase 2
Organizational Profile: Phase 1 Worksheets
69
Organizational Profile: Phase 2 Worksheets
70
71
Organizational Profile Assessment
Examiner Roles
• 2-4 Examiners
– Team Leader, Team Members
– No Item Leads
• 1 Consensus Call for Each Phase
• Generally no Site Visit
– May be consultative discussion of feedback
72
Agenda Review
 Quick Review
 Independent Review Process
 Consensus
 Creating Consensus Comments and Rationale
 Feedback (Applicant) Ready Comments
 Practice, practice, practice
 Examiner Roles, Ethics & Conflict of Interest
 The Organizational Profile Assessment
 After Consensus
 Key Themes
– Introduction to Site Visit
73
The 50-Page Application Evaluation Process
Check for
Conflict
of Interest
Evaluate Each Criteria Item
Assemble
Materials
and Read
Application
Draft
Key Factors
Key Themes
Complete
Checklist and
Assemble
Scorebook
74
The Process of Developing
Feedback
Scoring
Band
Themes
Scores
Comments
Scores
Comments
Independent Review
IR
Scores
Scores
Comments
Comments
IR
IR
IR
IR
IR
IR
75
Comments
IR
Independent Review
IR
IR
Independent Review
Scores
IR
Independent Review
IR
Key Themes
•
•
•
•
Serve as an executive summary; addressed to the applicant’s senior
leaders
Summarize the most vital issues for the organization as a whole, including
what the organization must do to remain or become competitive and to
ensure long-term sustainability.
Strategic in nature; may address how well the organization is managing a
major change or improvement; competitiveness or social issues; or
significant customer, market, product, or technological opportunities and
challenges.
Must be traceable to Item comments
76
Key Themes
• Developed by Team Leader or other Designated Examiner
– Input and review by all team members
• First Draft during Consensus Process
• Finalized at/after Consensus Meeting
• Revised and finalized during Site Visit
• Meet Comment Guidelines
77
Key Themes Worksheet
 Organize into four sections to address
a. most important strengths or outstanding practices
b. most significant opportunities, concerns, or vulnerabilities
c. Most significant strengths found in Results Items
d. most significant strengths, opportunities, vulnerabilities, and/or gaps
found in its response to Results Items
 2-3 pages in length
 Complete sentences and that meet the Comment Guidelines
 Delineate comments with bullets
78
Sample Key Themes: Process Strength
79
Sample Key Themes: Process
Opportunities, concerns, vulnerabilities
80
Sample Key Themes: Results Strengths
81
Opportunities, vulnerabilities, and/or gaps
Sample Key Themes: Results
82
Agenda Review
 Quick Review
 Independent Review Process
 Consensus
 Creating Consensus Comments and Rationale
 Feedback (Applicant) Ready Comments
 Practice, practice, practice
 Examiner Roles, Ethics & Conflict of Interest
 The Organizational Profile Assessment
 After Consensus
 Key Themes
 Introduction to Site Visit
83
Review:
The Baldrige Application Evaluation Process
Site Visit
Comments
Consensus
Comments
Stage 3
Stage 2
Independent
Observations
Stage 1
84
Goal: Value Added
Feedback
To the Applicant
Site Visit
85
Planning the Site Visit
Preparation, Preparation, Preparation!
• Your Input
– Consensus Feedback
• What are you looking for?
–
–
–
–
–
–
Accuracy (verify strengths)
Clarification of all areas for improvement
Breadth and depth of deployment
Integration
Maturity of systems
Linkages to key themes
86
Draft Site Visit Issues (SVIs)
• What are Site Visit Issues?
– Issues that need to be verified or clarified on site.
– Related to a key factor, making it relevant and important to
the applicant
– Linked to comments making each relevant and important to
the applicant
– Focus on a Process
• Normally 2-4 SVI’s per Item
87
Draft Site Visit Issues (SVIs)
• Verify Strengths
– The applicant’s approach, the extent of deployment of the
approach, and the results presented
• Clarify OFIs
– Things that were unclear or not addressed in the application
– Central to the Item requirements
– Relevant and important to the applicant’s organization
• All OFIs, significant strengths, and all strength
comments linked to a key theme must be verified or
clarified (things that have an impact on scoring)
88
Site Visit to Final Feedback
Input:
Consensus Comments
Output:
Verified & Clarified Comments
Feedback Report
Develop Site Visit Issues
Verify : “Walk the Wall”
Create SVI Worksheets for
Process and Results
Update Comments
Site Visit Comment Worksheets
Create Strategy for each SVI
Complete SVI Worksheets
Conduct Site Visit
Document Findings
•Scheduled Interviews
•Walking Around Questions
•Requested Document and Results
Denotes Worksheet
89
Summary: From Consensus to Final
Comments
Key Theme
Consensus
Comment
Consensus
Comment
Consensus
Comment
Site Visit
Issue
Final
Comment
Consensus
Comment
Consensus
Comment
Consensus
Comment
Site Visit
Issue
Final
Comment
Final
Comment
Key Theme
90
Final
Comment
Final
Comment
At Site Visit
• Professionalism is an absolute – they are watching your
every step….
• Try to understand their business
• Conduct interviews
– Explain the purpose of interview
– Say “Thank You”!
•
•
•
•
Don’t say “Great” or other potentially judgmental phrases
Review Documentation (on-site only)
Document observations in relation to issues
May record evidence of non-SVI issues, but… don’t go
looking
• Record Site Visit Issue Evidence and Conclusions
91
Agenda Review
 Quick Review
 Independent Review Process
 In Depth:
 Consensus Observations
 Observations to Comments
 Practice, practice, practice
 Examiner Roles, Ethics & Conflict of Interest
 The Organizational Profile Assessment
 After Consensus
 Key Themes
 Introduction to Site Visit
92
Agenda Review
 Quick Review
 Independent Review Process
 Consensus
 Creating Consensus Comments and Rationale
 Feedback (Applicant) Ready Comments
 Practice, practice, practice
 Examiner Roles, Ethics & Conflict of Interest
 The Organizational Profile Assessment
 After Consensus
 Key Themes
 Introduction to Site Visit
93
Complete NuGrain Packet
http://www.nist.gov/baldrige/publications/tillingate.cfm
Tillingate Living Case Study Packet 2012
Case Study
Scorebook – Examiner’s product
Feedback Report – Delivered to Applicant
94
Final Questions
&
Feedback?
On-Line Evaluation is coming
Check your E-mail
95