Michigan Quality Leadership
Award
New Examiner Training
2014
Agenda
•Welcome and Introductions
•Begin with the End in Mind
•Code of Ethical Conduct
•Key Factors
•Six-Step Process
•Evaluating Items
•Preparing for March 12
Clever Introductions
• Find a partner to introduce to the
class.
• Share name, affiliation, Baldrige-like
experience.
• Share another item of your choice, or
complete this sentence:
My zodiac sign is ______ and it
will help me be an awesome
Examiner because ……
Examiner Roles and Responsibilities
for Today’s Learning
• Take an active role in your own
learning by asking questions,
sharing information, viewpoints, and
roles during the training program.
• Take an active role in helping others
to increase knowledge and develop
skills.
Award Process Cycle
Application
Submission
Independent
Review
Not selected
Consensus
Review
Judges’
Meeting
Feedback
Report to
Applicant
Selected
Site Visit
Selected
Judges Name
Applicant an Award
Recipient
Judges’
Meeting
Not selected
Feedback
Report to
Applicant
Begin with the End in Mind
• Feedback Report
– Constructive, actionable feedback that helps
applicant
• Identify and leverage Strengths
• Prioritize and address Opportunities for
Improvement (OFIs)
Baldrige Criteria Framework:
A Systems Perspective
Ensuring Examiner Ethical
Behavior
A key success factor for the Award
program is promoting and ensuring
ethical behavior.
• Code of Ethical Conduct
• Conflict of Interest
Code of Ethical Conduct
Four Principles:
1. Protect the Integrity of the Award
Process.
2. Exhibit Professional Conduct at All
Times.
3. Protect the Promise of Confidentiality.
4. Protect the Program’s Intellectual
Property.
Conflict of Interest Statement
• Upon notification of application,
immediately review for conflicts of interest.
• Sign and email back the Conflict of Interest
statement.
• Call Geri Markley, Michigan Quality
Council, with all questions.
Exercise: Code of Ethics
In your small table groups:
• Assign roles.
• Discuss your assigned ethics
scenario.
• Prepare to share your consensus
response.
Scenario 1: the answer is ……
Examiners should not exploit their selection to the Board of Examiners.
There are no “secrets” for receiving the award and no guarantees for
receiving a site visit. Also, the program prefers to understate the
“winning” aspect of the process; it encourages a focus on continued
self-improvement and the sharing of best practices to help organizations
achieve performance excellence.
Examiners may not use the MQC logo in any advertising/promotion
(and business cards may not include “MQLA examiner” or the MQC
logo).
Principles: Protecting the integrity of the award process, the
program’s intellectual property, and the reputation of the BOE.
Scenario 2: the answer is ……
Any assistance with an organization’s application would
result in a conflict of interest for an examiner. The
examiner would be considered an employee of the
organization. If the organization reapplies, the examiner
would not be permitted to evaluate the application.
Examiners advising or participating with an
organization in preparing an award application shall not
reveal or discuss that participation with other examiners
during training or at any other time.
Principles: Protecting the promise of confidentiality
and the integrity of the award process.
Scenario 3: the answer is ……
No. At no time during Independent Review or
Consensus Review should an examiner contact the
applicant. Examiners shall not communicate with the
applicant organization or in any manner seek additional
documentation, information, or clarification. This includes
Internet searches or other external sources of
information.
Examiners shall safeguard the confidences of all parties
involved in the judging or examination of current or
former applicants.
Principles: Protecting the promise of confidentiality
and the integrity of the award process.
Scenario 4: the answer is ……
No. Examiners must avoid disclosures that may in any
way influence the award integrity or process, currently or
in the future, and treat as confidential all information
about the applicant. While this information has been
publicly released, it is not part of the application, and,
therefore, it is not relevant to the evaluation. While
potentially interesting, this information should not be
discussed and must not be considered in the evaluation.
The program shares with the team leader any known
information that could affect selection as a role model,
and the team leader seeks clarification during the site
visit.
Principle: protecting the integrity of the award
process.
Scenario 5: the answer is ……
Yes. Examiners may use the Internet and other sources
to familiarize themselves with common or generic
terminology. Examiners shall not communicate with the
applicant organizations or in any manner seek additional
documentation, information, or clarification specific to the
applicant. This includes Internet searches or other
external sources of information.
Principle: protecting the promise of confidentiality.
Master List of Key Factors
•
“Facts” about the organization
– Indicates relevance and
importance
•
Not processes or evaluation of
processes
•
Mainly from the Organizational
Profile
Examiners View Applicants
through Two Lenses:
Criteria and Key Factors
Examples of Key Factors
• Size
a) 39 employees located in one building
b) 73,000 employees located in twelve states
• Strategic Challenges
a) Heavily regulated and intensely competitive
b) Rapid expansion from acquiring many small
competitors
• Products
a) Highly technical and changes often
b) Has not changed in 37 years
Key Factors
•
How are key factors used?
•
Why are they important?
•
Case Study Key Factors
Exercise: Key Factors
In your small table groups:
• Assign roles.
• Identify key factors for your assigned
section of the Organization Profile.
• Prepare to share your key factors.
Six-Step Item Evaluation Process
Independent Review Steps
1.
2.
3.
4.
5.
Ground yourself in the criteria.
Select the relevant 4-6 key factors for the item.
Read the relevant section of the application.
Draft around 6 strengths & OFIs.
Write two feedback-ready comments.
 One strength
 One OFI
6. Determine the scoring range and score for the
item.
Step 1: Ground yourself in the
Criteria item.
Step 1 – Ground Yourself in
Criteria Requirements
• Read the Criteria item to understand the
Item requirements.
– Look at 1.1a(1) – Senior Leadership (p.7)
How do SENIOR LEADERS set your organization’s
VISION and VALUES? How do SENIOR LEADERS
DEPLOY the VISION and VALUES through your
LEADERSHIP SYSTEM, to the WORKFORCE, to KEY
suppliers and PARTNERS, and to CUSTOMERS and
other STAKEHOLDERS, as appropriate? How do
SENIOR LEADERS’ actions reflect a commitment to
those VALUES?
Step 2: Select the relevant key
factors for the item.
Step 2 –Select the Relevant Key
Factors for the Item
• Review master list of key factors;
determine the attributes of the
organization that would influence its
responses to the item requirements.
• Select the most relevant 4-6 key factors
for that item.
Example: 1.1a(1) would likely have a key factor
related to the organization’s vision and
values.
Step 3: Read the relevant section
of the application.
Step 3 – Read the Relevant Section
of the Application
• Identify the processes or approach the
applicant uses to meet item requirements.
• Flag, mark up and/or take notes as
needed.
– Item 1.1a(1) may describe an approach used
to set vision and values, how they are
deployed, etc.
Step 4: Draft around six strengths and
opportunities for improvement.
Step 4 – Draft Around Six
Strengths and OFIs
• Select relevant key factors for each
approach.
• Provide brief statement of approach.
• Provide evidence.
• Select appropriate evaluation factors
(ADLI).
• Provide reference to specific areas of the
item (e.g., a(1), b(3)).
Step 5: Draft feedback-ready
comments.
Step 5 – Draft one each FeedbackReady Strength and OFI
• Use Comment Guidelines to help craft
comments that capture your findings and
analysis.
• Include these key elements:
– Brief “nugget”
– An example or two
– Why this feedback is important
Step 6: Determine the scoring range
and score for the item.
Step 6 – Determine Scoring Range
and Score for the Item
•
•
•
•
Review balance of observations.
Use scoring guidelines (pp. 32-33).
Determine “best fit” scoring range.
Determine score.
Analyzing a Process Item
The purpose of process items is to permit diagnosis of
your organization’s most important processes:
the ones that contribute most
to organizational performance improvement
and to key outcomes or performance results.
Assessing Process Maturity
Process = the methods the applicant uses to address
item requirements in Criteria categories 1-6. Four
factors are used to evaluate process maturity:
• Approach
• Deployment
• Learning
• Integration
Approach
Approach refers to
• Appropriateness of methods
• Effectiveness of methods
• Degree to which the approach is
systematic (repeatable and based on
reliable data and information)
• Degree to which the approach addresses
basic, overall, or multiple requirements of
the criteria
Criteria Item Format
Item 1.1 – Approach (p. 7)
a.(1) How do SENIOR LEADERS set your
organization’s VISION and VALUES? How do
SENIOR LEADERS DEPLOY the VISION and
VALUES through your LEADERSHIP SYSTEM,
to the WORKFORCE, to KEY suppliers and
PARTNERS, and to CUSTOMERS and other
STAKEHOLDERS, as appropriate? How do
SENIOR LEADERS’ actions reflect a
commitment to those VALUES?
Deployment
Deployment refers to the extent to which
• The approach is applied consistently
• The approach is used by all appropriate
work units
• The approach is applied to all appropriate
areas, customers, suppliers, partners, etc.
Tip: Key factors are important in
assessing deployment.
Learning
Learning refers to
• Refining the approach through cycles of
evaluation and improvement
• Encouraging breakthrough change to the
approach through innovation
• Sharing refinements and innovations with
other work units and processes in the
organization
Process Maturity - Learning
• Three dimensions of organizational
learning:
– Continuous improvement of existing
processes
– Innovation leading to breakthrough
changes
– Knowledge-sharing of such improvements
and innovations
Integration
Integration refers to the extent to which
• The approach is aligned with
organizational needs.
• Measures, information, and improvement
systems are complementary across
processes and work units.
• Plans, processes, results, analyses,
learning, and actions are harmonized
across processes and work units to
support organization-wide goals.
Process Maturity - Integration
Alignment and Integration
Capturing Observations
D
U
o
Ty
se
u
pe
r
bl
e
S
tr
e
B
n
R
g
t
h
Description
The applicant has an integrated and systematic
approach for assessing its workforce capability
and capacity needs, including the skills,
competencies, certifications, and staffing levels.
Evidence
ADLI
- HR Capability Assessment is conducted annually
- The quarterly Capacity Study ensures a match between
position requirements, required competencies, and
qualified EOs for each position.
- The capabilities are a specific element of the strategic
A,D,I
(long-term) and action (short-term) plans resulting from
the SPP
- action (short-term) plans resulting from the SPP (Figure
2.1-1)
Item
Ref
a(1)
How do I evaluate Process Items?
By using the Six-Step Process!!
1. Ground yourself in the criteria
item.
2. Select the relevant key factors
for the item.
3. Read the relevant section of the
application.
4. Draft around 6 strengths and
opportunities for improvement
(OFIs).
5. Draft feedback-ready comments
(Two for IR).
6. Determine the scoring range and
the score for the item.
Independent Review
Exercise: Evaluate a Process Item
(Steps 1 - 4)
In your small table groups:
• Assign roles.
• Complete Steps 1 - 4.
– Ground yourselves in the Criteria by summarizing
– Identify 4 – 6 most important Key Factors
– Review the applicant’s response
– Identify around six observations with evidence
• Prepare to report out.
Well-Written Comments
Why are comments important?
• Feedback is the basis for improvement.
• They demonstrate understanding of the
applicant and its key factors.
What are characteristics of well-written comments?
• Concise
• Precise
• Show insight
• Actionable
Applicant Requirements
• A concise opening sentence that
expresses a single thought—the “nugget”
(or essence, or main point) of the
comment
• One or a few examples (not every
example)
• The relevance or importance of the
nugget to the applicant (think key factors)
Sample Comment
6.2b(1) The organization ensures that work
processes meet key requirements day-to-day
by utilizing technology for real-time monitoring
of patient length of stay, test results,
turnaround time, and bed availability. Other
methods of evaluation include day-to-day
rounding by administration, audits, and
scorecard reporting. This approach supports
the core competency of operational
excellence and helps address the strategic
challenge of cost control.
Comments Should Not
• Be prescriptive
• Parrot the application
• Go beyond the criteria
Exercise: Write a Comment (Step 5)
In your small table groups:
• Assign roles.
• Review observations.
• Select one and write a feedback-ready
comment.
• Prepare to report out.
Scoring System
• Each Item is assigned a point value.
• Scoring guidelines describe percentage
range of the points based on evaluation
factors.
• The Examiner selects a scoring range that
is “most descriptive” of the applicant’s
achievement level in that item.
• The Examiner selects the best score
within that range.
Exercise: Score (Step 6)
In your small table groups:
• Assign roles.
• Review observations and comment.
• Discuss and agree on scoring range.
• Agree on score.
• Prepare to report out.
How do I evaluate Results Items?
By using the Six-Step Process!!
1. Ground yourself in the criteria
item.
2. Select the relevant key factors
for the item.
3. Read the relevant section of the
application.
4. Draft around 6 strengths and
opportunities for improvement
(OFIs).
5. Draft feedback ready comments
(Two for IR).
6. Determine the scoring range and
the score for the item.
Independent Review
What Should You Be Looking For?
Let’s See – “LeTCI”
• Levels
• Trends
• Comparisons
• Integration
Importance
• What results are required?
• How do you know what results are
important to this applicant?
• What comparisons are appropriate?
• What segmentation should you expect?
Appropriate Comparisons
• Industry mean
• Best competitor
• Best-in-class performance
Rate of Improvement & Comparisons
14,000
12,000
Meaningful, Well-Written Results
Observations
Describe
• How you grouped various Key Results
• Whether you think Levels and Trends shown
represent favorable or unfavorable performance
and why
• Whether you think the Comparisons provided
are appropriate and represent favorable or
unfavorable performance and why
• Whether you think the results demonstrate
Alignment and Linkage to the applicant’s key
factors and key process requirements
Meaningful, Well-Written Results
Observations
• What Missing Results (Gaps) you noted
and why it matters to the applicant,
including expected results based on
Criteria, process, or other stakeholder
requirements
• What Overall Item observations you have
noted, if any.
Tip: Be sure to consider results
embedded in the text as well as the
charts and graphs provided.
Results OFI Comment
7.1a Comparisons to local competitors
are limited for customer outcomes,
although there are comparisons to top decile performance levels from national
or state databases. Regularly comparing
performance to others in the marketplace
may help the organization find opportunities
to outperform competitors and gain
market share, a key strategic challenge.
Exercise: Evaluate a Results Item
(Steps 1–6)
In your small table groups:
• Assign roles.
• Complete Steps 1-6 for assigned Item.
• Prepare to report out.
Preparing for March 12
Prework using Baldrige Online Scoring
System (BOSS)
Demo
Prework
• Read Collin Technology Case Study.
• In BOSS:
– Complete master list of key factors.
– Complete evaluation (Steps 1 – 6) for these
items: 1.1, 5.2, 6.2, 7.4.
• In Word:
– Write a brief description of these items (format
provided): 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 6.1,
7.1, 7.2, 7.5
• 5.1 and 7.3 were provided as examples.
Prework Tips
• Don’t procrastinate and expect to do it all at
once.
• Use notes from today to help you complete your
item evaluations.
• Contact Geri to take a look at your prework
progress and provide feedback or offer tips.
• Don’t spend more than 20 hours.
– If you’re at 16 hours and think you’ll go over,
contact Geri to help brainstorm ways to move
forward.
Resources
•
•
•
•
•
•
•
Participant Notebook
Online Baldrige presentations (prework)
Welcome to BOSS Powerpoint
Conference Call Feb 7 at 10:00
Examiner Preparation Team Mentors
Email Geri
BOSS monitoring and follow-up emails by
Geri
Questions?