2012 EXAMINER TRAINING Introduction to Evaluating a Baldrige Application Presented by The Granite State Quality Council and The Northern New England Alliance for Excellence 1 Agenda • Stages of a Baldrige Evaluation • The Evaluation Process – – – – Before you begin Key Factors Key Themes Evaluating each Item • Process Item Demonstration • Results Item Demonstration • Review of the Assignment 2 What is a Baldrige Application? • A structured response to the questions posed by the Criteria – May be for internal use or for state/national award • Contains: – – – – Eligibility forms Glossary of Terms Organizational Profile (5 pages) Categories 1-7 (50 pages) 3 The Role of a Baldrige Examiner • Provide objective feedback on Strengths and Opportunities for Improvement (OFIs) – First pass is based solely on the Application – A Site Visit may follow to verify and clarify feedback from Application 4 The Goal of an Application Evaluation • Unbiased feedback on Strengths and Opportunities For Improvement (OFIs) • Baldrige feedback is – – – – Non-prescriptive Non-judgmental Grounded in the Criteria & Core Values Weighted on the Organization’s Profile • Feedback may be used to determine award level 5 The Baldrige Application Evaluation Process - Stages Site Visit Comments Consensus Comments Stage 3 Stage 2 Independent Observations Stage 1 6 Goal: Value Added Feedback To the Applicant Application Evaluation Process 7 8 9 The Application Evaluation Process – Independent Observations Check for Conflict of Interest Evaluate Each Criteria Item Assemble Materials and Read Application Finalize Key Themes Observations Draft Initial Key Factors Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook 10 Contents of a Baldrige Application Case Study: Tillingate Living • Eligibility Information – Pages i – x • Additional Information – Org Charts, Glossary/Abbreviations (pages xi – xv) • Responses to Baldrige Criteria Questions – Preface: Organizational Profile (pages xvi – xx) – Categories 1 – 7 (pages 1-50) 11 The Application Evaluation Process Check for Conflict of Interest Evaluate Each Criteria Item Assemble Materials and Read Application Finalize Key Themes Observations Draft Initial Key Factors Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook 12 What you will use to Evaluate the Application Application Criteria On-line Worksheets No independent research or contact with Applicant 13 What you will use to Evaluate the Application 14 On-line Worksheets The Narrative Application Evaluation Process Check for Conflict of Interest Evaluate Each Criteria Item Assemble Materials and Read Application Finalize Key Themes Observations Draft Initial Key Factors Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook 15 Key Factors • What are they? – Significant attributes of an organization – that influence the way the organization operates • Where do they come from? – The Organizational Profile – Criteria Items – Eligibility Forms • Guidelines – – – – – Group under the five Areas to Address in the Organizational Profile Write in phrases rather than complete sentences Numbered list (for use in rest of analysis) Revise throughout the evaluation process Limit to 1–3 pages in length 16 How to Identify Key Factors Answers to “What” questions from the Organizational Profile Org Profile Criteria Items Key Factors 17 No Evaluation of Key Factors Key Factors: Example • Key Factors: • Not Key Factors: – Mission, vision, and values – Employee/staff profile – Customer and market segments and customer requirements – Competitive position and critical success factors – Strategic challenges – Governance Answers “What” -structure facts or attributes that affect the way the organization operates 18 – Strategic planning approach – Complaint management process – Rewards and Recognition System – Methods to promote and ensure ethical behavior Answers “How” processes that the applicant uses in managing its work Criteria Page 4 Case Study page xvi 19 20 The Key Factors Worksheet Key Factors: a bulleted list of Applicant responses to questions in the Organizational Profile. Key Factors Worksheet P.1a Organizational Environment • For-profit, privately held organization/Care Model - For-profit, privately held organization providing assisted living and skilled nursing care in 23 facilities located in Pennsylvania, Kentucky, Tennessee, and Virginia whose facilities are known for a sense of societal responsibility. Care model is centered on advancing seniors’ independence and quality of life utilizing the Aging Actively Consortium's (AAC’s) dimensions of wellness: emotional, intellectual/cognitive, physical, spiritual, social, professional/vocational, and environmental. • Mission, Vision, Values Mission (Figure P.1-2): Provide ageless care and timeless living to individuals in a homelike environment that supports their lifestyles and need for care with dignity and respect Vision (Figure P.1-2): Be among the top 10% of skilled nursing facilities (SNFs) and assisted living facilities (ALFs) and be a top choice for care Values (Figure P.1-2): Agility, Patience, Empathy, and Excellence (APEX) • Core Competencies (Figure P.1-2) •Designing, innovating, and managing facilities to support various lifestyles and deliver excellent clinical outcomes •Developing clinical and service competencies for a caring and exceptional staff •Designing and delivering rehabilitation services to support residents’ activities of daily living •Creating an educational environment to support a sense of mastery for residents 21 The Narrative Application Evaluation Process Check for Conflict of Interest Evaluate Each Criteria Item Assemble Materials and Read Application Draft Initial Key Factors Finalize Key Themes Observations Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook 22 What are Key Themes? A Summary of Key Points found in the Evaluation Key Theme: A high-level strength or opportunity for improvement that – – – – Is common (cross-cutting) to more than one Item/Category and/or Is significant in terms of the applicant’s key factors and/or Addresses a Core Value of the Criteria Must be traceable to comments found in the Item Worksheets As a whole Key Themes: – – Summarize how well an applicant has addressed the Criteria requirements, Key Factors, and Core Values Serve as an Executive Summary Importance: – – – Key Themes are usually the first and sometimes the only part of the feedback report read by senior leaders Used by examiners to focus site visit Used by judges to get a “big picture” of the Applicant 23 Organization of the Key Themes Worksheet A. What are the most important strengths or outstanding practices found in Process Items? B. What are the most significant opportunities, concerns, or vulnerabilities identified in Process Items? C. what are the most significant strengths found in its response to Results Items? D. What are the most significant opportunities, vulnerabilities, and/or gaps found in the Results Items? 24 Steps in Developing Independent Key Themes 1. Read application & capture initial impressions. 2. Evaluate application at the Item-level. 3. Key Themes developed at Consensus – Refined after Site Visit 25 Agenda Review Stages of a Baldrige Evaluation The Evaluation Process Before you begin Key Factors Key Themes Evaluating each Item • Process Item Demonstration • Results Item Demonstration • Review of the Assignment 26 Baldrige National Quality Program Developing Independent Review Observations Process Item Demo 27 The Application Evaluation Process Check for Conflict of Interest Evaluate Each Criteria Item Assemble Materials and Read Application Draft Initial Key Factors Finalize Key Themes Observations Draft Initial Key Themes Observations Complete Checklist and Assemble Scorebook 28 Evaluate Each Criteria Item Read Criteria Requirements Determine 4-6 Key Factors Develop ~6 Strengths Write 2 Feedback and OFI’s and Ready Comments Supporting Evidence 29 Read Applicant's Response Determine Score Step 1: Read Criteria Requirements 30 Item 5.1 Requirements Support Info: • Notes • Category & Item Description • Page 42 Page 18 in Criteria Booklet 31 Step 2: Key Factors 32 Item 5.1 Independent Review Worksheet – Key Factors From Key Factors Worksheet – Handout P.1a Organizational Environment P.1b Organizational Relationships What 4-6 Key Factors do we most expect to be addressed by this Item? P.2a Competitive Environment (or – which Key Factors do you most want to keep in mind as you evaluate this Item) P.2b Strategic Context P.2c Performance Improvement System 4-6 Key Factors are requested at the top of every Item Worksheet All Key Factors Item Requirements Item Key Factors 33 Item 5.1 Independent Review Worksheet – Key Factors 4-6 Key Factors are requested at the top of every Item Worksheet 34 Step 3: Read Application Item Response Categories 1-6 Demonstrate Processes “HOW?” 35 Read through the Item: Example - Item 5.1a(1) What does the Criteria ask? How does the Applicant respond? Case Study: Page 22 & 23 36 Page 62 in Criteria Book Definition – PROCESS • • • Linked activities with the purpose of producing a product or service for a customer (user) within or outside the organization Involves combinations of people, machines, tools, techniques, materials, and improvements in a defined series of steps Service situations - used more generally – • May spell out what must be done including a preferred or expected sequence Knowledge work (e.g., strategic planning, research, development, and analysis) – – does not always imply formal sequences of steps implies understandings regarding competent performance, such as timing, options to be included, evaluation, and reporting. 37 Analyzing a Process Item The purpose of Process Items is to permit diagnosis of an organization’s most important processes — the ones that contribute most to organizational performance improvement and contribute to key outcomes or performance results. 38 KEY Processes Read through the Item: Example - Item 5.1a(1) What does the Criteria ask? What Processes can you identify in 5.1a(1)? How does the Applicant respond? Case Study: Page 22 & 23 39 Step 4: Develop Around 6 Total Strengths and OFI’s and Supporting Evidence 40 Developing Strengths and OFIs • Strengths – Practices/processes or results to be reinforced & build upon • Opportunity for Improvement – Potential gaps, blind spots & vulnerabilities • Categories 1-6 reflect Processes • Category 7 reflects Results 41 First Pass: Focus on identifying… Main Idea Supporting Evidence A-D-L-I Strengths Relationship with Key Factors Relationship with Criteria 42 Gaps and OFIs 43 From our Example - Item 5.1a(1) 44 Example OFIs from - Item 5.1 45 Evaluating a Process Item • Processes include – – – – Approach (A) Deployment (D) Learning (L) Integration (I) • A-D-L-I is a tool to help in application evaluation and scoring • A-D-L-I is the P-D-S-A of High Performing Organizations 46 Approach • The methods and processes used by an organization to accomplish its work • HOW an organization addresses the item requirements • Approach considers… – – – – Appropriateness for organization Effectiveness Alignment with the organization’s needs Appropriate review and improvement • Approaches are systematic if they – – – – Are well ordered Are repeatable Use data and information so learning is possible Build in the opportunity for evaluation, improvement, and sharing thereby permitting a gain in maturity 47 Deployment • Refers to the EXTENT to which an organization’s approach is applied • The breadth and depth of the approach to relevant work units throughout the organization • Implementation on steroids… Is the approach deployed from the President to the janitor? 48 Deployment Deployment is process specific. Appropriate deployment varies from item to item Examples: • Category 1 – Leadership Processes – Deployed to workforce segments, stakeholders, other key processes • Category 3 – Customer Processes – Deployed to customer segments, work force segments • Category 5 – Workforce Processes – Deployed to workforce segments, locations, departments 49 Learning: Organizational and Personal • New knowledge or skills acquired through evaluation, study, experience, and innovation. • Criteria looks for both personal and organizational learning • There are three dimensions of organizational learning: – – – Continuous improvement of existing processes Innovation leading to breakthrough changes Knowledge sharing of such improvements and innovations 50 Integration • The harmonization of plans, processes, information, resource decisions, actions, results, and analysis to support key organization-wide goals. – Begins with alignment – Culminates in interconnectivity • Integration asks: – How well do the individual components of a system operate as a fully interconnected unit? – How well is the approach aligned with the applicant’s organizational needs? – How well is the approach integrated with these needs? 51 Indicating A-D-L-I A-D-L-I 52 Identifying Process Strengths & OFIs 1. Criteria – Strength? – Gap? 2. Key Factors – Importance & Appropriateness 3. A-D-L-I – Specifics 53 Step 5: Write 2 Feedback Ready Comments 54 Feedback Ready Comments • Turns Strength/OFI & Evidence into feedback the applicant understands – – – – Polite, Professional, Positive Not: Judgmental or Prescriptive Concise, one message per comment Indicates • Criteria-based strength or potential OFI • Potential significance of strength/OFI 55 Creating a Feedback Ready Comment: Strength 2,4,5,6 Systematic integrated approach to employee health & safety - Exec Director of each facility = risk management/safety officer, serve on system wide Safety Committee - Review all facility/system health safety, security policies & procedures - Meet monthly to review key measures; APEX Goals - Integrated with daily rounds with Collaborative Care Teams - Use of Data to drive improvements b(1) b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure workplace health, safety, and security. These methods include a system-wide Safety Committee that meets monthly to review progress toward goals, unexpected events, and best practices. Daily rounding by Collaborative Care Teams helps identify potential safety hazards. 56 Creating a Feedback Ready Comment: OFI 1,2,3, 4, 6 Approaches to management of workforce capability/capacity & to maintain a safe, secure & supportive work climate do not address all segments of workforce - Little discussion of physicians, volunteers, students a,b - -Unclear if security is an issue at some of the locations (e.g. SLFs) - Approach for capacity planning is somewhat unclear – they just post the positions and move people around? a, b It is unclear how the applicant manages volunteers, physicians, and students to fully support its work. For example, it is unclear how volunteers are trained and managed and how approaches are deployed to precepted students and credentialed physicians. Approaches for managing these workforce groups may help the applicant deliver high quality care and services and maintain a safe, secure environment. 57 b(1) A variety of methods and associated goals (Figure 5.1-2) help ensure workplace health, safety, and security. These methods include a system-wide Safety Committee that meets monthly to review progress toward goals, unexpected events, and best practices. Daily rounding by Collaborative Care Teams helps identify potential safety hazards. Polite? Professional? Positive? Nugget: Key point based on Criteria Judgmental Examples for clarity & context Indicates Relevance to this applicant 58 Prescriptive a, b It is unclear how the applicant manages volunteers, physicians, and students to fully support its work. For example, it is unclear how volunteers are trained and managed and how approaches are deployed to precepted students and credentialed physicians. Approaches for managing these workforce groups may help the applicant deliver high quality care and services and maintain a safe, secure environment.’ Polite? Nugget: Key point based on Criteria Professional? Examples for clarity & context Positive? Indicates Relevance to this applicant 59 Judgmental Prescriptive Step 6: Scoring 60 Scoring a Process Item • Two components – A-D-L-I – Level of Response to Item Requirements 61 Baldrige Scoring: Reflects Organization’s Maturity in the Journey • Level of Response to Item Requirements • Two dimensions of scoring – For Process Items (Categories 1-6) • Approach • Deployment • Learning • Integration – For Results Items (Category 7) • Levels • Trend • Comparisons • Integration 62 The Performance Excellence Journey Beginning On the Road Mature Anecdotal evidence, beginning of systematic approaches Clear evidence of effective, systematic approaches fully responsive to current and changing business needs Little or no deployment Approaches consistently deployed to all relevant work units Reaction to problems; early stages of an improvement system Fact-based, systematic evaluation and improvement process; strong refinement and innovation Little or no alignment of key processes and work units Approaches, processes, and work units fully aligned and integrated with organizational needs Maturity of Processes 63 Basic Requirements 64 Overall Requirements 65 Multiple Requirements 66 Item Format 67 Criteria – Page 29 UNDERSTANDING SYSTEMATIC APPROACH Page 68 - Criteria 68 UNDERSTANDING DEPLOYMENT 69 UNDERSTANDING IMPROVEMENT/LEARNING/INNOVATION 70 UNDERSTANDING INTEGRATION 71 Scoring Basics • Scores should align with comments. • Base the scores on a holistic view of the item and the applicant’s responses • Avoid requiring more evidence than is reasonable given the applicant’s 50-page limit. • From low to high, the scoring ranges reflect a growth in organizational maturity. • Note that the scoring ranges reflect a growth in maturity. • In the scoring ranges, note how the language for each scoring factor (ADLI) changes to reflect higher levels of organizational maturity • Tip: Start at the 50 – 65% band 72 Holistic Scoring Rubric based on Maturity • ADLI • Basic – Overall - Multiple 73 Master Scoring Grid Criteria Point Values Page 3 - Criteria Determined by Examiners 74 Baldrige National Quality Program Developing Independent Review Observations Results Item Demo 75 The Model for Performance Excellence is a Results Driven Model! 76 Definition of Results “Outputs and outcomes achieved by an organization in addressing the requirements of a Baldrige Criteria Item” 77 Results Results document how the applicant measures – its performance over time – against competitors’ performance – against relevant comparative data 78 Results Measure A-D-L-I… • Make note of those things that the applicant says are important and that the criteria calls for in Categories 1-6 • Check to see if they are reported in Category 7 79 Not all Results are created equal… • • • • Current Performance Segmentation Trends Comparisons 80 % Strongly Agreed Results - Current Levels 35 30 25 20 Us 15 10 5 0 2011 Employee survey question – I am satisfied with my job. 81 Current Levels and Segments Us - 2011 16 14 12 10 8 6 4 2 0 Group A Group B Group C Employee survey question – I am satisfied with my job. 82 Results - Trends 35 30 25 20 Us 15 10 5 0 2009 2010 2011 Employee survey question – I am satisfied with my job. 83 Results - Comparisons 35 30 25 Us 20 The Best 15 The Rest 10 5 0 2009 2010 2011 Employee survey question – I am satisfied with my job. 84 Results: Evaluate Each Criteria Item Read Criteria Requirements Determine 4-6 Key Factors Develop ~6 Strengths Write 2 Feedback and OFI’s and Ready Comments Supporting Evidence 85 Read Applicant's Response Determine Score Step 1: Read Criteria Requirements 86 Results Example – Item 7.3 Page 24 of Criteria Support Info: • Notes • Category & Item Description • Pages 47 - 48 87 Step 2: Key Factors 88 Item 7.3 Key Factors for Tillingate Living What 4-6 Key Factors do we most expect to be addressed by this Item? (or – which Key Factors do you most want to keep in mind as you evaluate this Item) From master Key Factor Worksheet 1. Mission, Vision, Values (Figure P.1‐2) ‐ Mission: Provide ageless care and timeless living in a homelike environment, dignity, and respect Vision: Be among the top 10% of Skilled Nursing Facilities (SNFs) and Assisted Living Facilities (ALFs) and be a top choice for care Values: Agility, Patience, Empathy, and excellence (APEX) 2. Employee and Workforce Demographics (Figure P.1‐3) ‐ 3,718 employees; 3. Workforce Groups: Employees, physicians, 700 volunteers; Engagement Factors ‐ Good work environment; Good benefits; Positive relationships with coworkers; Pride in the organization; A voice in resident care 4. Workforce Health and Safety Requirements ‐ Protection for exposure to communicable diseases; Protection form injury; Support in managing residents; Support for a healthy lifestyle; 5. Key Competitors: all other CMS 5‐Star‐rated facilities within 50 miles of each facility; 6. Sources of comparative data: CAHPS questions, CMS Nursing Home Compare, USSN Data Dispatcher, Caring Colleagues, NursQM 89 Step 3: Read Application Item Response 90 Results Example – Item 7.3a(1) What Results are provided FOR 7.3a(1)? 91 Step 4: Develop Around 6 Strengths and OFI’s and Supporting Evidence 92 Defining “Key” Results? “Key” refers to • elements or factors most critical to achieving the intended outcome In terms of results, look for those • responsive to the Criteria requirements • most important to the organization’s success • results that are essential elements for the organization to pursue or monitor in order to achieve its desired outcome 93 Key Results 94 Results evaluated by different factors Le-T-C-I: Let’s see: Evaluating outputs and outcomes Levels The current level of performance Trends • • Comparisons Integration Rate or slope of improvement Breadth or extent of deployment of results Performance relative to • appropriate comparisons (industry, similar organizations) • benchmarks or industry leaders Linkage of results to important requirements or processes • as identified in the Organizational Profile and Process Items Valid indicators of future performance Harmonized across processes and work units to support organization goals 95 Tillingate Living Item 7.3a(1) Strength Note that examiners found evidence outside of a(1) related to capability & capacity 96 Tillingate Living Item 7.3a(1) OFI 97 Step 5: Write 2 Feedback Ready Comments 98 Creating a Feedback Ready Comment: Results OFI a(1) The applicant has limited results for workforce capability and capacity which may challenge its ability to achieve strategic objectives and goals. For example, results on competencies or skill levels are not reported. Also, results for capacity are limited to ratios of nursing care time to health care outcomes. Additional capability and capacity outcomes may support advancing the quality of care and achieving top-decile resident satisfaction while achieving role-model financial stability. 99 a(1) The applicant has limited results for workforce capability and capacity which may challenge its ability to achieve strategic objectives and goals. For example, results on competencies or skill levels are not reported. Also, results for capacity are limited to ratios of nursing care time to health care outcomes. Additional capability and capacity outcomes may support advancing the quality of care and achieving top-decile resident satisfaction while achieving role-model financial stability. Nugget: Key point based on Criteria Polite? Examples for clarity & context Judgmental Professional? Positive? Indicates Relevance to this applicant 100 Prescriptive Creating a Feedback Ready Comment: Results Strength 101 Creating a Feedback Ready Comment: Results Strength a(3) Top-decile performance on employee satisfaction and engagement survey measures, as well as low vacancy and turnover rates (Figure 7.3-7), help the organization sustain its strategic advantage of high employee retention. Overall satisfaction has been at or better than the top-decile level since 2008 (Figure 7.3-4). Engagement results for "I am proud to work [here]" and "I would recommend [applicant] to family members“ survey items were at top-decile levels in 2010 and 2011 (Figure 7.3-6). Polite? Professional? Positive? Nugget: Key point based on Criteria Judgmental Examples for clarity & context Indicates Relevance to this applicant 102 Prescriptive Step 6: Scoring 103 Results Scoring Guidelines – Page 69 SCORING DIMENSION DESCRIPTOR LEVELS No results; poor results reported 0-5% Few results reported for Basic requirements Good results for Basic requirements Good results for Overall requirements Good to excellent results for Multiple requirements TRENDS Excellent results fully responsive to Multiple requirements 90-100% Trend data not reported or show adverse trends 0-5% Some trend data, with some adverse trends Some trend data, with majority presented as beneficial Beneficial trends in areas of importance Beneficial trends sustained over time in most areas of importance Beneficial trends sustained over time in all areas of importance 104 90-100% SCORING DIMENSION DESCRIPTOR COMPARISONS Comparative data not reported 0-5% Little or no comparative data reported Early stages of obtaining comparative data Some current performance levels evaluated against relevant comparisons Many to most trends evaluated against relevant comparisons or benchmarks; show levels of leadership and very good relative performance Industry and benchmark leadership in many areas INTEGRATION Results are not reported for areas of importance 90-100% 0-5% Results are reported for a few areas of importance Results are reported for many areas of importance Results reported for most key customer, market, and process requirements Results reported for most key customer, market, and process requirements and action plans Results and projections reported for most key customer, 105 market, and process requirements and action plans 90-100% Determine an overall scoring range and score 106 The Application Evaluation Process Check for Conflict of Interest Evaluate Each Criteria Item Assemble Materials and Read Application Draft Initial Key Factors Finalize Key Themes Draft Initial Key Themes (optional) Complete Checklist and Assemble Scorebook 107 Agenda Review Stages of a Baldrige Evaluation The Evaluation Process Before you begin Key Factors Key Themes Evaluating each Item Process Item Demonstration Results Item Demonstration Review of the Assignment 108 Materials for Assignment • Tillingate Living Case Study – GSQC Website • Criteria for Performance Excellence – http://www.nist.gov/baldrige/publications/upload/2011_2012_Health_ Care_Criteria.pdf • Worksheet Packet (e-mailed with slides, on websites) • Completed Key Factors • Blank Item 5.2 • Partially Completed Item 7.4 • Reference materials – Slides from today’s workshop(s) – Completed Item 5.1 109 The Assignment… • Read Tillingate Living Case Study • Review Key Factors Worksheet (Assignment Worksheets) – Any questions? Any changes? • Prepare Independent Review Worksheets for – Item 5.2: 4-6 Key Factors, ~ 6 Strengths/OFIs worksheets, Score. (2 comments are optional) – Complete Item 7.3: Review/Change Key Factors, ~ 6 Strengths/OFIs worksheets, Score. (2 comments are optional) • Plan on 5+ hours of work on Worksheets • Submit your completed Worksheets by October 22nd (anne@gsqc.com)110 Follow the 6-step process to complete Worksheets 111 Overview of mid-October Session • • • • Review and Refinement of assignment Questions and Clarification Overview of Scorebook Navigator tool Assignment for November 15th • Establish 2 hour webinar(s) for week of October 22nd….. 112 Final Questions & Feedback? 113