DESPFRWG Worked example – Combining 13/26 week outcome measures Worked example for combining 13/26 week outcome measures The following worked example provides a practical explanation of how the Star Ratings methodology would operate if the full, pathway and bonus outcome performance measures were combined (this would occur separately for 13 and 26-week outcome measures). Below is a demonstration of the six steps in the calculation process for a hypothetical Site A delivering DES-DMS during a ratings period. Six steps are used to calculate Star Ratings. The proposed method would have changes to two of the six steps currently used for these measures. The changes would occur in steps three and five of the calculation. Step 1 – Calculate the number of full, pathway and bonus outcomes achieved (unchanged). The measure of actual performance is simply the count of outcomes actually achieved. From the 20 participants on Site A’s caseload, the site achieved: 7 * 13 week full outcomes, 1 * 13 week pathway outcomes and 0 * 13 week bonus outcome. Step 2 - Calculate the expected number of full, pathway and bonus outcomes achieved. As with the current arrangements, this would involve separate regression models for full, pathway and bonus outcomes (unchanged). Based on the individual characteristics of the 20 participants on Site A’s caseload and the labour market conditions of the local area, Site A is expected to achieve: 5 * 13 week full outcomes, 2 * 13 week pathway outcomes and 0.5 * 13 week bonus outcome. Step 3 – Under current arrangements, an actual versus expected ratio is calculated for each measure individually. The new combined measure would compare the actual performance for each outcome type added together (i.e. full + pathway + bonus), with the expected performance for each outcome type added together (i.e. full + pathway + bonus). The outcomes will each need to be weighted according to the current performance measure weights. The actual performance for 13-week full outcomes is weighted by 20%, which is the current weighting for the 13-week full outcome measure. Likewise, the expected performance for 13-week full outcomes is also weighted by 20%. The actual and expected performance for pathway and bonus outcomes are each weighted by 5%, which is based on the current weightings for the 13-week pathway and 13-week bonus outcome measures. Page 1 of 3 DESPFRWG Worked example – Combining 13/26 week outcome measures Therefore, the actual versus expected equation will look like: 𝐴𝑐𝑡𝑢𝑎𝑙 𝑣𝑠 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 = (𝑎𝑐𝑡𝑢𝑎𝑙 𝑓𝑢𝑙𝑙 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗20%+(𝑎𝑐𝑡𝑢𝑎𝑙 𝑝𝑎𝑡ℎ𝑤𝑎𝑦 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5%+(𝑎𝑐𝑡𝑢𝑎𝑙 𝑏𝑜𝑛𝑢𝑠 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5% (𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑓𝑢𝑙𝑙 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗20%+(𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑝𝑎𝑡ℎ𝑤𝑎𝑦 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5%+(𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑏𝑜𝑛𝑢𝑠 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5% For Site A, the actual and expected results from steps 1 and 2 above are combined into one actual versus expected equation, with the weightings for DES DMS: 𝐴𝑐𝑡𝑢𝑎𝑙 𝑉𝑠 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 = (7) ∗ 20% + (1) ∗ 5% + (0) ∗ 5% (5) ∗ 20% + (2) ∗ 5% + (0.5) ∗ 5% 𝐴𝑐𝑡𝑢𝑎𝑙 𝑉𝑠 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 = 1.289 Step 4 - The performance measures are standardised to a 0 to 4 scale (unchanged). The un-standardised performance score for Site A for the combined 13-week outcome performance measure, which was equal to 1.289, might be equivalent to a standardised score of 2.6 (on a standard scale from zero to four). Note that the performance of Site A will still be above average, with the average being 2 on the standard scale. Step 5 - The performance measures would be weighted according to adjusted performance weights. Under the current weightings the 13-week full outcome measure has a weighting of 25%, the 13-week pathway outcome has a weighting of 5% and the 13-week bonus outcome measure has a weighting of 5%. Therefore, the total weighting for the new combined 13-week outcome measure is equal to 35% (25%+5%+5%). Using the same process, 26 week outcome measures have a combined weighting of 45% in DMS. The overall provider score for Site A will equal the sum of its score for each performance measure in the model, multiplied by the weighting of each KPI. So, if Site A received a standardised score of 2.6 for the combined 13-week outcome measure, then the contribution to the final score for this measure would be 2.6 multiplied by 35%. The overall score for Site A (in DMS) would equal the sum of: Performance Measure 1.1 Commencement to referral ratio 1.2 Time taken to achieve full 13 week employment outcomes 2.1 Job Placements 2.2-2.4 13 week outcomes 2.5-2.7 26 week outcomes 2.8 Ongoing Support DES DMS final Score Weighting Score 5% 2.2 5% 1.9 5% 2.1 35% 2.6 45% 2.7 5% 1.9 Weighted Score 0.11 0.095 0.105 0.91 1.215 0.095 2.53 Page 2 of 3 DESPFRWG Worked example – Combining 13/26 week outcome measures Step 6 - Allocate the final star ratings scores to bands (unchanged). Following the calculation of the overall weighted score, Site A has an overall score of 2.53 on a scale from zero to four. The average score for all providers turns out to be 2. In this situation, Site A has an overall score (‘star percentage’) that is 26.5% above the average ((2.53 – 2)/2). As a result, Site A will receive a 4-star rating, as the star percentage is between 20% and 40% above the average. Page 3 of 3