Postgraduate Diploma in General Pharmacy Practice Induction & Refresher session for Educational and Practice Supervisors Online presentation 4: Workplace-based assessment In this presentation we will cover 1. Philosophy of workplace-based assessment (WPBA) 2. Controversies around WPBA 3. Principles of WPBA in the diploma 4. Signposting to WPBA tools used in the diploma Note that assessments like MCQ/OSCE will be covered in a separate presentation What is WPBA? • Work-based learning inevitably leads to workplace-based assessment • Example: a practitioner meets a diploma learning outcome and a competency related to providing information to patients • They record evidence of this, e.g. a mini-CEX • Do they pass? Or fail? Or neither? What do we assess in the Diploma programme • In the diploma programme, we may assess knowledge or skills related to: – The General Level Framework competency framework – The curriculum guide/DAP learning outcomes Formative assessment and summative assessment • Formative assessment: – Assessment FOR learning • Summative assessment: – Assessment OF learning • The two are often confused, which causes anxiety and compromises the effectiveness of WPBA Controversies around WPBA • • • • • Confusion between formative vs summative Fear about the word ‘assessment’ Correct use of WPBA tools Adequate training to use the tools Progress decisions made on the basis of WPBA Formative vs. Summative in practice Compare driving lessons and driving tests (acknowledgement: General Medical Council and their partners) DRIVING LESSONS: •Learner drivers would not be happy if they did not receive feedback from their instructor (trainer) during and at the end of a driving lesson •It is not about pass/fail •Feedback matters more than the result (within reason, since there is a patient at the end of it) •This is assessment FOR learning DRIVING TEST: •When learners feel ready, they take a driving test •This is a pass/fail test •The result matters more than the feedback •This is assessment OF learning Evidence for WPBA • AMEE Guide No. 31 - NORCINI & BURCH (2007) • The research literature on work-based formative assessment and feedback suggests that it is a powerful means for changing the behaviour of learner • Several formative assessment methods have been developed for use in the workplace and there is preliminary evidence of their reliability and validity What about our practitioner who met a learning outcome/competency earlier? •Their mini-CEX was assessment FOR their learning. Academically, they won’t have ‘failed’ •Of course, an employing organisation can use formative assessments to identify poor practice What about competence and competency frameworks like the GLF? Competency frameworks such as the General Level Framework used in the diploma get mixed reviews: Favourable view “.. training programmes are required that both allow trainees to develop appropriate competences required to do their job. It was also accepted that, to learn, trainees need to be challenged in their use of knowledge, develop their skills and acquire the necessary competence.” Sir John Temple, 2010 Challenges ‘Tickbox’ mentality, misuse of frameworks (e.g inappropriate summative use), misunderstandings, seen as tedious General Level Framework New General Level Framework Delivery of Patient Care Personal Problem Solving Management and Organisation Patient consultation Organisation Gathering information Clinical governance Need for the drug Effective communication skills Knowledge Service Provision Analysing information Budget setting and reimbursement Providing information Organisations Selection of the drug Teamwork Drug specific issues Professionalism Provision of drug product Training Follow up Medicines information Monitoring drug therapy Evaluation of outcomes Staff management Procurement GLF Assessment Rating Consistently Demonstrates the expected standard of practice with very rare relapses 85-100% Mostly Implies standard practice with occasional lapses 51-84% Sometimes Much more haphazard than “mostly” 25-50% Rarely Very rarely meets the standard expected. No logical thought process appears to apply 0-24% 12 How should the GLF be used? •Formatively, to identify competence and need to develop •Use diploma WPBA tools, which are based on GLF competencies, as evidence against the GLF •Observational evidence is fine •Avoid seeking to just ‘tick the box’ of competencies met – look beyond this to the development associated with the competencies •Make it ‘aspirational’ rather than ‘impositional’ – it contains competencies expected of a band 6 pharmacist so it can be a valuable tool •As more competencies are met this will identify other areas to concentrate on •Use it over the 3 years of the programme – don’t rush it Evidence to support GLF JPB assessment tools can provide evidence of meeting GLF competencies, e.g.. mini-PAT, mini-CEX, CbD, MRCF Example: Delivery of patient care competencies: Competence: Need for drug Suggested assessment tools, areas(s) to collect evidence Relevant patient background Mini-CEX, CbD, MI enquiries From: Assessment Handbook Appendix 4 14 Principles of Diploma WPBA • • • • • • • • Formative Based on what you need them to be able to do – captured in the GLF and learning outcomes Practitioner-led with your support Aid practitioner development Help to identify “performance issues” early on Some subjectivity in assessment No ‘mark’ but good feedback essential Look for trends 15 What diploma practitioners want from their WPBA assessors Encourages and motivates Well informed about the assessment tools Impartial Listens Approachable Clear purpose Promotes reflection (Feedback) and action planning Provides direction when needed What WPBA assessors want from diploma practitioners Organised Knowledgeable Does not expect to be taught or directed Appreciates assessors’ time & input Uses assessors’ time efficiently Personable Visibly learns and develops Responds appropriately to feedback 16 Competency hierarchy (David Webb, Director of Specialist Pharmacy Services, East and South East England) Assessment Strategies Deployed Performance mini-CEX, mini-PAT, CbD, MRCF,DOPS Experience Bundled Competencies OSCE, simulation Integration Developed Knowledge, Skills and Abilities Education and training Innate Characteristics and Traits MCQ, MEQ DipGPP Assessment Extended intervention Formative WPBA mapped to GLF Mini-PAT MCQ Mini-CEX OSCE CBD ? Change to Supervised Learning Experiences Summative ? and MRCF Viva DOPS Assessments of performance Skills Timeframe Assessment/Activity Minimum Numbers 2 (4 & 8 months) 12 months RITA Meeting with Educational Supervisor. Mini-PAT 2* Sign off 12 month competency Mini-CEX 2 CbD 2 MRCF 2 DOPS 1 Log of Significant Interventions Extended Interventions illustrating the practice of the practitioner 3 Social Care 1 GPhC CPD records 12 GLF Evidence Summary Matrix GLF 1 1 signed off by ES at 12 months Learning Set Work (as described at Learning Sets) 18 months All of the above plus RITA 1 (15 months) Mini-PAT 1* Mini-CEX 3 CbD 3 MRCF 3 DOPS 1 Log of Significant Interventions Illustrating the practice of the practitioner 7 Extended Interventions 1 Social Care 6 GPhC CPD records GLF Evidence Summary Matrix GLF Learning Set Work (as described at Learning Sets) updated from 12 month portfolio 1 signed off by ES at 18 months 19 The Diploma Portfolio • • • • Assessment handbook says what should be in the portfolio, e.g. numbers of each assessment tool Don’t clutter with underpinning evidence Contents signed-off by Educational Supervisor and reviewed on assessment days at UCL Remember to use the assessment handbook 20 Diploma portfolio tension • There is a tension between needing to ‘pass’ a portfolio review when the contents of it are formative • Remember that the portfolio review involves assessment through questioning the practitioner on the day regarding their knowledge and skills • A practitioner can only proceed to a portfolio review if: – They have the minimum number of WPBA in it – The Trust is satisfied with work-based performance • We need to be honest about this tension • Medicine is looking at renaming some WPBA to try and resolve this. We may do the same Summary • We have looked at WPBA including what it is, some controversies, and how it is used for the diploma • We have outlined how the GLF should be used and how it might be misused • We have outlined the competency hierarchy and the range of WPBA in the diploma programme • We have looked at the portfolio review and the inherent tension to acknowledge • Please see if there are other online presentations that will help you and remember to refer back to the ‘Diploma Supervisor induction’ document