How do we get the best specialists? Professor Charlotte Ringsted, MD, MScHPE, PHD BMO Chair in Health Professions Education Research Director and Scientist, The Wilson Centre Department of Anesthesia University of Toronto and The University Health Network SCIENTISTS • Promote creative synergies between diverse theoretical perspectives, and between theory and practice Overview • Competency-, outcome-based education – Framework and Conception • Clinical training – Curriculum design Frameworks of competence • CanMEDS roles – Medical Expert – Health Advocate – Communicator – Collaborator – Manager – Scholar – Professional • ACGME competencies – Medical knowledge – Patient care – Interpersonal and communication skills – System based practice – Practice based learning and improvement – Professionalism The seven roles • EFPO project, 1992 – Undergraduate education, Ontario, society’s needs, eight roles • CanMEDs project, 2000 – Postgraduate education, RCPSC, entire Canada, seven roles Medical expert Communicator Collaborator Scholar Health advocate Manager Professional Whole person DK Canada and Denmark – Red and white; Neighbours; Hans Island 3 persons per km2 No 3 125 persons per km2 North America • “Assessment rich area” – National exams – Flooded by psychometricians – Heavy focus on reliability of tests and exams – Strong tradition of cognitive psychology and behaviourism SAQ CEX OSCE ITER CEX MCQ SAQ ESSAY MCQ OSCE Essay ITER Knowledge Skills Skills Knowledge WBA ITER Portfolio WBA ITER Profes sionalism Portfol io Professionalism B Hodges 2013 Competence as Specialist training Sausage Factory SausageasFactory Focus: Assessment and exams Denmark • “Assessment free area” – Focus on training programs and evaluation of education – No specialist exams and no psychometricians • “To emphasize the educational purpose of training, comprehensive formative evaluation is suggested as alternative to specialist examinations.” Karle, Nystrup ME1995 SAQ CEX MCQ OSCE Essay ITER Simulation Clinical training Logbooks Knowledge Skills Skills Knowledge National Courses Seminars Reading WBA ITER Rotations Programs Trainees’ evaluation of quality of program Profes sionalism Portfol io Professionalism Supervisor Appraisal meetings B Hodges/C Ringsted 2013 Competence as Specialist training Sausage Factory SausageasFactory Focus: Training and Evaluation DK reform: C/OBE and ITA PGME 1991 • NBH rules, guidelines • Goals and objectives PGME reform 2001 • NBH rules, guidelines • Goals and objectives – Specialist societies • • • • • • • Speciality courses Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation No exams – CanMEDS framework • • • • • • • Plus ’general’ courses* Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation In-training assessment WBA, In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003 Clinical skills assessments (12) Observation in vivo / vitro Longitudinal assessments •Cusum scoring •Logbook on experience •Learning portfolio Assessment based on practice data and written reflective assignments/reports •Communication skills (1) •Management/collaboration (2) •Academic competence (3) Factors related to value of ITA Ringsted et al. ME 2004, Med Teach 2003, ASS 2003 • The link to practice – Help in structuring teaching, training and learning • Outcomes clear, monitoring progress, identify problems • Coupling of theory to practice – Used as licence to practice rather than end-oftraining assessment • The effect on learning – Should include a challenge to the learner – ‘We all learn more’ • Assessors’ attitudes – Enthusiasm and rigour ITA-programs and psychometrics In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003 st 1 ar g ye nin i tra •Cusum scoring •Logbook on experience •Learning portfolio A challenge to psychoanalyse this Schuwirth & v.d. Vleuten ME 2006 A plea for new psychometric methods Future of Medical Education in Canada Toward a Competency-Based Approach Time Long DM, Acad Med 2000 Competency-based residency training – Reducing time from 3 years to 1½ year CanMEDS 2015 project • Hybrid of Time and Competence • In-training WPB assessment • EPAs and Milestones • Focus on “Intrinsic Roles” • Patient safety and interprofessional collaboration • Graded responsibility A call for systems-based education • Outcome-based curricula • Milestones, graded responsibility • Systems/society orientation • Teamwork within and across professions and institutions Current practice Future • Focus on individuals • Point-in-time sampling • Standardization • Focus on teams • Longitudinal WBA • Subjective, collective CLINICAL TRAINING A MATTER OF CURRICULUM DESIGN • The concrete task the near team – Patient consultation, ward round, amb., operation, diagn. investigation • The system context and the broader team – Primary, secundary, and tertiary sector and interplay within and between these plus other stakeholders • The wider context and the general perspective – The speciality/society, the profession, the region, the state, the society Specialist training • Experience and exposure – Time and volume • Professional development – Deliberation Oct 2013 Significant correlation between scores and complication rate Experience – number of procedures and years of practice Summary rating (1-5) Laparoscopic procedures Quartile 1 Quartile 2-3 Quartile 4 2.9 3.7 4.4* 53 96 157* 106 155 241* 137 110 123 111 98* 85* 11 9 11 Any procedure Time laparoscopic Time any Surgical practice (yrs) Experience and exposure Curriculum design • Logbook of experience – Help in designing the composition of the training program – Ensure breadth and depth in experience and exposure Experience is not enough Debilerate practice Guest et al, 2001, Coles 2002, Andersson, 2004 • Critical appraisal and reconstruction of practice instruction, monitoring, feedback and discussions, and opportunities to improve performance repeatedly Professional judgment • Not so much about finding the “right” answer but rather what is “best” in the situation. Coles 2002 • Ability to manage ambiguous problems, tolerate uncertainty and make decisions with limited information. Epstein and Hundert JAMA 2002 Performance Routine experts vs. Adaptive experts Expertise Most of us Innovative dimension ’Adaptive experts’ Efficiency dimension ’Routine experts’ Experience Ericsson, Guest et al., Choudhry et al. 2005 Schwartz et al. 2004 Self- regulation of learning and performance Zimmerman 2011 • • • • Self-regulated learning and performance Forethought Adaptation Evaluation • Characteristics – Motivation, proactive goal setting, strategic learning style, monitoring, adaptation, modelling learning environment, self-efficacy, assistanceseeking, - practice, practice, practice Thoracic surgeons – why and how did they learn a new procedure? • Video Assisted Thoracoscopic Surgery – New technique introduced in late 90’s – Henrik Jessen Hansen & René Horsleben Petersen • Jensen et al. studied why and how experts learn a new procedure – Interviews in 2011 with ten VATS experts/local pioneers Model – Experts learning VATS Selfdirected learning ”I didn’t learn it – I taught it myself” Self realisation Self-regulation of learning and performance Selfefficacy Motivation Incentive Quality Of care Monitoring outcomes Societybased coaching Social contagion Systems-regulation of learning and performance Social competition Jensen et al. 2012 Paper in progress Self – and system regulation Jensen et al 2012 Self – regulation System – regulation • Build on prior knowledge and skills of anatomy, disease, techniques, equipment • Highly creative in developing technique (’towel cover’) • Step-by-step approach, Zone of Proximal Development – time, elements, size and place • Organiational doubts and concerns; personal recognition • Finances, available equipment • Time constraints (the ’list’), co-workers (the team) • Expectations of patients and cospecialties • Monitor patient outcome • The concrete task the near team – Patient consultation, ward round, amb., operation, diagn. investigation • The system context and the broader team – Primary, secundary, and tertiary sector and interplay within and between these plus other stakeholders • The wider context and the general perspective – The speciality/society, the profession, the region, the state, the society Person-Task-Context TASK Simple ... complicated Part … Whole Performance PERSON Novice ... Advanced Knowledge, skills, experience CONTEXT Alone … Team Complexity Uncertainty Situated learning • Legitimate Peripheral Novice – Single task – Simple situation – Basic procedures Participation Advanced – Working context – Multi-professional teams – New procedures Professional develoment Dreyfus, Epstein & Hundert TASK Complex Atypical Year 4-5 TASK Simple Single Year 1 PERSON Adv. beginner Novice TASK Complicated Typical Year CONTEXT Complex systems Independent Supervising others PERSON Proficient Expert CONTEXT 2-3 Larger teams Distant supervision CONTEXT Small teams Close supervision PERSON Competent • The concrete task the near team – Patient consultation, ward round, amb., operation, diagn. investigation • The system context and the broader team – Primary, secundary, and tertiary sector and interplay within and between these plus other stakeholders • The wider context and the general perspective – The speciality/society, the profession, the region, the state, the society Integrating roles at 3 layers (EPAs) ROLES Medical expert Communicator Collaborator Advocate Manager Scholar Professional Layer 1 Layer 2 Layer 3 Roles at 3 levels (Milestones) Roles Med. expert Level 1 (Y1) Level 2 (Y2) Level 3 (Y 3-4) Green Green; Yellow Green; Yellow, Red Communicator Green Green; Yellow Green; Yellow, Red Collaborator Advocate Manager Scholar Professional Summary and conclusion Competence? • Competency = specific capability – ”Reflects expectations that are expressible in measurable behaviour; uses criterion standards for judging; informs learners and others about expectations” Albanese ME 2008 • Competence = holistic overall capacity – ”The habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served” Epstein and Hundert JAMA 2002 Future directions – the goals? Outcome goals Process goals • Clearly defined standards of performance • Training as Preparation for Future Learning (PFL) – – – – Checklists Competence cards Rating forms Quality of product Efficiency dimension – – – – Approach to the task Deliberation, reflection Adaptation to situation Critical re-construction Innovative dimension Schwartz 2004,2005 Coles 2002 Harden 1999 Curriculum design Plan and structure of the experience Appropriate level of difficulty Instruction and feedback LEARNING Repetition and correction of errors Critical appraisal of practice Questions and dialogue EDUCATION Curriculum design Med Educ 2011 • Careful and thoughtful planning of experience – Grade the tasks and responsibilities, acknowledge the contextual issues of learning • Coach – Stimulate innovative dimension and meta-cognition – as preparation for future learning • Critical appraisal of practice – own and general – Using paper assignments and students as resource Thank you for your attention ?????? Challenge in postgraduate education Postgraduate education Schoolbased Does Does Can Can Knows Undergraduate education Knows Workbased • Learn from managing cases • Learn how to manage cases •Reflect in and on practice Cultural dimensions CA DK SE NO 90 FI • Individualism 80 – ‘I’ vs. ‘We’ thinking 70 60 50 • Power distance 40 – Acceptance of hierarchies 30 20 10 0 IDV PDI Cultural dimensions CA DK SE NO FI 70 • Masculinity/Femininity 60 – Competition, ‘Be the best’, rewards for success 50 40 30 • Uncertainty avoidance 20 – Control of future, rules, principles, guidelines 10 0 MAS UAI ASSESSMENT EPAS AND MILESTONES CanMEDS framework in different contexts Final responsibility For patient care Training residents, students Supervision of residents Leader of individuals and teams Feedback Knowledge and skills EBM and up-to date Team work Management Time management Financial aspects Work in H organization Cultural dimensions Hofstede • Individualism – ‘I’ vs. ‘We’ thinking • Power distance – Acceptance of hierarchies 50 – Competition, ‘Be the best’, rewards for success PDI UAI 70 – Control of future, rules, principles, guidelines • Masculinity/Femininity DK 80 60 – (Short) Truth, quick results, normative NL 90 • Uncertainty avoidance • Long-term orientation CA 40 30 20 10 0 IDV LTO M/F Discussion • The importance of contextual aspects – – – – Cultural dimensions Working hours (48 vs. 37); Day-care facilities Age mean 36 (SD 4.0) vs. 44 (5.4) years Progressive independence of trainees appear to facilitate the transition North America: Entrustable professional activities And graded responsibility and Milestones – May be in conflict with organization and finansial models ITA-programs and psychometrics In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003 st 1 ar g ye nin i tra •Cusum scoring •Logbook on experience •Learning portfolio A challenge to do psycho..analysis of this Assessment of written assignment Write up a plan for this patient What if? Old vs. young Pregnant Breast-feeding Young woman Lower abd. Gen. anaesth. What if? Acute vs. chronic Diseases: liver, kidney, GI, CV, DM,psychiatric Explain changes in your plan Anatomy, Physiology, Pharmacology Assessment of written assignment • Reflection before and after case – Description of patient and operation – Theoretical and practical consideration regarding choice of anaesthesiological approach related to patient condition, wishes, surgery, and context – Describe potential problems and complications and discuss strategies to minimise these – Describe actual patient course and events – Reflection related to pre-operative considerations – Use references from literature in the reflection Trainees’ opinion of assessment (1-9) Ringsted et al. AAS 2003 10 3 14 8 6 14 lio fo rt po n. n ar io Le at ic un m g om lo C e nc rie pe g Ex in or sk 0) 2 um 8us (1 C . ss .a rit 7) W -1 16 .( 2) ab ol 1-1 C ( 15 14 14 14 N = 15 13 ls ki 13 .s lin C 0 11 5 4 10 6 2 2 About the written assignments “Extremely good learning experience - to do this review of a patient’s course ” “It was hard work” (Trainee) “This is really a valuable innovation in the education - these assignments” (Trainee) “It was more easy than I thought - to review these assignments” (Supervisor) “This is an advantage to the entire department - we all learn from these..” Kirsten Nørgaard, MHPE, 2004 Lessons learned • Outcome-based education – ‘CanMEDs roles’ is a nice mental framework. Need for both competency-goals (specific capability) and competence-goals (overall capacity) • In-training assessment programs – Meaningful programs are tailored to clinical context and trainees’ level of professional development, and drives learning in specialties’ weak areas. • The process – Useful to take a design-based research approach: Cycle of critical review of data (literature, quality of care reports, interviews); design; enactment, evaluation; and large working groups Mastery and Development Scoring Competency as capability related to specific tasks 9 8 7 6 5 4 3 2 1 Competence as holistic capacity related to any task Time 1. 2. 3. 4. 5. 6. No single method can measure it all – V.d .Vleuten 2010 assessment programs are recommended Defined by ‘supervision’ • Beginning • Surgery andsupervisory Anesthesiology (CA): – Difficulty despite efforts “We supervise them Developing – Needs supervisory assistance closely all the time!!” • Advancing – Often without supervisor • Internal Capable • medicine (DK): – Usually without supervisor “You Skillful mean observe them AMB care – unfamiliar cases – Always without supervisor watch what they are doing????” Defined by expectations to level Crossley et al., Med Educ 2011 • Below Foundation – Basic consultation skills, incomplete history be ‘conservative’ • Tend Level ofto completion of Foundation – Sound consultation skills, adequate history maintain status quo rather • Level of completion of early higher training than learning inhistory aspects – Gooddrive consultation skills, sound • not Levelintuitively expected during higher training emphasized – Excellent and timely consultation skills, comprehensive • Level expected on completion of higher training – Exemplary consultation skills, complex/difficult case Reliability Crossley et al, Med Educ 2011 Weiner et al 2010 From: Contextual Errors and Failures in Individualizing Patient Care: A Multicenter Study Ann Intern Med. 2010;153(2):69-75. doi:10.7326/0003-4819-153-2-201007200-00002 Data collection? 111 attending physicians Incognito patients presented biomedical and contextual red flags Responses to probing: •No complications •Biomedical complications •Contextual complications •Both types of complications Physicians probed fewer contextual (51%) than biomedical red flags (63%) Probing was necessary, but not sufficient for appropriate care Perspectives • Professionel competence – The habitual and judicious use of knowledge, skills, reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served • Approach to work – Ability to assess practice, question current practice – Life-long learning skills, search new information, critical appraise information and new technology and apply that in new practice – Ability to accept uncertainty and ambiguity, know your own limits, willingness to admit errors/mistakes and learn from these During rotation ITA Competence card Items •............X •............X •............X •............X •............X Score Signature Daily supervisor can be many different persons Daily supervisor Signing off Portfolio of competencies Competence card no. 6 •............ •........... •............ •............ •............ •............ X X X X X X Fine!! Let us discuss I’ll sign in Look! this competence the logbook Signature The trainee and the supervisor at the appraisal meeting Competence 1 Signature Competence 2 Signature Competence 3 Signature Competence 4 Signature Competence 5 Signature Competence 6 Signature Competence 7 .................. Competence 8 .................. Portfolio signatures Experience from internship Henriksen et al. UfL 2008 • ”We take it at the appraisal meeting – go over the list and then I sign. It is not like I observe what they are doing” • ”If he tells me he has done a procedure, I trust him and sign.” ”Its a bit like hunting for autographs” Perspectives • Professionel competence – The habitual and judicious use of knowledge, skills, reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served • Approach to work – Ability to assess practice, question current practice – Life-long learning skills, search new information, critical appraise information and new technology and apply that in new practice – Ability to accept uncertainty and ambiguity, know your own limits, willingness to admit errors/mistakes and learn from these Why in-training assessment? • Postgraduate education is work-based – 50% of the physician work-force are trainees – Quality of care relies on trainees’ competence during training – ”End-of training examination is like reading yesterday’s news” • In-training assessment, a tool for learning – Help clarify objectives according to broad aspects of competence (CanMEDS roles) – Stimulate deep learning – Support effective and efficient education Knowledge and skills • Causal understanding of concepts, principles, and tool design affects retention and transfer of learning Woods et al. 2006, 2007, Schwartz 2004 • Self-regulatory processes in development of expertise Zimmerman 2006 – Forethought: Task analysis, strategic planning – Performance: Contextual adaptation of strategies – Post-task: Evaluation and reflection Bech et al. EJVS 2010 Routine expert vs. Adaptive expert Performance Adaptive experts Routine experts Experience ”Most professionals reach a stable, average level of performance and maintain this mediocre status for the rest of their careers.” Ericsson, Guest et al., Choudhry et al. 2005, Schwartz 2004 Simulation training, clinical training, and follow up (Cusum-scoring) Pre-training in gastrointestional endoscopy utilizing computerized simulation Theory (mandatory) (Cotton & Williams, SADE’s textbook) Introduction (45 min) (Technical handling, structured instruction) Adamsen 2002 Pre-program scoring (”Cyber-scopy”) Gastroscopy: 20 scenarios Simulation program (2-3 days) pass pylorus retroflect • consecutive order total time < 15 min. • proceed only when all criteria met In-vivo Colonoscopy After Simulation Colonoscopy: 20 scenarios • last case with supervisor observing reach coecum < 15 min. excessive pressure < 5 x lost view < 5 x 4 Review of performance, post-program scoring Cusum score 3 (repeat program or part of program if needed) 2 In-vivo supervised endoscopy (same criteria as above) 1 Final pre-training evaluation 0 0 1 2 3 4 5 6 7 Scenario -1 8 9 10 11 12 13 14 15 Credentialling H:S PMI Postgraduate Medical Institute Copenhagen Hospital Corporation Training log (CUSUM score) Continuing reporting to project to evaluate impact of simulation Sven Adamsen Transatlantic comparison of the competence of surgeons at the start of their professional career M. P. Schijven et al. BJS 2010 Table 3 Canadian (84 hours) and Dutch (55 hours) residents on the four primary outcome measures Canadians Dutch t40 P CIP 0·52(0·05) 0·53(0·07) 0·18 0·856 PAME 0·85(0·06) 0·79(0·05) 3·90 <0·001 OSATS-C 0·78(0·06) 0·75(0·06) 1·33 0·192 OSATS-G 0·75(0·06) 0·74(0·07) 0·66 0·515 Values are mean(s.d.). CIP, Comprehensive Integrative Puzzle; PAME, Patient Assessment and Management Examination; OSATS-C, Objective Structured Assessment of Technical Skill checklist; OSATS-G, Objective Structured Assessment of Technical Skill global rating scale. System-based practice (manager) • Work effectively in various health care settings and systems. Coordinate patient care within the system • Consider cost and risk-benefit. Advocate for quality in patient care and optimal patient care systems • Work in inter-professional teams to enhance patient safety and improve quality. Participate in identifying system errors and implementing potential systems solutions Learning outcome and transfer • Defining and measuring learning Wulf & Shea ME 2010 – A relatively permanent change in a person’s capability to perform, must be demonstrated by retention or transfer tests • The concept of transfer Schwartz et al. 2004 – Assessment according to concept of preparation for future learning (PFL) rather than direct application according to an outcome-defined standard Challenges • Describing competence and outcome – Too detailed – Be able to manage - Lists of procedural skills, diseases (+300 competencies) – Too general - The ‘intrinsic’ roles - difficult to define • Disintegration of the concept ‘competence’ – Seven disciplines rather than an integrated, context-based concept of competence • Expectations at various levels of training? – EPAs and Milestones? Aspects of competence CanMEDS roles – Medical Expert – Health Advocate – Communicator – Collaborator – Manager – Scholar – Professional ACGME competencies – Medical knowledge – Patient care – Interpersonal and communication skills – System based practice – Practice based learning and improvement – Professionalism Confident Survey among Nordic junior Doctors N = 621 More than 24 months Why focus on theory and reflection? Klemola and Norros, ME 1997, 2001 Anaesthesiology – Clinical physiology and pharmacology; Procedural skills; Monitoring of respiratory and cardiovascular parameters; Context – patient, surgery, team Two distinct patterns related to ‘experts’ • Realistic orientation – Recognition of uncertainty and unpredictability – Communicative relationship: each patient is unique • Objectivistic orientation – No recognition of uncertainty and unpredictability – Authoritative relationship: ’a case’: coronary, asthmatic, etc Habit of action • Interpretative – Combine monitor information with situational information and background knowledge – Recognition of the versatility of information from several resources, oxygen SAT, End-tidal CO2, etc. Klemola and Norros, ME 1997, 2001 • Reactive – Operate directly with the numbers – Contradictory use of monitors, emphasising importance regarding patient safety without understanding the mediated character and versatility of information Knowledge and Anaesthsiology Klemola and Norros, ME 1997, 2001 • Forethought: physiological potentials – ”He can go uphill without getting out of breath, so probably he will tolerate anaesthesia well. Major problem might be oxygenation and ventilation.” • Adaptation: physio-pharmacological experiment – ”You can’t tell how an elderly patient will react. You have to check his responses and give drugs accordingly.” • Evaluation and reflection-in-action Flexner ? – ”The patient has capacity to compensate for side-effects of anaesthesia through sympathetic activation, a kind of capacity that elderly patients do not necessarily possess. That is a safe thing to observe” Influence of society of VATS • Societal contagion – Cohesion, close direct relation, ’friendship’ • Conversation, discussion (dyad system) – Structural equivalence, identically positioned, but not necessarily in direct contact • Competition, status (social system) • Both cohesion and structural equivalence – Inspiration, coach, competition » Burt 1987 on Coleman, Katz, and Menzel's (1966) Medical Innovation (Tetracycline) Experience is not enough Debilerate practice Guest et al, 2001, Coles 2002, Andersson, 2004 • Critical appraisal and reconstruction of practice instruction, monitoring, feedback and discussions, and opportunities to improve performance repeatedly Concrete experience New experience new situations LEARNING Abstraction, generalisation Observations and reflection Curriculum design Plan and structure of the experience New experience new situations Appropriate level of difficulty Concrete experience LEARNING Instruction and feedback Observations and reflection Abstraction, generalisation Repetition and correction of errors Critical appraisal of practice Questions and dialogue EDUCATION Training curriculum Work-based education Does Can Knows • Learn how to manage tasks – Preparation, instruction and feedback • Simulation • Leaning guides and ITA • Learn from managing tasks – Wide experience – Reflection in and on practice – own and unit’s pracitice