Guide to Planning and Implementing Independent Validation and Moderation of Assessment December 2013 NSW Public Sector ITAB Guide to Planning and Implementing Independent Validation and Moderation of Assessment Acknowledgements This Guide has been produced by the NSW Public Sector ITAB in conjunction with GE Consulting Pty Ltd, as part of a pilot project on Independent Validation of Assessment that was funded by the NSW Department of Education and Communities. The Guide draws on previous work in this area and acknowledges sources of documents and templates, where known. The NSW Public Sector ITAB would like to thank the project’s participants from the Water Industry for their contribution in shaping the contents of this Guide that can be utilised by RTOs from any industry. Guide to Planning and Implementing Independent Validation and Moderation of Assessment Table of Contents Section Purpose of Guide How to use the Guide 1 Page 1 1 Introduction to Independent Validation and Moderation of Assessment Integrating industry consultation with validation and moderation 2 2 Definitions 4 3 Benefits of Independent Validation and Moderation of Assessment 5 4 Timing of Validation Activities 6 5 Steps in the Validation/Moderation Process 7 6 Templates and Forms 1 – Validation and Moderation Schedule 2 – Validation and Moderation Plan 3 – Steps for Planning the Validation/Moderation Process 4 – Steps for Conducting/Participating in Validation and Moderation Activities 5 – Steps for Reviewing Validation/Moderation activity 6 – Validation of Assessment Tools 7 – Moderation of Assessment Evidence 8 – Validation of Assessment Process 9 – Checklist of what to organise when planning the activities 10 – Confidentiality Agreement 11 – Questions for Independent Parties 12 – Sample Student Questions 13 – Roles and responsibilities of Internal and External Parties 14 – Questions for evaluation of validation activity 13 14 15 16 18 Code of Professional Practice for RTOs and Independent Parties Participating in Validation and Moderation 39 7 3 19 20 24 27 30 32 34 36 37 38 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Purpose of the Guide This Guide has been designed for Registered Training Organisations (RTOs) to use and customise when undertaking independent validation and moderation of assessments. The purpose of the Guide it to: encourage you to recognise the benefits of adopting independent validation and moderation describe the process for the inclusion of independent (or external) parties in assessment validation and moderation enable you to reflect on your current practice provide you with suggested templates to use to plan and record the outcomes from independent validation and moderation of assessment How to use the Guide The Guide has a number of sections and depending on your needs you may not need to read the whole document. The sections include 1. An introduction to independent validation and moderation of assessments 2. Definitions 3. Benefits of independent validation of assessment and moderation 4. Timing of validation and moderation activities 5. Steps in the process 6. Templates and forms 7. Code of Practice for independent validation and moderation of assessment December 2013 Page 1 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Section 1: Introduction to Independent Validation and Moderation of Assessment Recent research has revealed that unlike other comparable countries, Australia’s vocational education and training (VET) sector does not have well-established internal and external processes for the validation and moderation of assessment. Given assessment practices have long been a concern in the VET sector, the Commonwealth Government is now moving towards developing a national model or a set of principles that can be adopted by the VET sector to ensure independent validation of assessment is undertaken by RTOs. Validation and moderation are essential components of a quality review process, with the outcomes informing the development and improvement of assessment practice. The input from independent parties is invaluable in improving the confidence of employers and students in vocational education and training. In simple terms, independent validation and moderation is really about: Engaging with stakeholders involved in the assessment process Ensuring that RTOs and/or assessors are being inclusive in the development and implementation of assessment Ensuring that what you are doing is meeting the client, learners and other stakeholders’ needs RTOs need to consider that validation is a planned, systematic and ongoing approach which could include multiple activities that captures the information you require. It may involve informal activities such as asking questions of supervisors or students at work placement or traineeship visits. It may also involve formal meetings with a group or panel of suitably experienced industry members or assessors outside of the RTO. However, all validation activities and outcomes must be documented, using the RTO’s agreed processes. Validation and moderation activities do not necessarily need to be undertaken in a face to face environment. This may not be possible for many independent parties. The use of technology in validation/moderation such as Skype, teleconferences and video links can be useful and provide a valid enabler for the activity. December 2013 Page 2 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Integrating industry consultation with validation and moderation It is not difficult to incorporate independent validation and moderation with your regular interaction with clients. You should be lateral with your thinking and questioning of managers, supervisors and employees. The templates in this Guide may assist you with your questioning. The flow diagram below illustrates how validation/moderation can be integrated with your industry consultation. December 2013 Page 3 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Section 2: Definitions Independent parties are those external to the RTO, therefore not employed or contracted by the RTO. External parties could include industry associations, small or large business, assessors from other RTOs or any unbiased or impartial third-party. Students can contribute to the validation and moderation process by providing feedback on the assessment tasks, assessment process and decision making (refer to Sample Templates). The importance is that independent representatives have the knowledge, skills and relevant experience in the area themselves. This experience can either be as an industry expert or a qualified VET assessor. Validation “… is a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes”. Ref: Code of Professional Practice for Validation and Moderation, p 7 Moderation “……is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.” Ref: Code of Professional Practice for Validation and Moderation, p 8 Ongoing, systematic approach Collecting evidence of validation can take many forms and include a number of activities. The key is to use a number of methods over a period of time, showing how they are used to improve the assessment practice within the RTO. This could include asking some key questions of supervisors or students in the workplace and documenting the information as part of the review process. Anyone within the RTO can collect this information using some of the suggested sample questions in the templates. December 2013 Page 4 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Section 3: Benefits of Independent Validation and Moderation of Assessment The key benefit of any validation and moderation process is that it is likely to strengthen the validity and consistency of assessment. For RTOs and assessors, it: Encourages consistent understanding of the competency standards and assessment practices and processes Allows the evaluation of the quality of the assessments Promotes discussion of issues of concern about the assessment process Provides for suggested improvements to the assessment system or processes Fosters the development of informal assessor networks and the exchange of information Provides for the development of benchmark performances (examples of what competent performance looks like) Increases confidence amongst assessors and RTOs in decision making Enables assessors to keep up-to-date about what is happening in their industry Enables RTOs and assessors to review their own practices and professional development needs For industry, it: Promotes consultation with RTOs Facilitates ongoing relationships with RTOS Facilitates partnerships with RTOs Engages industry in the assessment process Allows employers to have their needs addressed Promotes confidence amongst industry December 2013 Page 5 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Section 4: Timing of Validation Activities Validation may occur before, during and after assessment. Validation before assessment Involves assessors and independent parties collaborating together, concentrating on: Interpreting the unit/s of competency Coming to a common understanding of the standard to be achieved so agreeing on what a competent person would ‘look like’ The overall design of the assessment process; includes what evidence would be collected, where it would be collected and the number of assessments. This then informs; the development of the assessment tools which includes all student assessment tasks and assessors documents including evidence guides briefing relevant workplace personnel gathering of relevant workplace and/or industry documentation ask questions of independent parties before and during design (refer to Templates) You can involve one or a number of independent parties at this developmental stage, when gathering information about the industry. Validation during assessment May involve assessors and independent parties working in teams or panels or with a ‘lead’ assessor, concentrating on: The assessment process and the role of the assessor The actual performance of the student Validation after assessment Occurs when reviewing or comparing assessment tools, evidence and moderating assessment decisions, concentrating on: the effectiveness of the student assessment tasks and process the effectiveness of the assessors’ documents questions for independent parties (refer to Sample Templates) the validity of the evidence collected (moderation) the standard of performance achieved (moderation) the consistency and accuracy of decision making (moderation) the selection of a random sample of student assessments where a decision has been made (moderation) reporting and recording December 2013 Page 6 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Section 5: Steps in the validation/moderation process The size and staffing arrangements within an RTO will guide how the independent validation and moderation process is managed and implemented. Your RTO may be large enough that there is a person responsible for the managing and planning of the schedule and who oversees the process. This person would undertake the tasks in Table 1 which assumes a mix of planning and management responsibilities. If your RTO is small, the staff member overseeing validation and moderation may undertake a combination of the tasks in Tables 1, 2 and 3. What do you need to do? Tables 1, 2 and 3 on the following pages describe the steps required to plan, conduct and review the validation and moderation activities. Section 6 of this document provides these tables as checklists (Templates 3, 4 and 5 respectively) which may assist you through the process. December 2013 Page 7 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Table 1: Steps for Planning the Validation/Moderation Process Steps Responsibility Comments Sample Template (will depend on structure of the RTO) 1 Establish assessment validation/moderation schedule for the year Training Manager, Training Coordinator The schedule should be developed for a 12 month period and based on a ‘risk assessment’ approach. Ensure the schedule includes the random selection of students assessments across the units/qualifications 1, 3 2 Identify independent parties to be involved in the process Management, Assessors, Training Manager, Training coordinator Selection would depend on what units/qualification are being validated/moderated. Internal and independent parties must be involved and listed in the schedule. This may involve students, trainers/assessors, industry representatives, professional association representatives, assessor networks, small and large businesses or other relevant independent parties. 1, 3 3 Plan the activities using schedule in step 1 Training Manager. Training Coordinator Ensure it includes random selection of evidence for moderation 2, 4 4 Inform relevant independent parties of validation/moderation requirements Training Manager, Training Coordinator This would involve invitations being extended to all relevant independent parties. Include an explanation about their roles and responsibilities. 2, 4, 13 5 Review information already on file that has been gathered by others over time. Training Manager, Training Coordinator This is the information provided by independent parties that have answered the questions in an ongoing/systematic way. 11, 12 6 Ensure activities are undertaken and documented Training Manager Training Coordinator Documentation must be completed, signed and dated by all in attendance using the 2, 6-8 (relevant ones) December 2013 Page 8 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Steps Responsibility Comments (will depend on structure of the RTO) Sample Template Lead person (chair) organisational templates supplied 7 Review process and outcomes Training Manager Training Coordinator This could be undertaken with lead person (chair or independent participants) 5, 14 8 Ensure recommendations are reviewed and implemented where appropriate Training Manager Training Coordinator Completed templates have a follow up section which must be completed and checked to ensure there is a responsible person All relevant templates 9 Review schedule Training Manager, Training Coordinator Schedule should be reviewed at least yearly taking into account validation and moderation activities and results 1 December 2013 Page 9 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Table 2: Steps for conducting/participating in validation and moderation activities Steps Responsibility Comments Sample Template (will depend on the structure of the RTO) 1 Review plan developed from schedule and confirm appropriateness of planned activity Internal and independent participants This should be completed before the validation activity is conducted to ensure relevance. 2 2 Book venue and/or electronic equipment, arrange resources. Identify lead (chair) person for the validation activity Training Manager, Training Coordinator This could involve some participants meeting face-to-face, whilst others use technology for the same activity. Resources must include training package, AQF levels, relevant legislation/regulations that may impact, all assessment documentation, all assessment evidence. 9 3 Review assessment documentation forwarded by the RTO Internal and independent participants This ideally should be completed before activity to save time 3, 4, 9 4 Ensure independent parties are briefed about their role Lead person (chair) The lead person’s role is to ensure the activity remains on track and everyone has input into the discussion. Review roles and responsibilities of all participants. Ensure Code of Practice and/or Confidently agreement is in place 9, 10, 13 5 Ensure all documentation is available Lead person (chair) This would include training package, AQF levels, relevant legislation/regulations, workplace documents, all assessment tasks and tools (validation), all student evidence used to make the assessment decision (moderation). This also should include information gathered in answer to questions(ongoing/systematic) 3, 4, 9 6 Participate in activity Internal and independent participants This includes group agreement on how to work together (see attachment for further assistance) and clarifying roles of participants 5-8 December 2013 Page 10 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Steps Responsibility Comments Sample Template (will depend on the structure of the RTO) 7 Complete documentation as required Lead person (chair) Documentation must be completed, signed and dated by all in attendance using the organisational templates supplied. 2, 6-8 (relevant ones) 8 Undertake modifications as directed Nominated person Follow the RTO’s continuous improvement procedure 6, 7, 8 (whichever is relevant) December 2013 Page 11 Guide to Planning and Implementing Independent Validation and Moderation of Assessment Table 3: Steps for reviewing validation/moderation activity 1 Steps Who is responsible Comments Sample Template Collect feedback from participants regarding activities Lead person/chair Open discussion on how participants felt about the activity 5 , 14 What was learnt from the activity 2 Discuss how the next validation may be streamlined or improved Internal and independent participants This should be documented as part of the RTOs processes 5 , 14 3 Review results Internal and independent participants Open discussion 5 , 14 4 Provide feedback to RTO Lead person/chair Using RTOs documentation and processes RTOs processes December 2013 Page 12 Section 6: Templates and Forms December 2013 Page 13 Template 1 - Validation and Moderation Schedule Period from: To: This schedule should be completed by the RTO on a yearly basis and must include independent parties involved in any activity Training Package Title and Code Unit/s within this training package to be reviewed Activity (meeting, teleconference, video conference etc) Proposed dates Authorised Person’s Name: Role: Signature: Date: Review date: Person to undertake review: December 2013 Resources required Parties required (industry, other RTOs, internal assessors etc) Page 14 Template 2- Validation and Moderation Plan This plan is to be completed by the RTO for each validation and moderation activity. RTO Name and National ID: Date: Training Package Title and Code: Unit/s within this training package to be reviewed Focus of the activity (assessment process, tools evidence, decision making) Methods/approaches to be used (meeting, teleconference, video conference etc) Parties involved (industry, other RTOs, internal assessors etc) Name of Person Responsible for Plan: Role: Signature: Date: Resources required Documents to be used to record the activity Name of RTO Manager/coordinator: Signature: December 2013 Page 15 Template 3 - Steps for Planning the Validation/Moderation Process This checklist can be used by the RTO to assist when planning the validation and moderation process. It can be used as a guide and as evidence the process the process has been managed. Refer to Table 1 in the Guide to Planning and Implementing Independent Validation and Moderation of Assessment. Steps Has the following been completed? 1 Establish assessment validation/moderation schedule for the year 2 Identify independent parties to be involved in the process 3 Plan the activities using schedule in step 1 4 Inform relevant independent parties of validation/moderation requirements 5 Review information already on file that has been gathered by others over time 6 Ensure activities are undertaken and documented 7 Review process and outcomes 8 Ensure recommendations are reviewed and implemented where appropriate 9 Review schedule December 2013 Yes No Comments/actions to be taken Page 16 Steps Has the following been completed? Yes No Comments/actions to be taken Name and signature of person completing the checklist: Date: December 2013 Page 17 Template 4 - Steps for Conducting/Participating in Validation and Moderation activities This checklist can be used to assist when conducting and participating in the validation and moderation process. It can be used as a guide and as evidence of undertaking the process. Refer to Table 2 in the Guide to Planning and Implementing Independent Assessment Validation and Moderation. Steps Has the following been completed? 1 Review plan developed from schedule and confirm appropriateness of activity 2 Book venue and/or electronic equipment, arrange resources. Identify Lead Person(chair) for the validation activity 3 Review assessment documentation forwarded by the RTO 4 Ensure independent parties are briefed about their role 5 Ensure all documentation is available 6 Participate in activity 7 Complete documentation as required 8 Undertake modifications as directed Yes No Comments/actions to be taken Name and signature of person completing the checklist: Date: December 2013 Page 18 Template 5 - Steps for reviewing validation/moderation activity This checklist can be used to assist when reviewing in the validation and moderation activity. It can be used as a guide and as evidence of undertaking the process. Steps 1-4 are to be undertaken as a discussion and led by the ‘chair’ or the person ‘facilitating the activity’. It is not a ‘feedback form’, it is to be used to generate the open feedback. Ensure the discussion is documented and reviewed. Steps 5 and 6 relate to actions taken by the RTO after the validation activity to ensure recommendations are implemented. Refer to Table 3 in the Guide to Planning and Implementing Independent Assessment Validation and Moderation. Steps Has the following been completed? 1 Collect feedback from participants regarding activities 2 Discuss how the next validation may be streamlined or improved 3 Review results 4 Provide feedback to RTO 5 Ensure recommendations are reviewed and implemented where appropriate 6 Review schedule Yes No Discussion/feedback Name and signature of person completing the checklist: Date: December 2013 Page 19 Template 6 - Validation of Assessment Tools Comments about this template Names of validation activity participants and their role eg external RTO and/or assessor, industry and VET person Use this document when validating internally within your RTO and/or externally with other RTOs and industry representatives who have technical expertise in the areas covered by the unit/s. All participants may be issued with this form but one person must be responsible for completing the form for the RTO. 1.______________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ Date of activity Unit/s of competency List all assessment documents reviewed in the activity. Note all assessment tasks, tools; assessor’s documents must be available. List all support documents referred to today Recommendations Explanation of recommendations: Modification required YES NO Assessment Tools ready for use YES NO Modifications: Name of person who is to complete modifications Date modifications to be completed by December 2013 Page 20 Signatures: 1: 2: 3: Modifications completed Date /Signature of authorising officer December 2013 Page 21 Clearly describe in the action column, the adjustments that are required to the tools. Principles of assessment Criteria met? Validity Yes Comments/ Actions No Tools assess ALL components of the unit including knowledge, skills and critical aspects? Tools address the level of performance required by unit and the AQF? Tools address task skills, task management skills, contingency management skills and role environment skills Assessment tasks relate to real work place contexts and situations? Assessment tasks have been designed to allow holistic and integrated assessment of knowledge and skills? The tools demonstrate a variety of evidence over time and contexts? Tools reflect the literacy and numeracy requirements of the unit? Reliability Tools consistently measure against the unit AQF level, and workplace context? Instructions ensure consistent application of the tools by assessor, workplace supervisors (where applicable) and candidates? Tools achieve consistent results in different situations and with different candidates? Assessment exemplars, model responses, checklists etc have been prepared for use by assessors? There is sufficient space for the assessor to write comments on the assessment tools for future reference in decision making Flexibility The tools allow for flexibility in the assessment to meet the needs of the candidate and workplaces? December 2013 Page 22 Fairness Tools meet the needs of the candidate in various contexts? Tools are free of bias? Written information is worded appropriately to the requirements of the units of competency and the context for assessment, including candidate needs? The level of difficulty of the activity is appropriate to the competency or competencies and qualification being assessed? Assessment conditions are clearly defined? Rules of Evidence Validity – Tools cover all components of assessment and consider the workplace context, and candidate? See also criteria above SufficiencyTools allow sufficient evidence of skills and knowledge to be demonstrated over a period of time? Authenticity – Tools ensure that the evidence gathered is the candidate’s own work? Currency Tools allow candidate to demonstrate currency industry / work based competency? Additional Questions Can these tools be easily used by another industry assessor? Will these tools allow for consistent decisions to be made? Source: Adapted from: Assessment validation and continuous improvement workshop guide, DEC NSW, 2012 December 2013 Page 23 Template 7- Moderation of Assessment Evidence When to use this template Use this document when moderating internally within your RTO and/or externally with other RTOs and industry representatives with technical expertise in the areas covered by the unit/s. All participants may be issued with this form but one person must be responsible for completing the form for the RTO. Moderation activity participant names: 1.______________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ Date Unit/s of competency List evidence reviewed. All candidates evidence must be available during the activity List all support documents referred to today Recommendations: Explanation of recommendations: Signatures: 1:. 2: 3: December 2013 Page 24 Follow up Date /Signature of authorising officer Is the evidence Directly related to the units of competence being assessed Based on the critical aspects of evidence Appropriate for the AQF level Sufficient to make a decision about competence Current Authentic Based on realistic workplace activities and contexts Sufficient to cover the full range of knowledge and skills needed to demonstrate competence Taken from more than one source over a period of time Collected in a timely and cost effective way December 2013 Yes No Comments /Actions Page 25 Is the evidence Yes No Comments /Actions Demonstrating task skills, task management skills, contingency management skills and role environment skills Source: Adapted from Assessment validation and continuous improvement workshop guide, DEC NSW, 2012 December 2013 Page 26 Template 8 - Validation Assessment Process When to use this template Use this document when validating internally within your RTO and/or externally with other RTOs and industry representatives with technical expertise in the areas covered by the unit/s. All participants may be issued with this form but one person must be responsible for completing the form for the RTO. Validation participant names: 1.______________________________________________________ 2. _____________________________________________________ 3. _____________________________________________________ 4. _____________________________________________________ 5. _____________________________________________________ Date Assessment processes being validated Choose an assessment process to work on, for example: Policies, procedures and guidelines that currently govern this process: List them Recommendations: Explanation of recommendations: Signatures: 1.__________________________________________ Candidate briefing Giving feedback to candidates Collecting feedback from candidates Conducting question and answer assessments with groups 2. __________________________________________ 3. __________________________________________ 4. __________________________________________ 5. __________________________________________ December 2013 Page 27 Follow up: Date /Signature of authorising officer Does the process Yes No Comments /Actions Clearly identify the steps involved in the assessment process for example in a flow chart Provide plain English instructions to assessor Provide guidance to the assessor on preparing the candidate Provide guidance to the assessor on collecting evidence Provide guidance to the assessor on making decisions Provide guidance to the assessor on reviewing the process Provide advice on reasonable adjustment in assessment Provide guidance to the assessor on providing feedback to the candidate Provide guidance to the assessor in case of appeal Allow for feedback to be provided by candidates and /or clients Allow for feedback to be provided by assessors Provide plain English instructions to candidates and/or clients December 2013 Page 28 Does the process Yes No Comments /Actions Allow for flexibility to be adapted to suit specific candidates and/or workplace needs Allow for documentation of agreement between assessor and candidate and/or client Allow for support mechanisms Clearly document and disseminate appeals processes Provide assessors with documentation to complete for record keeping Document confidentiality and privacy processes December 2013 Page 29 Template 9- Checklist of what to organise when planning the activities The following template can be used to assist in preparing for the validation or moderation activity. The checklist would be part of your Quality Processes. Arrangements to make Availability of participants- external and internal Yes No Comments Ensuring participants have the knowledge/skills required Venue booking Catering if required Any technology requirements How recording/reporting of outcomes will occur Dissemination of validation and moderation plan Dissemination of documents to participants before activity Documents to organise Assessment plans /process Assessment documents ie student tasks, assessors tools Completed assessment evidence Units of competency Training package assessment guidelines Qualification packaging rules AQF descriptors December 2013 Page 30 Source documents/materials such as candidate feedback , supervisor feedback Relevant regulation/legislation Code of practice and/or confidentiality agreement Name of person completing this checklist: Signature: December 2013 Date: Page 31 Template 10 - Confidentiality Agreement This agreement can be used when conducting independent validation or as part of your validation process. All participants who are taking part in validation are requested to sign this agreement. I agree to observe the principles of confidentiality and mutual trust and respect with regard to the activity I am participating in. This includes the assessment plan, process documentation, assessment tools and evidence shared in this validation meeting. I will not use the information for any reason not acceptable to the validation group and code of conduct for assessors. I will also respect others who participate in this meeting and use open and clear communication. RTO Name: RTO Representative: Qualification/units of competence: Meeting date: Location: Name: Organisation: Email: Phone: Signature: Name: Organisation: Email: Phone: Signature: Name: Organisation: December 2013 Page 32 Email: Phone: Signature: December 2013 Page 33 Template 11 – Questions for Independent Parties The following questions can be asked by anyone within the RTO of independent stakeholders to assist in gathering information in an ongoing manner to be used as part of the validation and moderation process. Document the responses which can then be used for consultation and validation and moderation. Name of RTO Representative: Date: Name of Individual Interviewed: Role: Organisation: Questions at various stages in the process Before assessment 1. 2. 3. 4. 5. 6. 7. 8. Is the assessment process consistent with your expectations? Will the assessment practice fit in with your organisational activities and workplace culture? Do the selected assessment methods and tools reflect current industry standards and practices? Are there any areas you consider to be a risk in your work environment we need to be aware of? How do you consider a worker could show their competence? Are the tasks relevant to your work environment? Will the workplace supervisors able to complete the documentation easily? Will the assessment process having a positive impact on your organisation? During assessment 1. Is the assessment process meeting your expectations? 2. Is the assessment fitting in with your workplace activities and culture? 3. Is the workplace supervisor able to easily complete the documentation? After assessment 1. 2. 3. 4. 5. 6. 7. 8. 9. Were the tasks relevant to your work environment? Does the evidence meet the standards you would expect in the industry? Has the assessment process had a positive impact on your organisation? Did the assessment practice fit in with your organisational activities and workplace culture? Did the selected assessment methods and tools reflect current industry standards and practices? Were the workplace supervisors able to complete the documentation as required? Would you employ these students? What key areas of improvement have you identified? What recommendations would you make? December 2013 Page 34 Template 12- Sample Student Questions The following questions can be asked to gain feedback from the student regarding the assessment. Document the responses which can then be used for consultation and validation and moderation. The questions can be asked at various stages and be contained in a number of documents already in place within the RTO. The table below provides examples. Question Possible RTO document where question can be asked and have student sign off 1. Were you informed of the assessment process? Assessment cover sheet, enrolment form, assessment guidelines 2. Were you given clear written instructions about what to do in the assessment? Assessment cover sheet, assessment task instructions 3. Were you given sufficient information about what was going to be assessed and how and when you were going to be assessed? Cover sheet Student assessment guide 4. Were you informed of who the assessor was? Cover sheet Student assessment guide 5. Were you informed of any available support during the process? Assessment guidelines 6. Do you think that the assessment process was fair and equitable? Feedback form 7. Was the assessment linked to the workplace? Feedback form 8. Do you think that the person observing your performance was skilled, fair and unbiased? Feedback form 9. Do you think that the overall judgement about your performance was accurate and fair? Feedback form 10. Were you given adequate feedback about your performance after the assessment? Feedback form 11. Were you told about your right to an assessment appeal? 12. Were you given useful advice about further learning after the assessment? December 2013 Assessment guidelines Enrolment form Feedback form Page 35 Template 13 - Roles and responsibilities of Internal and External Participants The inclusion of independent parties during validation and moderation means that all should be aware of their role and responsibilities. The table below assists in identifying this for internal and external parties. Internal participants External participants Could include program developers, assessors, RTO relevant staff Could include industry representatives, small and large business, assessors from other RTOs, specific client or any unbiased or impartial third-party Plan and organise activities on an ongoing basis Interpret the unit/s of competency Come to a common understanding of the standard to be achieved so agreeing on what a competent person would ‘look like’ Contribute to design of the assessment process Contribute to the development of the assessment tools Ask questions of external parties before and during design Share their knowledge of training and assessment Share their industry knowledge Provide feedback to external parties during the process Contribute to feedback and discussion on the effectiveness of the student assessment tasks and process; the assessors’ documents Encourage participation of others Contribute to discussion on the validity of the evidence collected; the standard of performance achieved; the consistency and accuracy of decision making Reporting and recording December 2013 Come to a common understanding of the standard to be achieved Contribute to the development of the assessment tools Share their industry knowledge Provide information of what would work in the work environment Discuss the standard required in the workplace Describe workplace culture Discuss implications for the workplace of the assessment process Provide feedback to external parties during the process Contribute to feedback and discussion on the effectiveness of the student assessment tasks and assessment process Encourage participation of others Contribute to discussion on the validity of the evidence collected; the standard of performance achieved; decision making Reporting and recording Page 36 Template 14- Questions for evaluation of validation activity To ensure continuous improvement all validation and moderation activities should be reviewed and the feedback used to guide further activity. This process should be ‘chairperson’ or facilitator led with answers documented. All participants should be part of this review process. 1. Was participating in this activity what you thought it would be? 2. What are 2 key things you learnt from participating in this activity? 3. Would you have preferred to be more involved in the activity? If so how? 4. Can you provide two ways that the activity could be streamlined or improved? 5. Were the results of the activity what you anticipated? If not why not? 6. Would you agree to participate in further activities? If not why not? December 2013 Page 37 Section 7 Code of Practice for Validation and Moderation December 2013 Page 38 Code of Professional Practice for RTOs and Independent Parties Participating in Validation and Moderation TABLE OF CONTENTS About this Code .................................................................................................................................................... 41 1 Purpose ........................................................................................................................................................ 41 2. Agreement .................................................................................................................................................... 41 2.1 Intention ................................................................................................................................................. 41 2.2 Overarching agreement ......................................................................................................................... 41 2.3 Principles of working together ............................................................................................................... 42 2.4 Approach .................................................................................................................................................. 4 2.5 Outcomes ................................................................................................................................................. 4 2.6 Definitions .............................................................................................................................................. 44 Validation................................................................................................................................................ 44 Moderation ............................................................................................................................................. 44 Independent parties ............................................................................................................................... 44 Continuous Improvement....................................................................................................................... 44 Assessment ............................................................................................................................................. 45 Assessment Requirements ..................................................................................................................... 45 Principles of Assessment ........................................................................................................................ 45 Rules of evidence .................................................................................................................................... 46 3. References and Further reading ................................................................................................................... 46 Code of Professional Practice for Validation About this Code 1. Purpose The purpose of this professional code of practice for validation is to: Set out principles and processes for RTOs and Independent Parties when participating in validation and moderation activities. Document an agreement between participating VET practitioners and independent parties. Ensure practitioners meet VET obligations as outlined in the VET Quality Framework. Build a culture of continuous professional improvement. Practitioners and Registered Training Organisations working with independent parties will be required to agree to and fulfill the requirements of this code. 2. Agreement This agreement is between participating RTOs, VET practitioners and independent parties involved in the Independent Validation of Assessment Project. 2.1 Intention Practitioners will be required to participate with the following intentions: To improve the quality of their own professional practice. To work to improve the standards of the VET sector as a whole by engaging in a process of continuous improvement in assessment practices. to improve the confidence of stakeholders in VET qualifications 2.2 Overarching agreement Participating practitioners will collaborate with colleagues to: Agree and commit to completing the planned actions; and also that if planned actions are consistently not completed the practitioner may request, and/or be excluded. Share assessment resources with others while adhering to confidentiality and privacy principles. Ensure they attend all meetings or, should that not be possible, to send an alternate delegate from their organisation of similar professional stature and commitment. If this is the case that person must be briefed fully so they can actively participate in the process. Adhere to principles of privacy and confidentiality Be open, transparent and fair in their discussions with others Strive to reach agreement or consensus with others Listen to others point of view December 2013 Page 41 Code of Professional Practice for Validation 2.3 Principles of working together To make consensus possible all group members must: agree this is a ‘safe environment’ where all parties have the opportunity to contribute to the conversation and shares their thoughts start by reaching agreement on what ‘competency’ means for each unit – ie. what outcomes should the candidate demonstrate and to what level. clarify roles - chair/ timekeeper, record keeper, reporter to whole group listen actively – be sure you have heard each others’ comments and are not simply impatiently waiting to speak yourself summarise as you go - be able to paraphrase each others’ comments briefly and straightforwardly before making another point or disagreeing with them be open to others ideas - avoid taking a position, digging in and defending it – no matter how ridiculous it becomes be inclusive – take note of quiet members of the group, encourage them to have a say, then show you have understood what they said, e.g. paraphrase it do not assume silence = agreement, check everyone agrees before moving on work with others to reduce or eliminate conflict 2.4 Approach Assessor Partnerships The validation and moderation process to be used is referred to as Consensus Approach. The Consensus Approach involves working with a group of assessors (even just two) with a focus on quality review and involves: Utilising a collaborative approach to foster collegiality, mutual assistance and confirmation. This is achieved by: - Sharing and discussing each other’s assessment tools, processes and outcomes - Helping each other through support and assistance to work through and resolve any issues - Comparing assessment tools and decisions made against the criteria outlined in units of competency and other aspects of the Qualifications Framework. December 2013 Page 42 Code of Professional Practice for Validation 2.5 Outcomes Practitioners conducting validation and moderation will be required to contribute to continuous review processes, where recommendations for improvements are made (including a justification for the recommendation). Resources produced from this validation project will be made publicly available. This material may be freely used by Australian Training Organisations for educational purposes. You may not make it available for hire or re-sale with financial gain. Practitioner agreement: I agree to adhere to the principles and processes of validation and moderation as outlined in this Code of Practice. I also understand that if I consistently ignore my commitments to this agreement I may be forfeiting my participation in the project. Name: Organisation: Position: Signature: Date: December 2013 Page 43 Code of Professional Practice for Validation 2.6 Definitions Validation “… a quality review process. It involves checking that the assessment tool produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes”. Ref: Adapted from NQC A Code of Practice for Validation and Moderation, 2009. moderation “……process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.” Ref: Adapted from NQC A Code of Practice for Validation and Moderation, 2009. Independent parties ... are those independent to the RTO therefore not employed or contracted by the RTO. The independent parties could include industry representatives, small and large business, assessors from other RTOs or any unbiased or impartial third-party. Feedback from students can contribute to the validation and moderation process by providing feedback on the assessment tasks, assessment process and decision making (see attachment). The importance is ensuring the independent representatives have the knowledge, skills and experience in the area themselves. Continuous Improvement “Continuous improvement means a planned and ongoing process that enables an RTO to systematically review and improve its policies, procedures, products and services in order to generate better outcomes for clients and to meet changing needs. It allows an RTO to constantly review its performance against the VET Quality Framework and to plan ongoing improvements to its performance. Continuous improvement involves collecting, analysing and acting on relevant information collected from clients and other interested parties, including the RTO’s staff” (Standards for NVR Registered Training Organisations, 2011, SNR 3). December 2013 Page 44 Code of Professional Practice for Validation Assessment “Means the process of collecting evidence and making judgements on whether competency has been achieved, to confirm that an individual can perform to the standard expected in the workplace, as expressed by the relevant endorsed industry/enterprise competency standards of a Training Package or by the learning outcomes of a VET accredited course” (Standards for NVR Registered Training Organisations, 2011, SNR 3). Assessment Requirements “Assessment requirements are the endorsed component of a Training Package. Assessment requirements set out the industry’s approach to valid, reliable, flexible and fair assessment” (Standards for NVR Registered Training Organisations, 2011, SNR 3). Principles of Assessment “Principles of assessment are required to ensure quality outcomes. Assessments should be fair, flexible, valid and reliable as follows: a) Fairness: Fairness requires consideration of the individual candidate’s needs and characteristics, and any reasonable adjustments that need to be applied to take account of them. It requires clear communication between the assessor and the candidate to ensure that the candidate is fully informed about, understands, and is able to participate in, the assessment process, and agrees that the process is appropriate. It also includes an opportunity for the person being assessed to challenge the result of the assessment and to be reassessed if necessary. b) Flexible: To be flexible, assessment should reflect the candidate’s needs; provide for recognition of competencies no matter how, where or when they have been acquired; draw on a range of methods appropriate to the context, competency and the candidate; and, support continuous competency development. c) Validity: or relevance-In simple terms, the assessment must assess what it proposes to assess. it requires assessment against specified criteria; a sufficient range of the performance of the person being assessed; assessors having demonstrated competence in the area they are assessing; using multiple approaches to assessment d) Reliability: or consistency-In general, reliability is an estimate of how accurate or precise the task is as a measurement instrument. Reliability is concerned with how much error is included in the evidence” (adapted from Standards for NVR Registered Training Organisations, 2011, SNR 3). December 2013 Page 45 Code of Professional Practice for Validation Rules of evidence Rules of evidence are closely related to the principles of assessment and provide guidance on the collection of evidence to ensure that it is valid, sufficient, authentic and current as follows: a) Validity: see Principles of assessment. is the evidence sufficiently relevant to the competencies being assessed. b) Sufficiency: Sufficiency relates to the quality and quantity of evidence assessed. It requires collection of enough appropriate evidence to ensure that all aspects of competency have been satisfied and that competency can be demonstrated repeatedly. Supplementary sources of evidence may be necessary. The specific evidence requirements of each unit of competency provide advice on sufficiency. c) Authenticity: To accept evidence as authentic, an assessor must be assured that the evidence presented for assessment is the candidate’s own work. d) Currency: Currency relates to the age of the evidence presented by candidates to demonstrate that they are still competent. Competency requires demonstration of current performance, so the evidence must be from either the present or the very recent past. 3. References and Further reading 1. 2. 3. NQC, A Code of Practice for Validation and Moderation, 2009. Standards for NVR Registered Training Organisations, 2011. http://www.comlaw.gov.au/Details/F2011L01356/ExplanatoryStatement/Text Australian Qualifications Framework (AQF) http://www.aqf.edu.au/ December 2013 Page 46