Manage an assessment system for training outcomes D1.HRD.CL9.08 D1.HHR.CL8.06 Trainee Manual Manage an assessment system for training outcomes D1.HRD.CL9.08 D1.HHR.CL8.06 Trainee Manual Project Base William Angliss Institute of TAFE 555 La Trobe Street Melbourne 3000 Victoria Telephone: (03) 9606 2111 Facsimile: (03) 9670 1330 Acknowledgements Project Director: Chief Writer: Subject Writer: Project Manager: Editor: DTP/Production: Wayne Crosbie Alan Hickman Quentin Derrick Alan Maguire Jim Irwin Daniel Chee, Mai Vu, Riny Yasin The Association of Southeast Asian Nations (ASEAN) was established on 8 August 1967. The Member States of the Association are Brunei Darussalam, Cambodia, Indonesia, Lao PDR, Malaysia, Myanmar, Philippines, Singapore, Thailand and Viet Nam. The ASEAN Secretariat is based in Jakarta, Indonesia. General Information on ASEAN appears online at the ASEAN Website: www.asean.org. All text is produced by William Angliss Institute of TAFE for the ASEAN Project on “Toolbox Development for Front Office, Food and Beverage Services and Food Production Divisions”. This publication is supported by Australian Aid through the ASEAN-Australia Development Cooperation Program Phase II (AADCP II) Copyright: Association of Southeast Asian Nations (ASEAN) 2013 All rights reserved. Disclaimer Every effort has been made to ensure that this publication is free from errors or omissions. However, you should conduct your own enquiries and seek professional advice before relying on any fact, statement or matter contained in this book. ASEAN Secretariat and William Angliss Institute of TAFE are not responsible for any injury, loss or damage as a result of material included or omitted from this course. Information in this module is current at the time of publication. Time of publication is indicated in the date stamp at the bottom of each page. Some images appearing in this resource have been purchased from various stock photography suppliers and other third party copyright owners and as such are non-transferable and non-exclusive. Additional images have been sourced from Flickr and are used under: http://creativecommons.org/licenses/by/2.0/deed.en http://www.sxc.hu/ http://www.stockfreeimages.com File name: TM Manage assessment sys training outcomes_Final Table of contents Introduction to trainee manual........................................................................................... 1 Unit descriptor................................................................................................................... 3 Assessment matrix ........................................................................................................... 5 Glossary ........................................................................................................................... 7 Element 1: Communicate the assessment system ............................................................ 9 Element 2: Provide support for enterprise assessors ...................................................... 21 Element 3: Manage the assessment record keeping system........................................... 47 Element 4: Maintain quality assurance procedures ......................................................... 63 Element 5: Report on the contribution made by the assessment system to enterprise training and performance ................................................................................................ 77 Presentation of written work ............................................................................................ 93 Recommended reading................................................................................................... 95 Trainee evaluation sheet ................................................................................................. 97 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Introduction to trainee manual Introduction to trainee manual To the Trainee Congratulations on joining this course. This Trainee Manual is one part of a ‘toolbox’ which is a resource provided to trainees, trainers and assessors to help you become competent in various areas of your work. The ‘toolbox’ consists of three elements: A Trainee Manual for you to read and study at home or in class A Trainer Guide with Power Point slides to help your Trainer explain the content of the training material and provide class activities to help with practice An Assessment Manual which provides your Assessor with oral and written questions and other assessment tasks to establish whether or not you have achieved competency. The first thing you may notice is that this training program and the information you find in the Trainee Manual seems different to the textbooks you have used previously. This is because the method of instruction and examination is different. The method used is called Competency based training (CBT) and Competency based assessment (CBA). CBT and CBA is the training and assessment system chosen by ASEAN (Association of SouthEast Asian Nations) to train people to work in the tourism and hospitality industry throughout all the ASEAN member states. What is the CBT and CBA system and why has it been adopted by ASEAN? CBT is a way of training that concentrates on what a worker can do or is required to do at work. The aim is of the training is to enable trainees to perform tasks and duties at a standard expected by employers. CBT seeks to develop the skills, knowledge and attitudes (or recognise the ones the trainee already possesses) to achieve the required competency standard. ASEAN has adopted the CBT/CBA training system as it is able to produce the type of worker that industry is looking for and this therefore increases trainee’s chances of obtaining employment. CBA involves collecting evidence and making a judgement of the extent to which a worker can perform his/her duties at the required competency standard. Where a trainee can already demonstrate a degree of competency, either due to prior training or work experience, a process of ‘Recognition of Prior Learning’ (RPL) is available to trainees to recognise this. Please speak to your trainer about RPL if you think this applies to you. What is a competency standard? Competency standards are descriptions of the skills and knowledge required to perform a task or activity at the level of a required standard. 242 competency standards for the tourism and hospitality industries throughout the ASEAN region have been developed to cover all the knowledge, skills and attitudes required to work in the following occupational areas: Housekeeping Food Production Food and Beverage Service © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 1 Introduction to trainee manual Front Office Travel Agencies Tour Operations. All of these competency standards are available for you to look at. In fact you will find a summary of each one at the beginning of each Trainee Manual under the heading ‘Unit Descriptor’. The unit descriptor describes the content of the unit you will be studying in the Trainee Manual and provides a table of contents which are divided up into ‘Elements’ and ‘Performance Criteria”. An element is a description of one aspect of what has to be achieved in the workplace. The ‘Performance Criteria’ below each element details the level of performance that needs to be demonstrated to be declared competent. There are other components of the competency standard: Unit Title: statement about what is to be done in the workplace Unit Number: unique number identifying the particular competency Nominal hours: number of classroom or practical hours usually needed to complete the competency. We call them ‘nominal’ hours because they can vary e.g. sometimes it will take an individual less time to complete a unit of competency because he/she has prior knowledge or work experience in that area. The final heading you will see before you start reading the Trainee Manual is the ‘Assessment Matrix’. Competency based assessment requires trainees to be assessed in at least 2 – 3 different ways, one of which must be practical. This section outlines three ways assessment can be carried out and includes work projects, written questions and oral questions. The matrix is designed to show you which performance criteria will be assessed and how they will be assessed. Your trainer and/or assessor may also use other assessment methods including ‘Observation Checklist’ and ‘Third Party Statement’. An observation checklist is a way of recording how you perform at work and a third party statement is a statement by a supervisor or employer about the degree of competence they believe you have achieved. This can be based on observing your workplace performance, inspecting your work or gaining feedback from fellow workers. Your trainer and/or assessor may use other methods to assess you such as: Journals Oral presentations Role plays Log books Group projects Practical demonstrations. Remember your trainer is there to help you succeed and become competent. Please feel free to ask him or her for more explanation of what you have just read and of what is expected from you and best wishes for your future studies and future career in tourism and hospitality. 2 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Unit descriptor Unit descriptor Manage an assessment system for training outcomes This unit deals with the skills and knowledge required to Manage an assessment system for training outcomes in a range of settings within the hotel and travel industries workplace context. Unit Code: D1.HRD.CL9.08 D1.HHR.CL8.06 Nominal Hours: 40 hours Element 1: Communicate the assessment system Performance Criteria 1.1 Define the purpose of the assessment system 1.2 Document and circulate assessment system features and procedures to relevant stakeholders 1.3 Establish procedures for keeping relevant stakeholders informed about key features of the assessment system Element 2: Provide support for enterprise assessors Performance Criteria 2.1 Verify assessors meet required competency standards 2.2 Identify required training for assessors 2.3 Provide for moderation of assessments 2.4 Provide feedback to assessors on their performance 2.5 Facilitate networking amongst assessors Element 3: Manage the assessment record keeping system Performance Criteria 3.1 Identify and develop the internal records required to support the assessment system 3.2 Describe the requirements for completing the assessment records 3.3 File the assessment records 3.4 Review and up-date the assessment record keeping system © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 3 Unit descriptor Element 4: Maintain quality assurance procedures Performance Criteria 4.1 Define quality assurance procedures for the assessment system 4.2 Undertake internal audits of the assessment system 4.3 Use the results of quality assurance audits to improve the assessment system 4.4 Review the applicability of the existing quality assurance procedures to the dynamic nature of the assessment system Element 5: Report on the contribution made by the assessment system to enterprise training and performance Performance Criteria 5.1 Identify the outcomes that the assessment system has produced 5.2 Review the assessment system 5.3 Make recommendations for change to the assessment system 5.4 Produce and circulate a report 5.5 Revise the existing assessment system on the basis of responses to the report 4 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Assessment matrix Assessment matrix Showing mapping of Performance Criteria against Work Projects, Written Questions and Oral Questions Work Projects Written Questions Oral Questions Element 1: Communicate the assessment system 1.1 Define the purpose of the assessment system 1.1 1, 2 1 1.2 Document and circulate assessment system features and procedures to relevant stakeholders 1.2 3 2 Establish procedures for keeping relevant stakeholders informed about key features of the assessment system 1.3 4 3 1.3 Element 2: Provide support for enterprise assessors 2.1 Verify assessors meet required competency standards 2.1 5 4 2.2 Identify required training for assessors 2.2 6 5 2.3 Provide for moderation of assessments 2.3 7 6 2.4 Provide feedback to assessors on their performance 2.4 8 7 2.5 Facilitate networking amongst assessors 2.5 9 8 Element 3: Manage the assessment record keeping system 3.1 Identify and develop the internal records required to support the assessment system 3.1 10 9 3.2 Describe the requirements for completing the assessment records 3.2 10 10 3.3 File the assessment records 3.3 11 11 3.4 Review and up-date the assessment record keeping system 3.4 12, 13, 14 12 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 5 Assessment matrix Work Projects Written Questions Oral Questions Element 4: Maintain quality assurance procedures 4.1 Define quality assurance procedures for the assessment system 4.1 14, 15 13 4.2 Undertake internal audits of the assessment system 4.2 16 14 4.3 Use the results of quality assurance audits to improve the assessment system 4.3 17 15 4.4 Review the applicability of the existing quality assurance procedures to the dynamic nature of the assessment system 4.4 18 16 Element 5: Report on the contribution made by the assessment system to enterprise training and performance 5.1 Identify the outcomes that the assessment system has produced 5.1 19 17 5.2 Review the assessment system 5.2 20 18 5.3 Make recommendations for change to the assessment system 5.3 21 19 Produce and circulate a report 5.4 22, 23, 24 20 Revise the existing assessment system on the basis of responses to the report 5.5 25 21 5.4 5.5 6 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Glossary Glossary Term Explanation Accredited Officially recognised as meeting the essential requirements Assessor A person who makes assessments Certification A recognised statement of attainment of something Confidentiality In strict privacy Diagnose Determine what is wrong Ensure To make sure of something Evaluation An appraisal or check of something’s effectiveness Internal Within something Mechanism The means by which a purpose is accomplished Moderation Checking that something is within the given boundaries or requirements or standards Procedure A particular course or mode of action Policy A defined course of action Qualification An accomplishment that is recognised by an authority or organisation Requirement Something that has to be achieved or done before something else can occur Review A critical report that investigates something Skill Ability to do something properly Stakeholder A person or group that has an investment or interest in something Standard The level of accomplishment that is set © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 7 Glossary 8 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 1: Communicate the assessment system Element 1: Communicate the assessment system 1.1 Define the purpose of the assessment system Introduction There are many different purposes of an assessment system and it is important to distinguish between assessment and testing. Testing can take many forms and can be used as one part of an assessment system but it is necessary to define the purpose of the assessment system in order to determine how you will assess participants and what assessment tools, such as a test, you will use. Defining the purpose of an assessment system The first step in designing an effective assessment system is to determine what the assessment will be assessing. There are many different purposes for assessment systems and it is important to define a clear statement of the purpose for the assessment. This helps to make sure that the assessment matches the need or area that is being assessed. Some purposes for developing an assessment system may include: Recognising or determining the current competencies or skills of staff Identifying training needs of staff Checking that workplace skill levels are being maintained Issuing internal certificates of achievement or awards for excellence Recognising progress in learning or achievement of desired learning outcomes Identifying areas for improvement or change Maintaining current skill levels of staff. Activity In pairs, write down some other reasons why you might develop an assessment system in the workplace. What is the reason for assessment? Assessment is defined by ANTA as “the process of gathering and judging evidence in order to decide whether a person has achieved a standard or objective” (www.wikitaa.com). Making this decision means following a process, but before looking at that, it is worth understanding the term “context of assessment”. Context does describe the purpose behind our assessment, but is also concerned with many other things closely related to this. An example of putting assessment into context may be a young person who has just joined the workforce and is in a junior position in an office with a lot to learn. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 9 Element 1: Communicate the assessment system In this case the context of any assessment being carried out is not just about this person being able to answer questions and demonstrate skill, but also issues such as the work environment (which would be new and possibly threatening), who is supervising and training this person, how the skills being covered are relating back to the actual work this person is expected to do and so forth. Understanding context means any assessment is undertaken in a more holistic manner which takes these and other issues into account. Before embarking on any assessment activity, there must be a clear focus on what we are trying to achieve through the activity. Within any assessment system, assessment can respond to externally endorsed units of competency and their associated elements and performance criteria and/or internal organisational requirements and standards. The people who are to undergo assessment are referred to as candidates and could conceivably be assessed for one of the following reasons: To confirm prior training To check current level of skill To determine language, literacy and numeracy needs Recognition of Prior Learning (RPL). For each of these reasons, candidates must be able to demonstrate the correct skills, knowledge and attitude against appropriate benchmarks. These benchmarks are very important as they provide the framework within which assessment takes place. Competency standards Product specifications Benchmarks Criteria on a course curriculum 10 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Performance specifications Element 1: Communicate the assessment system Activity Look at the assessment system data and information below and determine what the purpose is for each one. 1. Name Qualifications Industry Experience Salary Scale BA Tourism 2 years 2 Diploma of Hotel Management 5 years 3 Angela Snow Cert IV Hotel Operations 3 years 2 Carrie Jones BA Tourism, MBA 8 years 4 Seema Indira MA Hotel Management 14 years 6 Bob Smith Jeremy Mannis 2. Mike Island Job duties Rating Recommendations Satisfactory Focus on office duties including emailing, filing and document control Excellent Keep up the good work! Satisfactory Respond more promptly to guest feedback Staff scheduling Excellent Maintain clear communication with staff Staff training Excellent Continue regular training sessions with staff Office administration Guest check in/out Responding to complaints 3. Kitchen Staff Name Combi oven Char grill Dishwasher Wok burners Emily Simms × Shaun Stiles × × © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 11 Element 1: Communicate the assessment system 1.2 Document and circulate assessment system features and procedures to relevant stakeholders Introduction After determining the purpose of an assessment system, all of the relevant stakeholders need to clearly understand the system features and how to conduct the assessment. The system and procedures must be transparent and readily accessible in order for it to be effective. Unclear system features or communication can lead to an unsuccessful assessment system where stakeholders are not clearly informed about the features of the system and as a result the assessment system may not be achieving the desired outcomes. Reliability and Validity There are many different components in an assessment system ranging from assessment procedures to where documents are stored. Two important factors that influence all assessment systems are reliability and validity. Reliability: refers to the ability for an assessment system or feature to achieve the same assessment outcome for the same candidate in the same circumstance using the same assessment system. Reliability is important for all stakeholders to believe that the assessment system is effective. Validity: refers to how completely an assessment system assesses what is being assessed. In other words, how effectively does the assessment match the information or skills that are being assessed? If there is a gap between the training or information and the assessment system, then the assessment system is not valid and ultimately does not provide a true reflection of the information or required skill. For example, if an assessment system gives a different result for a candidate who undertakes the assessment in the same conditions, it is unreliable. If an assessment system checks an outcome that is not aligned with the training or purpose of the assessment, then it is not valid. All assessment systems must include the concepts of reliability and validity in order for them to be effective. Does Shows How Performance/Action Competence Knows How Knows Miller’s Pyramid: Miller, G, 1990 the assessment of clinical skills/competence/performance Academic Medicine. 12 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 1: Communicate the assessment system Four principles underlying assessment There are four main principles that should be considered when developing any assessment system. Valid Is the assessment being carried out relevant to the skill and knowledge areas being covered? Assessors ensure validity by focussing on the right areas of skill and competence and ensuring that the assessment tasks approximate to the sort of tasks the candidate would actually encounter in the workplace. Reliable Can the assessment activity be carried out with consistent results over a period of time with different candidates? Assessors can better achieve reliability through collecting evidence via multiple methods and being clear in terms of the competencies or standards which are being tested. Flexible Can the assessment be carried out in a range of different locations and times, on and off the job and still produce an acceptable outcome? Flexible assessment ensures different methods are used which suit varying circumstances and recognise competence for what it is regardless of the experience which lead to it or the conditions in which it is demonstrated. An important concept implicit within this part of AQTF is for the assessor to tailor units of competency or a qualification to suit certain needs. It is not about changing the integrity of outcomes related to the unit but ensuring that assessment suits a particular enterprise or learner. You may find yourself assessing one day in a large city restaurant and the next at a tiny country road house against the same unit. Ultimately the requirements of the unit don’t change but the way you assess should. Fair Can the assessment take into account various differences amongst candidates without disadvantaging anyone? An assessor is fair in terms of helping candidates understand what is expected of them and is not affected by any factors other than those being judged against the relevant standards. Assessors also need to be aware of their obligations to be non discriminatory in their practices with candidates. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 13 Element 1: Communicate the assessment system Assessment system features Once the purpose of the assessment system has been defined, the features and procedures of the system need to be documented and circulated to all relevant stakeholders. The system features may include: An ability to determine the required competency levels and qualifications of assessors Policies and procedures for obtaining, updating and storing records and information of candidates and assessors. This may include version control for document filing, guidelines for completing trainee information, and policies and procedures outlining where and for how long trainee and assessor data should be kept How to conduct assessment including the timing and duration specifics. Consideration may need to given for operational issues where conducting assessment could interfere with day to day operations. A certain degree of flexibility is still needed in order for the training and assessment to remain relevant and valid Flexibility in order to respond to individual needs of participants who may have particular challenges such as physical or language and literacy needs. The assessment system must include features that cater for individual differences of participants including access and equity for all participants Outline country specific occupational health and safety regulations, employment requirements and equal opportunity legislation Procedures for recognising previous educational qualifications or experience gained at other training organisations or employment contexts. The assessment system should provide a mechanism that allows for prior learning and experience to be recognised if relevant to the current training or learning context Information about where assessment materials are located and where assessment can take place. This may be in a training room, work setting, computer lab, outside or a combination of places A review and evaluation process to determine if the assessment system is effective or needs improvement. This should be an ongoing process and should involve all stakeholders’ input Policies and procedures that allow for feedback including, grievances, moderation, reassessment A statement relating to competency levels and remuneration levels appropriate to those skills attained. 14 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 1: Communicate the assessment system Assessment system planning In the first instance, when a person is enrolled in a course of training which will lead to some form of assessment the assessor must ensure that participants are made aware of the assessment system requirements before they begin training or assessment tasks. This means that potential candidates for assessment are made aware of what the process will involve. This is a collaborative process in which they will be supported to a reasonable degree. All assessment systems should, as a part of their training and assessment process, have explanatory materials which the assessor can provide which explain the assessment procedures to candidates prior to any assessment taking place. Assessing an individual is a process which requires thought and pre planning in order to ensure it is carried out in an appropriate manner. It is also important to remember the individual needs of learners which mean including the following steps in an assessment system: 1. Who you are assessing? - You need to carefully consider the person you will be assessing. This includes looking at any special characteristics or needs which the person might have. If you have considered something such as a language or literacy issue beforehand, you can accommodate this within your plan 2. Standards - You must also consider the standards the assessment is being carried out against. This may be against externally recognised competency standards and their performance criteria or perhaps against workplace performance standards. The nature of your assessment activity will be reflected by the sort of standards you are checking. For example, the competency may be based around an activity which can only be assessed via direct observation of someone doing it. So you would have to factor this into the way your session would be run 3. Assessment methods and tools - Central to any assessment are the kinds of physical resources which may be required along with necessary procedures and the methods chosen to carry out the assessment. As well as how you will go through this, you need to be sure that everything is present to enable the candidate to demonstrate their level of competency 4. Consultation - Another issue to remember is who needs to be consulted and informed about your assessment activity. It is a matter of common courtesy and professionalism to arrange dates for assessment well in advance. As well as the candidate, you should also be talking to other stakeholders such as the candidate’s manager, affected colleagues’ and anyone else who may need to know about your activity. Some assessments can be quite disruptive to workplaces and advance notice can help minimise this and provide a chance to make necessary preparations 5. Assessment meeting - Once dates are in place consideration needs to be given to the structure of the meeting. Experienced assessors usually make allowance at the start of the meeting to catch up on what the candidate has been up to and generally make them feel more comfortable. Example 1 below gives a suggested breakdown of an assessment meeting. You should also be prepared for what might change the structure of your meeting. This might range from timing alterations right through to the personal circumstance of a candidate 6. Be adaptable - Based on your plan you also need to have the flexibility to change parts of the above, should you receive notice prior to your visit. An Assessment Plan is included at Appendix 5. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 15 Element 1: Communicate the assessment system Some types of assessment 1 Formative is assessment which is conducted over a period of time, usually during a training program and provides people with feedback on an ongoing basis. Many qualifications or training programs have a Formative approach which means participants are being asked to complete assessment activities and get feedback on this before the completion of the program 2 Summative is assessment which takes place after a course and is usually involved with making a judgement on total learning which has occurred over that time 3 Diagnostic is as the name suggests about diagnosing a situation. It will commonly be carried out in the absence of or before training as a preliminary Gap Analysis tool to determine what training may be required to meet shortfalls against required standards 4 RPL (Recognition of Prior Learning) involves assessment in the absence of training to verify existing skills, knowledge and experience. Stakeholders After documenting the assessment system features and procedures it is important that all relevant stakeholders are aware of the assessment system, its purpose and the features of it. Communicating the different aspects of the assessment system enables stakeholders to develop a clear understanding of the system and this will lead to a more effective and successful system as a whole. Some of the relevant stakeholders may include: Department heads, team leaders, managers and the business owners Participants and candidates from both current learners and past learners Trainers and assessors. External assessors and trainers may also need to be informed of the assessment system especially if there is an external audit process or certification requirements Technical experts who may be required to provide feedback about the relevance and currency of skills and assessment Training and assessment coordinators Government and industry bodies related to the assessment components. Document and circulate assessment system and features All of the assessment system features and procedures must be documented in a clear and logical manner which enables all stakeholders to understand what and how participants are being assessed. The system features need to be clearly outlined so all stakeholders can access the assessment details easily. Once all of the system details have been documented the assessment system procedures should be distributed to all relevant stakeholders. This can be done via email or a hard copy of the document can be circulated. Using electronic files is much more effective and the data can be updated easily. 16 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 1: Communicate the assessment system 1.3 Establish procedures for keeping relevant stakeholders informed about key features of the assessment system Introduction An assessment system is not a stagnant thing; it is dynamic and constantly changing in order to reflect the changing nature of the workplace or training context. Once the assessment system features have been documented and circulated, any changes or updates need to be communicated to all relevant stakeholders. This keeps the assessment system operating in an effective manner and allows feedback from stakeholders to be implemented to improve the system. The procedures for updating stakeholders of changes to the assessment system should be clearly outlined and all stakeholders should be informed of how any updates will be communicated to them. Keeping stakeholders informed All systems are constantly changing and evolving. An assessment system needs to have mechanisms that allow for feedback and change. When these changes or updates are agreed and implemented, all stakeholders need to be informed of the changes in order for the system to remain effective. As a result, procedures need to be developed to inform stakeholders quickly and efficiently about any information relating to the assessment system. There are several ways that stakeholders could be kept informed, including: Scheduling regular meetings that relate directly to assessment issues. These meetings may not always need to be long as there are not always issues to deal with, but regularly scheduled meetings allow stakeholders an opportunity to provide feedback and to listen and share ideas or thoughts about the system Special meetings which may be necessary at various times due to urgent issues or feedback Regular staff meetings having a standing agenda item included that allows for feedback and comments regarding assessment issues The organisation of a dedicated newsletter or fact sheet that provides updates on assessment system changes or feedback. The newsletter could be in electronic or paper-based format Developing an internal network of assessors who can communicate directly with each other and discuss particular issues that may be relevant to the assessment system delivery and effectiveness. Issues discussed within the network could then be raised to assessment system coordinators or managers. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 17 Element 1: Communicate the assessment system Work Projects It is a requirement of this Unit you complete Work Projects as advised by your Trainer. You must submit documentation, suitable evidence or other relevant proof of completion of the project to your Trainer by the agreed date. 1.1 Provide a list of assessment purposes from your current organisation or a past job. These could include: work performance reviews, identifying training needs or qualifications, or confirming progress in learning. 1.2. Develop a list of all the relevant stakeholders in your organisation that may need to have knowledge of the assessment system. List the assessment features of your current assessment system at your organisation. 1.3. Document a set of procedures from your workplace that outlines which stakeholders need to be informed and how they will be informed regarding assessment system details. 18 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 1: Communicate the assessment system Summary Communicate the assessment system When developing an assessment system, it is important to understand the purpose of the system. Once the purpose has been identified, further development of the assessment system content should always align with the stated purpose of the system. An assessment system should have: A stated purpose that is clear and attainable Documented features that clearly outline the areas of the system A mechanism to enable the system features and purpose to be distributed to relevant stakeholders Procedures for informing stakeholders of the system features or any changes that have been made to the system. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 19 Element 1: Communicate the assessment system 20 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Element 2: Provide support for enterprise assessors 2.1 Verify assessors meet required competency standards Introduction In order for assessors to be able to assess effectively, it is critical that they have the necessary skills and experience required to assess participants. If the assessors have limited experience in the assessment item, then the assessment may not be a true reflection of the participant’s ability in that skill. It is important then to check all assessors experience and qualifications to ensure that they have the necessary skills to be able to conduct the assessment properly. Verifying assessors skills and experience To be able to assess people’s performance, you first need to have capable and qualified people to conduct the assessment. If assessors do not have sufficient skills and experience to carry out the assessment, the assessment may be invalid. It is then important to verify assessor qualifications and skills in order to ensure that they meet the competency standards. These standards may be set by the organisation or they may be required by an external body such as a training provider or government agency. The first step in verifying assessor competency is to determine the assessment that is being carried out. If the assessment is for Certificate II level then the assessor needs to have at least competency at Certificate II level and experience in the area that is being assessed. Competency standards could be set by an external agency and if this is the case, then assessors need to be checked that they meet the standards outlined by these agencies. Alternatively, the competency standards may be set internally and may include formal assessor qualifications which need to be verified. It is also important to check the currency of the qualification or experience of an assessor. If the assessor completed qualifications many years earlier and they have not worked in the relevant industry or department, their qualification may no longer be relevant. Also, if their qualification or experience is in an area that is not applicable to the area being assessed, then they may not be eligible to conduct certain types of assessment. Basically, skills, qualifications and experience need to be current, up to date, aligned with the area being assessed, and meet the documented standards in order for an assessor to be able to assess effectively. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 21 Element 2: Provide support for enterprise assessors Methods of Assessment Assessment methods describe the techniques used to gather evidence, while assessment tools are the instruments and procedures used to organise and interpret that evidence. The assessment process relies strongly on the collection of this evidence. In order to collect evidence the assessor must employ a number of methods. Some of the methods employed include: 3rd party report This describes where another person, usually in a supervisory position, provides written verification of a candidate’s ability to carry out a task or that they have certain knowledge. Case study The assessor may provide a candidate with an example of a situation which mirrors aspects of the skills they have been trained in. The candidate must respond to this with a series of answers to questions relating to the scenario. Observation In this case the assessor observes the candidate carrying out a task and applying knowledge. They make a written record of this which then becomes evidence in itself. This record is generally as a checklist with space for feedback and comments. Presentation The candidate may be asked to prepare a presentation either for the assessor or another group and their performance is recorded. Questioning This can be done either through the use of oral questions to which the assessor records the answers or via the collection of written answers which have been given to a candidate. Role play In some cases candidates will be asked to play a role in a scenario which will be observed by the assessor. Often the assessor will provide a pre-written role for the candidate and then let the candidate work within this. An example of where this might be used is in customer service training where a candidate will be observed dealing with various situations according to their role. Simulation In some cases a simulated environment will be created for a candidate to gauge how they deal with different situations. This may be carried out to ensure a greater level of control over the situation and for safety reasons. Written/Projects Evidence may also be collected in the form of responses to longer questions or projects. Some candidates are given projects relating to their workplace which they must research and complete over time before submitting this to the assessor. The list above is by no means exhaustive and assessors will employ many other innovative and situation suited methods in order to gather the evidence they require. 22 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Making arrangements for assessment The efficient execution of assessment activities relies to a large part on the organisation. Having a plan in place is critical, but there can also be quite a number of things to arrange. Prior to the actual assessment you should consider the following details: Timings – What time is best for the meeting? Whilst this needs to be organised with the candidate and other work based stakeholders, you should give careful consideration to the time of day. Consider whether your appointment clashes with other distractions, or is too close to lunch or the end of the day Location – You must ensure the assessment is held in an appropriate venue which is free of discomfort or distractions and is also safe. Like training, the safety of your candidates is paramount and you should not conduct any assessment activity if you have concerns over anyone’s safety, including your own Resources – Candidates will need certain resources, equipment, documents etc to be able to demonstrate competency. Part of making arrangements is to ensure that these resources are available in a form that is required for the assessment Their role – You must ensure that candidates understand what they will be expected to do during the assessment, the criteria they will be assessed against and the methods you will use to collect evidence. They must also be clear on the type of evidence they will be expected to produce. What is competence? Competency based assessment is a form of vocational training and assessment. This makes a judgement on what a person can do in the workplace as a result of having completed prior training or on the experience they have and the learning they have undergone through their work. It is characterised by being focused on work based outcomes as the principal means of measuring skill and ability. This is accomplished by the measurement being made against objective criterion based on nationally endorsed standards. These standards are met by the candidate providing evidence that they are able to perform in various tasks to the level specified. Competence is not just about being able to do something but to apply that ability in a work environment. In assessing competence the assessor is concerned with testing skill, knowledge and attitude, in the four dimensions and all nationally endorsed competency standards have been developed to incorporate these dimensions. Assessors use a competency standard to collect a range of evidence which is then used to make a decision. Assessment outcomes are either: Competent (C), where the candidate has produced evidence to show they have attained competency or Not Yet Competent (NYC) where they must produce more evidence to satisfy the assessor that they have reached the required standard. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 23 Element 2: Provide support for enterprise assessors The role of evidence The assessment process is geared principally towards the collection of evidence. Evidence is something we use to assist us in making a decision as to the competence or otherwise of a candidate. It may consist of anything which the candidate can produce which demonstrates that they possess skill and knowledge in a particular area and are able to use this to achieve outcomes at a certain level. Measurable evidence can be broken up into the four main categories of direct, indirect, supplementary and historic as mentioned previously. Some examples of measurable evidence that an assessor may consider are: Something that has actually been produced i.e. a product such as a baked loaf of bread A task that has been carried out such as completion of a workplace procedure e.g. making a bed to a standard Background knowledge and understanding such as knowing why something must be done and the reasons for that. For e.g. a person who is demonstrating how to use a program such as Microsoft Word would need to have knowledge of what the program can and can’t do as well as being able to execute commands A person’s attitude to aspects of their work. This is reflected in how they feel about their work and the level of application and care they show in doing it. Determining the evidence needed to demonstrate competency When preparing an assessment plan, you must read and interpret the assessment benchmarks of your organisation’s assessment system or external standards to determine the evidence that needs to be gathered to demonstrate competency against each benchmark. In competency based assessment, evidence is the information, materials and products that support a candidate’s claim to demonstrate competence against an assessment benchmark. While there are three main types of evidence, there are many evidence gathering methods (also known as assessment methods) and any combination of these can be used. The rules of evidence When verifying assessors meet required competency standards in order to be used as part of an assessment decision, evidence must meet the following four rules of evidence criteria: Valid, Sufficient, Current and Authentic. It must be Valid. This means the evidence must clearly relate to the unit of competence being assessed. The performance criteria within the unit must be used as the basis of all evidence collection and assessors need to guard against the collection of superfluous evidence. Evidence must be Sufficient. The assessor must collect enough to be sure that the candidate is competent across all applicable competency elements or industry standards. Competency over a period of time needs to be addressed and all dimensions of competency need to be covered. This also includes assessing in different contexts. 24 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors There must be Currency. The evidence collected should relate to recent activities (usually within the last three years) Current knowledge and skills are what you should be assessing and the candidate’s evidence should reflect this. Imagine a person providing you with evidence relating to something they did 10 years ago. Your question as an assessor must centre on what they can do now and the evidence they provide supporting this. The evidence also needs to be Authentic. The assessor needs to verify that the evidence collected was actually produced by the candidate. This may require confirmation from a third party where there is any doubt. Plagiarism is a major issue today and as an assessor you need to promote and maintain a focus on authentication when it comes to checking the credibility of any evidence provided. It is the responsibility of the assessor to verify that all of the evidence received from assessor meets these criteria before accepting it to be used as part of the assessment process. Activity Look at the assessor checklist below and in pairs discuss each assessor’s qualifications and experience. Complete the chart with your own qualifications and experience. What type of assessment would each person be able to conduct? Why? Name Position Qualifications Experience Mandy Ledger Front Office manager BA Hotel Operations Workplace Training and Assessment 5 years Front Office staff 1 year Front Office Manager Customer Service Management Diploma Levi Humberger Waiter Certificate III in Hospitality 2 years restaurant waiter Sanesh Murindi Barista Certificate in Barista service 3 years barista Master Assessor in Barista service MA Finance and Accounting 9 years general accountant MBA BA in Accounting 8 years senior accountant Certified Practising Accountant Workplace Training and Assessment Mike Andrews Chief Accountant © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 25 Element 2: Provide support for enterprise assessors 2.2 Identify required training for assessors Introduction The previous unit explained the need to verify assessor qualifications and experience in order to determine if they have the correct skills to be able to perform assessment in certain areas. Once the assessors have had their qualifications and experience checked it may be necessary to provide training for assessors if they do not meet the required standards. Assessing competence - Creating the correct environment Like an interview, an assessment visit should be conducted in a manner which gives the candidate the maximum opportunity to perform to the best of their ability. An emphasis should be placed on clear communication and also building of rapport. The environment this produces will hopefully encourage the candidate in producing high quality evidence in support of his/her claims of competency. Assessment is a participatory process and for this reason you should openly discuss with the candidate all aspects of the process and gain input on their needs. An assessor who acts in a non friendly or even secretive manner may negatively affect the assessment environment and ultimately the results obtained. For this reason assessors need to be aware of the importance of their actions and words in generating the right environment. Be sure that you work through the following checklist with the candidate: What is the purpose of the assessment? – this should be explained clearly Explain performance criteria – have a copy of the performance criteria against which the assessment will be carried out and go through it with the candidate Check assessment conditions – ensure that the conditions are appropriate. Don’t start the assessment if the candidate is not able to apply themselves fully or the workplace is too distracting Verify time and place – this should be done by phone or email well in advance of a physical visit. It is wise to confirm this at least the day before if travel is involved Explain method and type of assessment – go through the assessment tool with the candidate and possibly their supervisor explaining how you are going to collect evidence Describe how feedback will be given Ensure they are ready to start – don’t commence until the candidate has indicated they are ready. 26 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Gathering evidence The process of gathering evidence can be involved or straightforward depending on the candidate and their circumstances. It should be the key focus of the assessment activity, and provide a clear basis to make a decision regarding the competency of the candidate. Many assessors make the mistake of not collecting the right forms of evidence and later require more to help them make their decision. The key to good assessment lies in the assessor’s capacity to collect with the rules of evidence in mind as well as by being able to do this in an efficient manner. Organisation is the key in this situation as you may have to collect a large amount of evidence which needs to be reviewed and recorded. In collecting evidence within workplaces, you should ensure that your activities are not likely to cause disruption to normal business. A way to minimise this problem is to ensure relevant people within the organisation are notified in advance in order to minimise any inconvenience. The presence of an assessor may be quite distracting and require the assistance of other people. This means you should try to fit in as best you can. You should also ensure that necessary resources and arrangements to undertake assessment have been attended to. In order to do this, relevant equipment, consumables, venues and even the assistance other of people need to be arranged. Arriving for an assessment without these things in hand is likely to cause you and the candidate considerable stress. With all required resources in place you can concentrate on evidence collection. Assessors should also be prepared to gather evidence in a holistic manner. Many activities and tasks that you observe individuals perform (or verified records of such work) will employ various skills and knowledge to complete. In such cases, rather than assessing a candidate unit by unit, it is often possible to cluster units and assess them in the context of a whole task. An example of this could be someone working in a call centre who is to be assessed against telephone skills and basic computer use. Whilst undertaking their normal duty of taking calls and using the computer the assessor should be able to employ a holistic approach and gather evidence against both of these areas at once. The safety of you and the candidate are paramount in any situation. For this reason you should only assess where you feel that there are no safety hazards and nothing can endanger you or the candidate. In the same way that before a training course we ensure the room is safe and the group is comfortable. In assessment we also have a duty of care to the people we are working with to ensure all OH&S considerations have been seen to. Another thing to keep in mind when collecting evidence is to be aware of any workplace standards or performance criteria which may relate to the job of the candidate being assessed. This could be useful evidence in support of the other activities the candidate has to undertake and in many cases they may align closely with the unit of competency being covered. You will be collecting evidence against a range of standards but should also consider any internal industry or workplace standards which may be useful in determining the competency of the candidate. The successful performance of internal standards which can be verified by a third party may cover some or all of the areas being covered via an external competency standard. For this reason, we should carefully consider as evidence any standards which can be produced. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 27 Element 2: Provide support for enterprise assessors What are we assessing and how can we do this? The assessment process concerns itself with assessing competency against three principle areas: Skills In assessing skills we are focussing on the actual performance of a skill which can be observed and recorded. This may entail a person carrying out a component of a job in the workplace which requires skill. Observation is the most typical way we would assess skills, but we might also gather evidence based on the application of skill. The candidate might produce something which they made or even organised and this will show a level of skill in itself. Such products or samples will need to be verified to ensure that the output in question was actually their own. Knowledge As well as the performance of a skill, assessors also need to check on underlying knowledge and understanding. It is possible for someone to carry out a task without understanding what they are actually doing. The expression “monkey see, monkey do” might describe this well. Assessing knowledge via tests, questioning, interviews and samples helps us to ensure that any observed skill is supported by appropriate levels of knowledge and comprehension. Attitude It is possible for a candidate to have good knowledge and an ability to apply that knowledge in a skill sense but still not have the correct attitude required to complete a task to an appropriate. This might be significant in an area such as customer service, where an attitude to please is as important as the ability to carry out tasks and use knowledge. A lot of the assessment of attitude is implied rather than being directly seen in performance standards. You will need to focus on outcomes which show that application of skill and utilisation of knowledge is accompanied by an attitude appropriate for the task and circumstances involved. Reasonable adjustment In enabling assessment to take place in a way which suits individual needs, the principle of Reasonable Adjustment will need to be applied. This refers to the situation where an Assessor adjusts assessment approaches to accommodate the special needs and characteristics of a candidate to enable them to demonstrate their competency. An example could be a visually impaired person who may not be able to demonstrate competence in the typical manner but when a reasonable adjustment is made, perhaps giving them a modified computer terminal or information in Braille they are able to provide appropriate evidence. Reasonable adjustment may include: Taking into account language, literacy and numeracy requirements Using adaptive technology or special equipment Providing personal support services (e.g. interpreters, carers, readers) Providing assessment materials in a variety of formats (online, DVD) Making adjustments to the physical environment Considering their cultural beliefs, traditional practices and religious observances Considering their age and gender. 28 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Whilst these factors should be taken into account when making reasonable adjustment, the criteria against which the assessment is being carried out and the critical evidence will not change. In the end, candidates must still produce evidence like anyone else involved in the process! The use of questioning in assessment When collecting evidence to demonstrate skill, knowledge or appropriate attitude, one of our most useful tools is questioning. Questions can be used for a number of different reasons and they can vary in their application. Assessment tools will usually include structured questions, meaning that these are used with every candidate regardless of the location. Assessors will also need to use more unstructured questions which are geared towards particular situations or which seek additional information on top of questions already asked. Some common kinds of questioning techniques which assessors might use are: Type of question Explanation Example Closed Require a yes/no type response Do you know how to make an Espresso coffee? Open Encourages open ended answers What is most important in making an Espresso? Probing Used to get more information on a question which has already been asked When a person replies to your question on espresso with the statement “lots of things” you may ask something like “what kinds of things in particular?” Hypothetical Ask a candidate what they would do in a certain situation. These are often used where the candidate cannot carry out a particular task for some reason or they lack particular experience Tell me what you would do if the system went down? Behavioural Focus on actual experience. In this way a candidate will explain what they actually did Tell me what steps you took when the system went down? © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 29 Element 2: Provide support for enterprise assessors Matching training to needs It is important that training matches the needs of the participants and organisation. It is also rare to find suitable people within an organisation who can conduct all of the assessment needed. There will be times when there are not enough qualified or experienced people to carry out the assessment procedures within the organisation. Assessors will then need to be trained to be able to continue to implement an assessment system effectively. This training could be conducted for many reasons including: Having to meet external requirements where accredited training such as initial assessor training or additional training is needed to meet assessment system standards Situations where assessors need a refresher course to bring them back up to the required standard, if it has been a long time since they last conducted assessment Situations where there is an identified gap in assessor competencies. This could be due to industry changes or because new standards are developed internally which previously trained assessors are not experienced with Contexts where trained assessors may need job-specific training to provide added skills in order to assess areas that they are not familiar with. Example Assessment Plan Training and Assessment Pathway Candidate Rachel Watson, Office Trainee, ABC Transport and Logistics Assessor Sharon Smith, Workplace Trainer and Assessor, Big Training Institute Date prepared 3 March 2010 Date assessment due The assessment process will cover 1 month from the date of training. If additional time is needed then this will be negotiated with both Rachel and her Supervisor Assessment Benchmark or Standard Prepare workplace documents Purpose of assessment Identify and assess current skills and knowledge of Rachel Context of assessment Rachel’s job involves her preparing reports, invoices and other documents for the office staff at ABC Transport and Logistics. The assessment process involves using actual work activities conducted in Rachel’s office. These activities have been designed to recognise Rachel’s competence in preparing workplace documents 30 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Assessment methods Assigning Rachel with a written resource and written assessment Observing Rachel preparing workplace documents Questioning Rachel on her knowledge of computer programs, preparing, formatting and editing documents Reviewing the evidence Rachel provides in her portfolio Reviewing the report from Rachel’s supervisor in her ability to prepare workplace documents Assessment tools 3rd party report Observation checklist Oral questions to be used in conjunction with the observation checklist and written assessment Written assessment Evidence portfolio Location of assessment Rachel’s office at ABC Transport and Logistics, 61 – 69 Kent St Sydney NSW 2000 Venue requirements Access to Rachel’s office equipment including a printer and computer Evidence requirements Rachel’s assessment must confirm appropriate knowledge and skills to: Locate, interpret and apply relevant information Plan document in accordance with guidelines/pro-forma’s Prepare and check document using appropriate technology Complete forms to meet regulatory and workplace requirements Convey information in written and oral form Select and use appropriate workplace colloquial and technical language and communication technologies in the workplace context. Additional personnel Ashleigh Ferguson, Rachel’s Supervisor at ABC Transport and Logistics Provision for specialist support Rachel has been Language, Literacy and Numeracy assessed and does not have these issues. Specialist support is available if needed © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 31 Element 2: Provide support for enterprise assessors OHS issues OHS contact Reece Wilson is available Emergency evacuation procedures reviewed and available If First Aid kit is needed who is the contact Reasonable adjustment/s Work based project or case study may be used instead of written assessment if needed Special arrangements Sharon Smith must organise the next assessment visit with Rachel at the end of the training visit. This will then be approved and confirmed by ABC Transport and Logistics Recording and reporting arrangements for assessment Feedback form Assessment report Portfolio of evidence Evidence collection and evaluation checklist Review mechanisms Assessment Validation Candidate Feedback Form By signing this Assessment Plan all parties agree to the content: _______________________ Assessor signature _________________________ Print name ____________ Date _______________________ Candidate signature _________________________ Print name _____________ Date _______________________ Employer Authorised signature _________________________ Print name _____________ Date 32 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Activity Look at the qualification and skill chart then answer the questions below. Name John Lyons Patricia Simpson Tuyet Nguyen Qualification Bachelor of Tourism Master Assessor Diploma of Occupational Health and Safety Certificate of Hospitality: Cooking Basic Food Hygiene Certificate Made Alatas Diploma of Hospitality Operations Experience Current position 10 years tour guiding Tour guide leader 5 years OH&S manager OH&S manager 1 year as a Commis Chef Commis Chef No experience New employee 1 Could John Lyons assess tour guide skills? 2 What extra training might Patricia need to become an OH&S assessor? 3 Could Patricia assess a Barista’s coffee making skills? 4 What skills could Tuyet assess? 5 Could Made assess a waiter for table service skills? If not, what training would he need to do to be able to assess? © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 33 Element 2: Provide support for enterprise assessors 2.3 Provide for moderation of assessments Introduction After completing assessment of participants or candidates, it is important to check that the assessment results are fair and a true reflection of the areas that were being assessed. As a result, moderating assessment results through evaluation helps to maintain the integrity of the assessment system and there are many factors which influence moderation. The behaviour of the Assessor In order to establish the correct working relationship with candidates, assessors need to have good interpersonal skills in the following areas. Perception Because our ability to receive information through our senses is limited, we need to develop and use all our senses to make the most of our ability to communicate and not prejudge those we deal with. To ensure your perceptions about the people you are assessing are accurate, you must be careful to check your assumptions before you act on them. In other words, don’t stereotype. Candidates might be stereotyped because of the way they dress, speak or present themselves. We have to be careful that we are only assessing these people on the appropriate criteria and any perception we have of them does not get in the way of an objective decision. Remember the technical principle of assessment which states that we have to be fair! Active listening Listening and speaking are the most common forms of communicating and form a critical part of the assessment process. Usually, if we are positive and create the right environment, we will have little trouble getting people to talk. But how effectively do we listen to what they say? Listening is perhaps the most overlooked form of communication. Listening is a demanding and complex activity as well as being an acquired response. It is particularly important when collecting evidence as oral questioning and discussion usually forms an important part of the evidence collection process. 34 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors As assessors we need to be good proponents of what is known as Active Listening. This is all about concentrating on the job of listening and not just hearing. To listen actively means to tune in to what thoughts and feelings are being conveyed by the speaker. Some techniques that will help us actively listen are: Reduce distractions - It is important to eliminate both internal and external distractions. When the information is really important, try to remove anything that will interfere with transmission and reception Don't jump to conclusions - Even though we agree it is important to understand a person's ideas before making judgements, we are often guilty of jumping to conclusions. This is more likely to happen if we do not agree with what is being said. We don't always jump to negative conclusions; sometimes we make favourable decisions based on insufficient information. It is important to listen first, make sure we understand and then evaluate Identify key ideas - Use the spare time you have after receiving the message to identify the main theme being presented. If you are unable to understand what the speaker is telling you, use questioning and paraphrasing to clarify the information Ask questions. The methods we have discussed so far are passive in that they require no verbal input from the listener. Even though it is important not to take over the communication, it is necessary to clarify any points you don't understand. Asking questions seem to be a basic technique but how often do we take full advantage of this simple skill? Paraphrase Paraphrasing involves reflective listening. It requires you to restate in your own words the message the speaker sent without adding anything new. Paraphrasing consists of two parts: A statement of what the listener thought the speaker meant A request for clarification Paraphrasing can be particularly important in terms of checking understanding with the candidate and ensuring that we both understand what is being communicated. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 35 Element 2: Provide support for enterprise assessors Empathy Empathy is often described as walking a mile in someone else’s shoes. Active listening skills will not only help you to improve your communication with candidates, but will provide opportunities to help you understand their point of view. Moderation A useful tool which many organisations use is known as Moderation. This is a process where you can compare your own assessment activities with that of other people and evaluate your methods and procedures. It is concerned in particular with bringing assessment judgements and standards into alignment, thus ensuring the same standards are applied to all assessment results within the same unit of competency. Moderation will ensure that, as much as possible, quality standards will be achieved across a range of different assessors. Moderation also ensures minimum standards are met in terms of documentation e.g. version control, unit codes, unit name, assessment matrix and company branding. A common way of carrying out validation will be for assessors to receive evidence from a candidate who has already been seen and assessed by someone else. The assessors will examine the evidence again as if they are doing another assessment and then make their own judgement. After this has been done, the judgement of the reviewing assessors will be compared with the judgement of the original assessor and any differences of opinion will be examined to assist in ensuring a consistent approach in the way the organisation’s staff undertake assessment activities. The net effect of moderation should be to use the findings to improve assessment quality. An example may be that after going through this process, an organisation found that its assessors seemed to be employing different approaches to the same sort of assessment situations. Hopefully this would be identified via moderation and after group discussions and consideration of the respective approaches, a more standardised approach adopted and written into procedures. If such activities are undertaken regularly as part of a continuous improvement cycle, the quality of assessment practice will always improve. 36 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors 2.4 Provide feedback to assessors on their performance Introduction An important part of assessment is giving feedback to the candidate regarding their performance. Most people will be very keen to hear how they went and the assessor needs to be able to do this in a truthful and professional manner. When providing feedback you should be clear about the areas to be covered (in this case the relevant performance criteria). The feedback should always be constructive and if the candidate is judged as not yet competent, it should focus on what the candidate has done well and clearly explain the areas still requiring attention to bring them up the required standard. Making the assessment decision Once evidence collection has been completed and the candidate has had sufficient opportunity to demonstrate their ability for the assessor, the time comes for an assessment decision to be made. Whilst not a judicial process, assessors must still carefully weigh up the evidence they have collected against the performance criteria and check it off against the four rules of evidence. Assessing against the performance criteria is critically important because they also reflect the dimensions of competency which make up the ability to successfully accomplish tasks in the workplace. An example of a decision grid that could be used for this process is included below. Once you have gone through this process you are in a good position to justify your decision and to provide feedback to the candidate on why they were competent or not yet competent. Candidates are rated as Competent or Not Yet Competent where further evidence is required. Assessment matrix The competency standard being assessed Critical evidence required Evidence collected Does it relate to the standards being assessed? (Valid) Is there enough of it? (Sufficient) Is it up to date? (Current) Is it the candidates own work? (Authentic) © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 37 Element 2: Provide support for enterprise assessors Providing feedback to the candidate A suggested approach to delivering feedback is included below at Example 4. In this situation Step 1 is about you describing how the candidate has gone and why they have or have not achieved competency. The idea is that the candidate moves to the point of understanding (Step 2). This is exactly the situation as the assessor describes it. They now have a chance to respond to the feedback and the assessor needs to listen carefully to this to pick up on any concerns or issues they may have. After listening the assessor will be in a position to talk about the options (Step 3) moving forward with the candidate and will need to be assertive where necessary to restate any results from the assessment. This will then lead on to Step 4 where the assessor, in consultation with the candidate, will develop an action plan. This may involve steps towards achieving the next units of competency or an opportunity for re-assessment. The assessor will then get any final feedback from the candidate before setting dates and deadlines. Giving Feedback 1. Explaining situation Final Feedback from candidate Feedback to candidate 4. Selecting options for action. Develop action plan Listening to candidate Assertion 3. Identifying Options 38 2. Understanding © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Appealing a decision In some cases a candidate will not be satisfied with the explanation they are given by the assessor as to why they have not achieved competency. These situations are rare but organisations have obligations to deal with such complaints. Grounds for an Appeal against an assessment decision must be based on one of the four technical principles of assessment i.e. was the assessment conducted in a manner which was valid, reliable, flexible and fair? Organisations must ensure that participants undertaking training and assessment are made aware that they have the right of appeal in respect to assessment decisions and that there is a clearly defined and documented appeals procedure in place. In most cases the appeal will consist of a more senior member of the organisation or even a dedicated panel in larger organisations reviewing the assessment decision. Example of participant evaluation form Post-course Evaluation Form Training Program Master Trainer and Assessor Training Program I attended the following training program (please indicate) Master Assessor Training Program Master Trainer and Master Assessor Participant Name Gender Position Organisation Country Date © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 39 Element 2: Provide support for enterprise assessors 1. Program Objectives: Master Trainer As a participant of the Master Trainer Training Program answer the following questions by circling the appropriate response: 1.1 1.2 1.3 2. I am now able to provide training on the features and contents of the training program. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 1 2 3 4 5 I am now able to demonstrate and provide opportunities to trainers on how to use the program in a training context. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 1 2 3 4 5 I have gained the knowledge and skills required to successfully deliver training on using the materials to other teachers/educators. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 1 2 3 4 5 Program Objectives: Master Assessor As a participant of the Master Assessor Training Program answer the following questions by circling the appropriate response: 2.1 2.2 40 I am now able to provide training on how to use the materials as an assessment tool as part of a certification program Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 1 2 3 4 5 I have gained the knowledge, skills and attitudes required to successfully train other assessors on how to conduct assessments. Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree 1 2 3 4 5 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors 3. Evaluation of your response to the Workshop: 3.1 Did the workshop satisfy your original expectations when you commenced the course? Please give details. __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 3.2 If it didn’t meet all your expectations then why was this the case? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 3.3 What would be the best way to assist you to do this? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 4. Evaluation of the Workshop and Presenters Presenter(s) Qualities Briefing Materials Knowledge of material covered Presentation Clarity of explanations Relevance of topics Enthusiasm/maintained interest Workshop materials Question handling Overall Score Encouragement of participation Average score for each session attended Personal organisation Scale: 4 = Excellent 3 = Very Good 2 = Satisfactory 1 = Poor © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 41 Element 2: Provide support for enterprise assessors 5. Preparation for Workshop and living arrangements Please comment on the following aspects of the program: a) Application, interview and selection process ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ b) Venue facilities and meals ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ c) Hotel accommodation: (i) Big Resort and Spa _____________________________________________________________ OR (ii) Sands Hotel _____________________________________________________________ 6. Do you have any suggestions on how to improve the arrangements mentioned above for any future Workshop training? ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ 42 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors 2.5 Facilitate networking amongst assessors Introduction Often the best source of feedback ad information is from your colleagues and fellow assessors. As all of the assessors are involved in the same system, sharing information, issues, and ideas with each other is an effective way to ensure the quality of the assessment system and implement improvements as needed. Developing a network of assessors is excellent way to maintain the integrity of the assessment system and allows assessors to provide useful feedback as required. Developing networking Establishing networking amongst assessors provides opportunities for assessors to share information and ideas about their assessment activities. Networks allow assessors to: Learn, discuss and share information about assessment issues Promote the best standards for assessment Explore ways to improve the assessment system Learn from external assessment agencies Keep informed about new changes or standards Keep engaged with their assessment responsibilities. Networks can be developed in many different ways and may include: Formal associations Informal groups Internal organisations within a company External agencies linking groups of assessors. There are also many different ways that networking can be developed including: Meetings both regular and as required Seminars and workshops Conferences Notice boards, handouts, newsletters Electronically through websites, emails, blogs. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 43 Element 2: Provide support for enterprise assessors Network models Unstructured Structured • Relatively informal • May move into an incorporated body • Information exchange • Peer support focus • Depends on energy of one person or small group to drive network • Structured meetings with protocols • Often self-managed • Usually not funded • Combines informal learning with 'events' determined by the group Organisation sponsored Self-managed • Initiated by organisation rather than assessors • May begin being hosted by the organisation • Closely aligned with organisational purposes • Tends to be highly focused • Reliant on host organisation for support • May not be a network rather than databases and mailing lists • Requires time to reach selfmanagement • Pursues group outcomes as well as individual learning • Needs to develop group skills re: negotiation, consensus, planning Source: adapted from Jackson, E., 2000, Cross Boundaries –Effective Learning Networks 5-7, Commonwealth of Australia. 44 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 2: Provide support for enterprise assessors Work Projects It is a requirement of this Unit you complete Work Projects as advised by your Trainer. You must submit documentation, suitable evidence or other relevant proof of completion of the project to your Trainer by the agreed date. 2.1 Find out what external requirements for competency standards training providers or government organisations have for staff in your organisation. What are your internal competency standards? List them. 2.2 Interview an assessor and see if the assessor’s qualifications and experience match internal and external requirements for competency standards. Suggest what training might be necessary for this assessor to meet requirements. 2.3 Develop an assessment matrix which includes all the necessary information to conduct assessments. Complete the matrix for at least two competencies within your organisation. 2.4 Complete an assessor feedback template that includes all of the necessary information to provide relevant feedback to an assessor about their performance. 2.5 Provide your trainer with examples of how assessors network within your organisation and if applicable, with outside assessors. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 45 Element 2: Provide support for enterprise assessors Summary Provide support for enterprise assessors It is important to make sure that the assessors within an organisation are capable of assessing effectively. In order for assessors to be able to perform their duties properly they must: Meet the required competency standards or they will not have the skills to conduct the assessment Be trained if they do not have the required skills or experience Have opportunities for moderating assessments where assessment items are checked against different parameters to ensure their relevance and effectiveness Have opportunities for feedback to be provided relating to their performance in order for assessors to be able to improve their assessment skills Have an opportunity to network with other assessors in order to discuss issues and develop strategies for best practice in assessment delivery. 46 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system Element 3: Manage the assessment record keeping system 3.1 Identify and develop the internal records required to support the assessment system Introduction An important part of training and assessment is keeping records of contact with participants. In group training situations this will normally involve a roll book or attendance sheets. Candidates seen one on one will still require sheets which record their visit or assessment process. Carefully maintained records are important to track a participant’s progress and learning within an organisation. Recording and reporting assessment A critical part of the assessors work involves accurately recording the process of the assessment, activities undertaken, evidence collected and decisions made. There are different formats for recording assessment activities but generally speaking the following things will be recorded: Candidate name/personal details Training rolls/attendance records for groups Date and time of assessment The location of the assessment Copies of relevant training curricula/training plans Units or standards being assessed Methods used to assess Assessment results Comments and feedback Assessor name. The information collected during an assessment should be treated as confidential as it relates to the personal performance of the candidate. As a rule of thumb, the only people who should have access to this information are the assessor and their manager, the candidate and their manager and people within the training system involved in auditing or checking on the work of assessors. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 47 Element 3: Manage the assessment record keeping system As well as being important for the assessor and associated organisation, recording is also important for the candidate as learners should have timely access to current and accurate records of their participation and progress. We need to be able to show them any comments that have been made to assist them with their development. An example of a report format for an assessment visit is included below. Example Assessment Report Candidate: Joe Smith Employer: ABC Hotel Assessor: Matt Davis Date: 16 May 2013 Comments: Maintain Workplace Safety Joe described his role at the hotel and his responsibility in ensuring that he follows the various occupational health and safety requirements within the office. He completed the set assignment work and went through this with the assessor, explaining his answers. Joe took the assessor on a short tour of his work area, pointing out various safety issues and explaining how he maintains a safe work area. Joe gave examples of his previous job in a gaming room and some of the hazards which also existed there. Joe has a good appreciation of the need to sit properly, have breaks and ensure he lifts properly. Through questioning and discussion he was able to demonstrate a good level of knowledge and ability in this competency. This was corroborated by his supervisor. Competency successfully achieved through completion of exercises, questioning and demonstration. Organise workplace information Joe carries out a lot of his duties within the mail room at. In this capacity he is required to organise workplace information on a regular basis. He completed set exercises relating to mail handling and provided examples of various mail items which he must ensure are properly addressed. Joe took the assessor to his work area and explained how mail, both internal and external is sorted to ensure it goes to the correct recipients. He also explained this for the assessor in written form. Through questioning and discussion, Joe was able to explain the importance of correctly organising information within the mail room and his contribution to this. He also cited pertinent experience from his previous job in a gaming room. Competency achieved through completion of set exercises, submission of example, demonstration and answers to assessor questions. Performance Criteria Commenced Assessed Evidence collected Maintain Workplace Safety 03/11/2012 13/04/2011 Observation checklist Organise Workplace Information 03/11/2012 13/04/2011 Oral questions Written assignment 48 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system Recognition of Prior Learning (RPL) RPL is an assessment process where the skills, knowledge and experience that a person has obtained via learning outside the formal education and training system is acknowledged towards their achievement of competency. This is still a formal assessment process but instead of following training, involves an individual demonstrating via discussions with an assessor and submission of documentary evidence, competency against nationally endorsed standards. The chart below includes an example of a RPL kit which assessors could use as participants progress through the process. Where candidates are seeking RPL, they may or may not gain full RPL against the units or standards required and may still be required to undertake some gap training in areas where they do not yet have appropriate skills, knowledge and experience. The amount of RPL a person receives will be determined by the assessor by analysing the evidence presented against the relevant standards. In the RPL process, candidates are instructed to collect various forms of evidence for consideration by the assessor. This evidence may include some or all of the following: Direct Which the candidate has been observed actually doing via the report of a third party Indirect Which consists of information about the candidate such as testimonials, awards, prizes, personal statements and are a description of the candidate and their achievements which act in a supporting manner to the other evidence tendered Historic Such as a past academic statement and must be presented with at least one form of indirect evidence Supplementary Consisting of the answers to oral/written questions or other activities posed by the Assessor to add depth to evidence already submitted, especially in the context of completing workplace tasks. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 49 Element 3: Manage the assessment record keeping system Simple assessment tools In order to assess in a formative or summative manner the assessor needs to collect evidence from a candidate that they meet the required benchmarks. Assessors do this via the use of an assessment tool which put simply are the materials, instruments and procedures which enable the assessor to collect and interpret evidence of competence. A simple tool will include an instrument section which comprises activities or questions which must be completed as well as a procedures section containing information or instructions which need to be given to the candidate and the assessor to explain how the assessment will be conducted. Simple assessment tools may be developed by the assessors themselves, be developed by dedicated people within the organisation or come from commercial resource writers. Whatever their origin, they need to be structured with the following things in mind. Firstly, do the tool and its associated instruments adequately reflect the benchmark it was designed to collect evidence against? Benchmarks describe the endorsed units of competency and their associated guidelines. For this reason assessment tools may include a grid consisting of the standards being assessed and which can be used to check off evidence against the appropriate performance criteria. If you look at the example of an Assessment Tool below you will see that the activities which the candidate must be observed undertaking are recorded against the relevant performance criteria from each element of the competency. The assessment tool also needs to adhere to the technical principles of evidence. Can you say the tool reflects the unit it is meant to be gathering evidence against? Will the tool produce the same results when used across a range of industries and individuals? Does the tool lend itself to a flexible approach when being used in industry? Is it likely to unfairly discriminate against candidates because of the way it is put together or the activities and questions which are requested? In developing an assessment tool you need to ensure that it is well sequenced and easy to follow by a wide range of candidates. Any instructions should be clear and concise and leave the candidate in no doubt as to what is required of them. Many tools are also now able to be viewed via the web and electronic assessment items are becoming increasingly common. The Assessment Tool has a key role in evidence collection and for that reason needs to adhere to the rules of evidence as well. Any questions or activities or instructions on the tool which require the candidate to produce evidence need to ensure that this evidence is Valid and closely aligned with the unit being covered. Evidence also needs to be collected in sufficient quantities to meet all requirements and the tool must include evidence collection against all parts of a unit and specify a number of different forms of evidence. The tool at below requires candidates to demonstrate their skill as well as answer oral and written questions and produce samples of work in sufficient quantity to enable the assessor to make an informed judgement on their ability. Evidence specified by the tool also needs to be current and authentic, hence the requirement to produce direct evidence for the assessor which can be verified as the candidates own work. 50 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system A development approach for a simple assessment tool could include the following steps: Remember the context in which the assessment is taking place and any special considerations this may entail. 1. Select the benchmark which the tool is to be developed against. 2. Unpack the unit to determine the skills & knowledge it refers to as well as the critical evidence required to demonstrate competency. 3. Consider what a person competent in the skill/knowledge in question should be able to do? 4. Ask yourself what evidence will be required to verify the above. 5. How would your draft assessment tool’s assessment methods generate this evidence? 6. How will the tool need to look and feel depending on the people using it? © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 51 Element 3: Manage the assessment record keeping system 3.2 Describe the requirements for completing the assessment records Introduction Maintaining up to date and clear records of assessment activities is a critical part of any successful assessment system. Assessors and participants need to know how their assessment is going to be collected, monitored and stored. An assessment system may have numerous reporting formats that need to be completed and all assessors need to know how to complete assessment records properly to maintain the quality of the assessment system as a whole. Guidelines for completing assessment activities The guidelines detailed below are detailed here to support professionally responsible and ethical assessment practice and to guide assessors in the responsibilities of their work. They broadly explain how to complete an assessment activity, including reporting, effectively. This code is loosely based on an international code developed by the US-based National Council for Measurement in Education. The code reinforces the performance outcomes of an assessment system. They include: The differing needs and requirements of the candidates, the local enterprises and/or industry are identified and handled with sensitivity Potential forms of conflict of interest in the assessment process and/or outcomes are identified, and appropriate referrals are made, if necessary All forms of harassment are avoided throughout the assessment process and in the review and reporting of assessment outcomes The rights of candidates are protected during and after the assessment process Candidates are made aware of their rights and processes of appeal Personal or interpersonal factors that are irrelevant to the assessment of competence must not influence the assessment outcomes Evidence is verified against the rules of evidence Assessment decisions are based on available evidence that can be produced and verified by another assessor Assessments are conducted within the boundaries of the assessment system policies and procedures Formal agreement is obtained from candidates and the assessor that the assessment was carried out in accordance with agreed procedures Assessment systems and tools are consistent with equal opportunity legislation Candidates are informed of all assessment reporting processes prior to the assessment 52 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system Candidates are informed of all known potential consequences of assessment decisions, prior to the assessment Confidentiality is maintained regarding assessment decisions/outcomes and records of individual assessment outcomes which identify personal details and are only released with the written permission of the candidate/s Assessment outcomes are used consistently with the purposes explained to candidates Self-assessments are periodically conducted to ensure current competence against organisational or external competency standards Professional development opportunities are identified and sought Opportunities for networking amongst assessors are created and maintained Opportunities are created for technical assistance in planning, conducting and reviewing assessment practice and participating in validation. Describing requirements for completing assessment records Before assessors can complete records effectively, they must have a clear understanding of what is required to successfully complete assessment records. In order for assessors to know how to complete the records of assessment the requirements need to be explained to them. This may be done through: Demonstration where an assessor trainer or supervisor completes a form with the assessor Written guidelines that are distributed to all assessors Providing examples or templates that show how to complete the records Online tutorials with examples FAQ sheets Monitoring the compilation of records completed by assessors on an ongoing basis where feedback can be provided if required. It is important that instructions for completing assessment records are clear, accessible and regularly monitored and that feedback opportunities are available. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 53 Element 3: Manage the assessment record keeping system Assessment tool example: Prepare workplace documents Element 1: Plan workplace documents Why is workplace documentation so important to the efficient execution of your duties? What logs or diaries are you required to keep? How often must they be filled out? What sort of computer generated documents do you use? How are they generated? Element 2: Prepare workplace documents Collect four different types of workplace documents and answer the following: What is each one used for? How often are they filled out? Who is responsible for filling them out? Where are they stored? Take the Assessor through the filling out of these documents. Element 3: Complete workplace forms Imagine the following situations and explain what kind of documents you would need to complete: You have to take a day off due to illness Your home address details have changed You have been forced to halt deliveries due to mechanical problems. Produce a copy of each for your Assessor. 54 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system Prepare Workplace Documents Candidate Name: Location: Element 1: Plan Workplace Document Date of Assessment: C/NYC Method of Assessment Comments C/NYC Method of Assessment Comments C/NYC Method of Assessment Comments a) Purpose and audience for the document are identified b) Appropriate format for the document is established to meet workplace requirements c) Relevant information is identified and selected for inclusion in the document Element 2: Prepare Workplace Document Date of Assessment: a) A draft of the document is prepared in accordance with workplace procedures and conventions for sentence construction, grammar, spelling, style, punctuation and vocabulary that are appropriate for the reader(s) b) Document is edited and presented in a final version appropriate to the task Element 3: Complete Workplace Forms Date of Assessment: a) Work related form(s) is interpreted to identify information required for its completion b) Required information for completion of form is gathered from relevant sources in accordance with workplace procedures c) Form(s) is completed in accordance with workplace policy and procedures and any applicable regulations and codes Comments and evidence supplied: _______________________________________________________________________ _______________________________________________________________________ _______________________________________________________________________ Name of Trainer/Assessor and Signature: _____________________________________ Name of Workplace Coach’s/Supervisor and Signature: __________________________ Date: ___________________ Recommendation: C / NYC © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 55 Element 3: Manage the assessment record keeping system 3.3 File the assessment records Introduction Evidence of assessment is an important aspect of any assessment system and being able to access this information quickly and effectively is critical to the quality of the system. A filing system is necessary to keep information readily accessible and to make it easy to update assessment records as required. This enables an organisation to gain a clear picture of training needs and learning progress of staff. There may also be requirements from external agencies that stipulate how and for how long an organisation must keep assessment records in order for training and assessment to be recognised. File assessment records Any filing system needs to be easy to use in order for it to be effective. With computer technology readily available nowadays, a filing system that uses databases or computer generated filing systems is an easy and effective option for developing an assessment record system. This may be combined with a paper based system which offers back-up if the technology fails. If both paper-based and computer generated systems are used it is critical to have a system that enables both forms to updated, otherwise the system is ineffective. It is also important to consider who and why people may need to access assessment records before developing a filing system. Who needs access to assessment records? Assessors Managers External agencies Training providers and trainers Candidates or staff Administrators Assessment coordinators Why do they need access to assessment records? To update assessment records To evaluate assessment procedures and policies To provide feedback to candidates, staff, managers, external agencies, training providers and trainers To check learning and qualification progress To check work performance or promotion potential To identify gaps in training or skills 56 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system Any assessment record filing system should also: Be secure and keep information confidential Enable easy access to existing records Allow records to be updated easily Meet any external standards that are required by government, training providers or regulatory agencies Combine assessment records with training records in order to meet internal standards Outline procedures and methods to enable validation of assessment records. Activity Look at the assessment records below. In pairs discuss what is missing from the records. What extra information is required? Further evidence/training required Position Training completed Assessment result Bob Smith F&B Manager Customer Service Management Competent Jane Myers Front Office Manager Customer Service Management Not Yet Competent Complete customer journey map by 10/11/2012 Eli Mons Recreation Manager OH&S certificate Competent Refresher meeting required 10/6/2013 Name © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 57 Element 3: Manage the assessment record keeping system 3.4 Review and up-date the assessment record keeping system Introduction Keeping records that do not reflect the actual achievements or true results of candidates or staff is not an effective assessment recording system. Assessment records need to be kept up to date and regular reviews of the assessment record system need to be conducted to ensure that it is operating effectively. Procedures for reviewing and updating the record keeping system and when the reviews should be conducted need to be documented in order for the system to maintain integrity and quality. Reviewing and updating the record keeping system Regular reviews of the assessment record keeping system should be documented and carried out as per internal or external requirements. Reviews allow different stakeholders to provide comments and feedback relating to the record keeping system and enable suggestions and improvements to be implemented. A review mechanism should include: Dates and time of review Items for review Data/information/feedback gathered Issues or matters for attention Who conducted the review Recommendations for improvement Date for next review A mechanism for communicating updates or changes to the assessment record keeping system to relevant stakeholders. Regular meetings may also be used to review and update record keeping systems. In reviewing the system certain questions need to be asked and answered and these can be classified into different areas. The areas could include: Administration issues such as candidate information collection, staff or participant interviews, templates, database use, staffing, space requirements for storage of documents, security issues Content related issues such as new information required to maintain currency and alignment with current industry practice Technical issues including technology upgrades. 58 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system Review process Improved Assessment System Conduct Consultations with Stakeholders Ongoing Evaluation Review Cycle Implement Improvements Gather Evidence Develop Implemntation Strategy © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 59 Element 3: Manage the assessment record keeping system Work Projects It is a requirement of this Unit you complete Work Projects as advised by your Trainer. You must submit documentation, suitable evidence or other relevant proof of completion of the project to your Trainer by the agreed date. 3.1 Show your trainer three examples of assessment records that are used at your organisation. These could include examples of: Personal information relating to candidates Training rolls Copies of training curricula or training plans Assessment results for different items or competencies. 3.2 Provide your trainer with a copy of an assessment record from your organisation and explain how it must be completed. 3.3 Provide a detailed list of how to file assessment records to your trainer. Explain how security and confidentiality are maintained. 3.4 Develop an assessment review template that includes all of the necessary information to complete a review of a part or the whole assessment system at your organisation. 60 Show the template to your trainer. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 3: Manage the assessment record keeping system Summary Manage the assessment record keeping system Managing an assessment system requires good management of the record keeping system in order to maintain up to date information for all stakeholders Appropriate internal records need to be developed and clear procedures for completing the records need to be made available to assessors Keeping records in an easily accessible manner is also necessary in order to make the information available to stakeholders Reviews of the record keeping system are an important part of the overall assessment system and they should be completed at regular times in order to maintain current information within the record keeping system. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 61 Element 3: Manage the assessment record keeping system 62 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 4: Maintain quality assurance procedures Element 4: Maintain quality assurance procedures 4.1 Define quality assurance procedures for the assessment system Introduction All aspects of assessment need to be regularly monitored, reviewed and improved. This is driven partly by our need to improve as professional practitioners, partly due to changes and improvements in our understanding of assessment processes, partly to meet changes in outcome standards, and partly to respond to changes within industry itself. Quality Assurance procedures or continuous improvement is about applying good business practices to ensure the best outcomes for our clients, namely: students, staff, industry and the community. Our vigilance must extend beyond our own appraisal of the assessment systems we have established. We must therefore seek and incorporate feedback and advice from industry, employers, other assessment professionals and the people we assess. Continuous improvement Continuous improvement and quality assurance are based upon assessment validation, which is defined as follows. Validation is a quality review process. It involves checking that the assessment tool produces valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the training package or accredited course have been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes. (Source: AQTF Users’ Guide to the Essential Conditions and Standards for Continuing Registration) Why do we need to get involved in quality assurance? The quest for continuous improvement is a defining characteristic of professional practice. There are many reasons why organisations need to engage in the quality assurance of assessment systems as professionals in the hospitality and tourism sector. These include: Confirming the credibility and recognition of certification and awards Supporting the industry and community recognition of training participants Providing the best service for stakeholders – learners and industry Ensuring assessments reflect changes in current industry requirements © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 63 Element 4: Maintain quality assurance procedures Improving the validity, reliability, flexibility and fairness of assessments Providing greater justice for participants for assessment with improved relevance, transparency and support in assessment processes Ensuring more meaningful feedback and guidance for students after they have been assessed Supporting ongoing organisational quality assurance Enhancing the reputation and recognition of the organisation and its services Minimising risks associated with the assessment process Improving the management of assessment systems Informing assessor selection and guiding professional development. Defining quality assurance procedures There are certain areas that need to be outlined in implementing quality assurance procedures. Systematically collecting and analysing data Systematic approaches support quality assurance. They may include: Planning where data will be collected from, how it will be collected, the form it will take, how often it will be collected, and how it will be collated, analysed and used Ensuring that data collection and analysis confirm good or best practice and show where improvements need to be made Making improvements where analysis demonstrates they are needed Regularly reviewing data collection to assess its usefulness for improving products and services Giving feedback to those who have contributed to the data. Ensuring that data is relevant and sufficient The focus of qualitative data collection (for example, feedback from assessment moderation meetings) and quantitative data collection (for example, records of assessments undertaken and judgements made) could be informed by: Prior continuous improvement activities Feedback from stakeholders such as students, employers, industry Quality indicator data Assessing the relevance of the collected data to the lecturer’s training and assessment outcomes; and Deciding which aspects of training delivery are most critical to the organisation’s quality training and assessment. 64 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 4: Maintain quality assurance procedures Data sources relevant to improving training and assessment could include: Client satisfaction surveys/questionnaires Interviews, focus groups, and/or other data from consultation with students, enterprise clients, industry organisations and licensing bodies Records of staff/planning meetings and agreed actions Records of complaints and appeals, and their resolution Internal audit reports and organisational self-evaluation Staff performance appraisal reports. Demonstrating improvements Improvements to training and assessment could be demonstrated by changes to: Quality, currency, relevance and sufficiency of training and assessment resources, including reasonable adjustments made to meet the needs of participants with a disability, or other valid reasons for adjustment Professional development activities and outcomes Validity, reliability, flexibility and fairness of assessment processes. For continuous improvement, revised practices are analysed in light of further data collection. Ways of identifying opportunities for the improvement of assessment strategies could be through assessment system validation and the systematic improvement of processes set out above. Opportunities for improvement may also be identified through collaborative partnerships and risk management processes. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 65 Element 4: Maintain quality assurance procedures 4.2 Undertake internal audits of the assessment system Introduction Internal audits are an important part of an effective assessment system. These audits allow the organisation an opportunity to identify strengths and areas for improvement in the system. Audits can vary in what aspects they are focusing on and can include whole audits or part audits depending on the need or the organisation’s policy. Internal audits are a part of the quality assurance process and they should be conducted regularly. Conducting internal audits Internal audits are a necessary aspect of an assessment system and may be required by external agencies in order for an organisation to meet certain standards. Internal audits should: Be regularly scheduled either in full or parts of the system allocated for review/audit Have the areas for the audit clearly defined and the cycles for auditing clearly stated Include training for staff undertaking audits Allocate sufficient resources for conducting the audit including personnel, equipment, time and support Provide access to necessary documents to record audit information and results Be able to record information and store this information effectively and make it easily accessible Allow for evaluation and recommendations Include mechanisms for distributing results and recommendations to relevant stakeholders. What aspects of an assessment system need to be monitored for an internal audit? In developing quality assurance procedures for the assessment system there are five targets for validation that can be identified which should be included in an internal audit. They include: Assessment systems Assessment processes Assessment methods and tools Assessment evidence Assessment judgements. 66 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 4: Maintain quality assurance procedures Assessment systems include but may not be limited to: Grievances and appeals processes Assessment processes Validation systems and processes Administrative procedures such as reporting/recording arrangements Quality assurance mechanisms Risk management strategies Acquisition of physical and human resources Identifying roles and responsibilities Establishing partnership arrangements. Assessment processes include but may not be limited to: Providing RPL; dealing with appeals, complaints and grievances; managing resources and partnership arrangements Designing assessment methods/tools Selecting, managing and monitoring lecturers and providing them with professional development Managing the gathering of evidence, including third-party evidence gathering, workplace assessment, simulation and record keeping Making and recording judgements Providing students with information Providing students with feedback and guidance Ensuring validity and reliability, flexibility and fairness. Assessment methods and tools are used to gather assessment evidence, and may include but not be limited to methods such as: Observation of performance through normal work activities or simulated workplace activities Examining workplace products produced by the student Questioning and/or conducting interviews Seeking third-party reports Structured activities (for example, role-plays, projects, presentations) Examining portfolios. And tools such as: Observation checklists (for example, workplace and/or simulation) Knowledge tests (for example, written and/or oral) Third-party questionnaires The instructions provided to students and instructions for evidence gatherers. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 67 Element 4: Maintain quality assurance procedures Assessment evidence is information gathered which, when matched against the unit of competency requirements, provides proof of competence. Evidence can take many forms and be gathered from a number of sources. Assessors often categorise evidence in different ways, for example: Direct, indirect and supplementary sources of evidence Evidence collected by the student or evidence collected by the lecturer Historical and recent evidence collected by the student and current evidence collected by the lecturer. Quality evidence is valid, sufficient, current and authentic evidence that enables the assessor to make the assessment judgement. Assessment judgements involve the assessor evaluating whether the evidence gathered is valid, sufficient, current and authentic in order to make the assessment decision. The assessment decision will require using professional judgement in evaluating the evidence available about: The quality of evidence gathered using the assessment methods/tools The competency achievement of the participant based upon that evidence. Example Assessment System audit sheet Section for audit Specific area for audit Standards Result and recommendations Risk Participant personal details completed Compliant: All samples checked were completed correctly. Maintain regular monthly checks. Low Assessment records for each participant completed and up to date. Non-compliant: All but one sample checked was up to date and fully completed. Participant 1452 sample was missing assessor name. This should be completed and regular checks continued. Medium Assessment systems Administrative procedures 68 Recording assessment results © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 4: Maintain quality assurance procedures 4.3 Use the results of quality assurance audits to improve the assessment system Introduction Identifying areas for improvement of an assessment system is an important part of the quality assurance process. Once the issues have been identified it is necessary to respond appropriately to the feedback. The first step in responding to recommendations is to first ask the right questions in the relevant areas, in order to gather evidence or feedback that can then be used to improve the system. Any issues that are identified should then be used as a basis for change. It is critical that the evidence is sufficient and relevant. What strategies can we use to identify opportunities for improvement? There are three fundamental processes to identify opportunities for continuous improvement. These are: Reviewing Moderating Evaluating. Reviewing This involves the inspection of processes and products to determine the ‘face validity’ of the assessment strategy. In effect lecturers are asking, ‘Does it look right?’ and ‘Does it meet the principles of assessment?’ Examples of this process include: Mapping and matching assessment methods/tools to unit of competency and training package requirements Opinions of industry representatives about the proposed assessment process and assessment methods/tools Opinions of peers about the assessment process and assessment methods/tools Compliance of assessment processes and assessment methods/tools with internal or external requirements. These processes and other relevant indicators aid reviewing and support assessors’ professional judgement when applying an assessment strategy. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 69 Element 4: Maintain quality assurance procedures Example of reviewing The assessors working for a small organisation meet regularly with an industry focus group to seek feedback on their assessment tools. These assessors are particularly keen to be reassured that the assessment tools reflect current industry practice, and that industry is comfortable employing participants who have been judged as competent after using these tools for assessments. These assessors also meet as a team to review the organisation’s assessment process documentation and each other’s assessment tools to ensure that they match training package and internal requirements. They document opportunities for improvement identified through these internal and external reviews, along with the name of the person responsible for addressing each issue, and a date for completion. Changes made are reported at the next assessment review meeting. Comparing Professional judgement involves comparing assessment documentation and assessment outcomes to determine content validity, reliability, consistency and reproducibility of the assessment judgements. There is no single model for assessment comparisons; however, assessment moderation is a commonly used comparison process, although it is not completely flawless. There may be variation in assessors’ judgements, but moderation works to ensure that the margins of variation are minimal. Moderation includes comparing: The assessment methods/tools of two or more lecturers for the same unit(s) of competency The observations of two or more evidence gatherers for the same student’s performance The judgements made by two or more assessors based upon the same evidence The assessment results of two or more similar units of competency for the same student. For example, two units covering different technical activities but involving similar application of problem-solving skills can be looked at to determine that this aspect of the student’s competence is being validly and reliably assessed. Example of comparing Assessors from a number of organisations meet to carry out assessments of the competence of a candidate using a video recording of the participant’s performance and samples of the products produced by the participant. The assessors use their own assessment tools to assess the performance and products then compare their judgements and the evidence behind their judgements. Differences in the evidence gathered and in judgements are discussed to clarify the interpretation of the unit of competency until consensus is reached. Individual assessors then modify their assessment processes to maintain that consensus in future. Each organisation records its own participation in the meetings and the improvements made as a result of the meetings. 70 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 4: Maintain quality assurance procedures Evaluating This involves gathering stakeholder feedback to determine the predictive validity, impact, effectiveness and credibility of the assessment strategy. Assessors can evaluate the following five kinds of impact. 1. How it feels: Does the person feel comfortable with the assessment process? 2. How it works: Can the person apply the competency in a workplace role? 3. How it is used: Is the assessment process properly applied? 4. How it serves: Does the assessment process contribute to industry and organisational objectives? 5. How it rates: Is the assessment process the most time-effective and cost-effective option? Examples of gathering feedback include: Participants’ opinions of their assessment experience Evidence from assessment appeals, complaints or grievances about assessment Feedback from employers and industry representatives about participants’ competence Opinions of evidence gatherers about the assessment resources and processes, and their ease of use Feedback from auditors about non-compliances or opportunities for improvement. Example of evaluating To identify possible improvements, an organisation routinely surveys its current staff and trainers to gauge satisfaction with its training delivery and assessment services. Staff and trainers are surveyed by written questionnaires and managers are interviewed. Feedback relating to assessment processes is then provided to assessors/managers and action taken. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 71 Element 4: Maintain quality assurance procedures 4.4 Review the applicability of the existing quality assurance procedures to the dynamic nature of the assessment system Introduction As has been seen in previous units, reviewing and moderating the assessment system is an important aspect of maintaining the quality of the system. Most systems are ever-changing and dynamic. They are neither stagnant nor stable. It is important then to also check the quality assurance procedures to make sure that they are responding to the changing context in which they have been developed. The critical factor in reviewing the applicability of the quality assurance procedures is in including as many relevant stakeholders in the quality assurance process itself and allowing them the opportunity to provide honest and relevant feedback. Gathering evidence for quality assurance to ensure its applicability Just as assessors use an assessment plan to map their evidence-gathering sources in assessment, they can also plan their evidence-gathering sources for continuous improvement using a simple table. This should help them identify what they need to ask each stakeholder group (by reading the columns in the table below) and who they need to consult for each aspect of continuous improvement (by reading the rows). This allows specific information about the quality assurance process to be obtained from various stakeholders, which in turn provides a more thorough appraisal of the quality assurance procedures. In the table below, the demands upon each stakeholder group are limited to those aspects for which each is best placed to contribute. 72 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 4: Maintain quality assurance procedures Primary evidence sources Continuous Improvement Activity Industry Employers Review assessment process Review assessment methods and tools Technical and subject matter experts Review assessment judgment Review assessment evidence Participants Trainers Assessors Compare assessment methods and tools Compare assessment evidence Government and External agencies Compare assessment process Compare assessment judgement Evidence gatherers © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 73 Element 4: Maintain quality assurance procedures Primary evidence sources Continuous Improvement Activity Evaluate assessment process Industry Employers Evaluate assessment methods and tools Technical and subject matter experts Trainers Evaluate assessment evidence Evaluate assessment judgement Assessors Evidence gatherers Participants Government and External agencies Based on this table, assessors or reviewers should develop a set of questions that they may want to put to each group to determine if the assessment system quality assurance procedures are applicable to the system. It is important for assessors to bear in mind that ‘assessment’ includes assessment associated with training delivery and assessment through RPL and to take into consideration that some stakeholder groups may not be familiar with all the internal/external terminology and, as a result, some questions may need to be worded in a way that is accessible to all stakeholders. The answers to these questions become the items or results that should be used for the basis for changes of the assessment system and for checking the quality assurance procedures as a whole. Answers to these questions should identify if the quality assurance procedures are adequate and effective by revealing issues or areas for improvement. If no issues or feedback is gathered, there may be problems with the quality assurance process. 74 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 4: Maintain quality assurance procedures Work Projects It is a requirement of this Unit you complete Work Projects as advised by your Trainer. You must submit documentation, suitable evidence or other relevant proof of completion of the project to your Trainer by the agreed date. 4.1 Bring three samples of assessment documents to your trainer that include: The name of the document Version control When document was last reviewed. 4.2. Develop an audit template document for a part of the assessment system at your organisation and complete it. Show the results of the audit to your trainer and discuss the findings. 4.3. Use the results of the audit you conducted of a part of the assessment system and make suggestions for improving the system based upon the findings. Discuss the issues that have been identified with your trainer. 4.4. Look at the results from the audit and decide if the current quality assurance procedures are able to capture the changing nature of the assessment system. Do the current quality assurance procedures check that the assessment system is matching the demands and requirements of external and internal requirements? Discuss your findings with your trainer. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 75 Element 4: Maintain quality assurance procedures Summary Maintain quality assurance procedures Maintaining quality assurance procedures are necessary to ensure that the assessment system is producing the results that it was intended to. It is important to: Define the quality assurance procedures relating to the assessment system Conduct internal audits regularly to check the relevance and effectiveness of the system Use the results of any internal audits to improve the system by implementing responses to any issues that were identified during the audit Review the quality assurance procedures to make sure they are meeting the required needs of the organisation. As any system is constantly evolving, the quality assurance procedures must be able to adapt to these changes. 76 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 5: Report on the contribution made by the assessment system to enterprise training and performance Element 5: Report on the contribution made by the assessment system to enterprise training and performance 5.1 Identify the outcomes that the assessment system has produced Introduction Once the assessment system has been implemented, the next step is to check the outcomes that the system has produced. If one of the aims of the assessment system was to determine gaps in skills of staff, did the outcomes of the assessment show this? What are the gaps in skills? If the assessment system does not provide information or evidence that can help the organisation improve, then the system is not effective. What are the outcomes? There may be many different purposes for the assessment system and the outcomes of the system need to align with the purposes. The assessment system, for example, may have been implemented to identify workplace performance of staff. Interviews may have been conducted, feedback forms and questionnaires completed, and observations carried out. After all of the information has been collected, it must be analysed to determine if the outcomes meet the organisation’s needs. In identifying the outcomes that the assessment system has produced, it is necessary to match the actual outcomes with the intended purposes of the system. It is important to analyse the results or outcomes and identify: Whether the assessment system has been effective in achieving the desired outcomes of the system as a whole The benefits that the assessment system has brought to the organisation by noting the new skills attained, the amount of certificates issued, the increased opportunities and performance that the system has provided to the enterprise Increased professional development opportunities that have occurred due to the implementation of the assessment system Whether there were some areas of strength within the organisation that had not previously been acknowledged Increased staff engagement, productivity or workplace satisfaction. It is important that the outcomes of the assessment system are investigated in an honest and transparent so the evidence can be used constructively for positive change. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 77 Element 5: Report on the contribution made by the assessment system to enterprise training and performance 5.2 Review the assessment system Introduction Once the outcomes that the assessment system has produced are identified, the assessment system needs to be reviewed to determine the overall success or effectiveness of the system. In reviewing for the assessment system there are five targets for validation that can be identified which should be used as a guideline for the review. They include: Assessment systems Assessment processes Assessment methods and tools Assessment evidence Assessment judgements. The review should incorporate three aspects which include: Reviewing Comparing Evaluating. The chart below outlines the assessment review process. Prepare a schedule for the validation of the system Review the assesment ot the unit/item Compare assessments of the unit/items Evaluate the assessment of the unit /item Identify stakeholders for review of assessment documentation Identify assessors and/or evidence gatherers Identify stakeholders for distribution of evalualtion surveys or interviews Design and distributereview questions to stakeholders Set up moderation meeting to compare outcomes Collect and analyse stakeholder responses Analyse results of moderation meeting Prepare list of improvements and required action 78 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Design and put evaluation questions to stakeholders Collect and analyse stakeholder responses Element 5: Report on the contribution made by the assessment system to enterprise training and performance Review questions Industry Suggested questions (before implementing the assessment process as part of a review process): 1 Is the assessment process consistent with industry expectations? 2 Do the selected assessment methods and tools reflect current industry standards and practices? 3 Does the evidence meet the standards you would expect in industry? Suggested evaluation questions: 1 Is the industry comfortable employing participants who have demonstrated achievement through this assessment process? 2 Do you think that this assessment process will preserve or improve workplace performance standards in your industry? Employers Assessors could ask questions like these of employers of participants who are employees, trainees or apprentices or candidates who have been placed through work placement or work experience and are being assessed in the workplace. Suggested review questions: 1 Will the assessment process fit comfortably within your organisation’s activities and workplace culture? 2 Do the selected assessment methods and tools reflect current industry standards and work practices used in your business? Suggested evaluation questions: 1 Has the assessment process had a positive impact on your organisation? 2 Were supervisors able to use the assessment tools effectively? 3 Did the assessment tools provide your organisation with useful information about your employees/trainees/apprentices? 4 Does the evidence meet the standards you would require of your employees? 5 Were the judgements made about the participants’ competencies consistent with your knowledge of these employees/trainees/apprentices? 6 Have the participants assessed as competent continued to perform at the appropriate level? © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 79 Element 5: Report on the contribution made by the assessment system to enterprise training and performance Technical and subject matter experts Assessors could ask questions like these particularly if assessing on a team basis, pairing an assessment expert with a vocational/technical/subject matter expert drawn from industry. Suggested review questions: 1 Do the assessment methods and tools reflect the performance requirements of relevant units of competency? 2 Do the assessment methods and tools reflect current industry standards and workplace practices? Suggested evaluation question 1 What impact do you think the use of these assessment methods and tools will have on the future performance of these students? Participants Assessors may ask questions like these through a forum, questionnaires or interview. These are all suggested evaluation questions: 1 Were you told that you had the right to an RPL assessment before you started this course/training? 2 Were you given clear instructions about what to do in the assessment? 3 Do you feel that you were given sufficient information about what was going to be assessed and how and when you were going to be assessed? 4 Do you think that the assessment process was fair and equitable? 5 Do you think that the person observing your performance was fair and unbiased? 6 Do you think that the overall judgement about your competency performance was accurate and fair? 7 Were you given adequate feedback about your performance after the assessment? 8 Were you told about your right to an assessment appeal? 9 Were you given useful advice about further learning after the assessment? 10 Were you encouraged to monitor your own future progress? 80 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 5: Report on the contribution made by the assessment system to enterprise training and performance Trainers and teachers Trainers and teachers may also be assessors. The following questions could be asked of trainers or teachers who are not directly involved in the assessment process other than through assessments used to monitor participants’ progress. Note: The ‘assessment process’ relates to assessments used to determine competence. Suggested evaluation questions: 1 Was the assessment process compatible with the training delivery program? 2 Did participants have a clear understanding of the relationship between the training they are receiving and the assessment processes? 3 Did participants understand the difference between assessments used to monitor their progress and assessments used to determine their competence? 4 Did feedback from the assessment process help participants to better direct their learning efforts? 5 Was the final judgement of competency performance made through the assessment process consistent with your observations of each participant? Assessors Other assessors from within the organisation or from another organisation will be the most informed observers of assessors’ assessment systems. They can contribute to all aspects of assessors’ continuous improvement processes. It is understood that a natural reluctance to reveal good ideas or weaknesses to a competitor means that this approach has some risk, but the returns can be considerable. It is best if assessors form a working relationship with other assessors in their field or from other industry areas where they can ask questions like those which follow. Assessors may want other lecturers to sign a confidentiality agreement, or a reciprocal arrangement may be established regarding sharing materials to protect the interests of each party. Suggested review questions The first list of suggested questions applies more to planning and development processes; the second to implementation of assessment methods/tools. 1 Is the assessment process consistent with the relevant training package assessment guidelines or accredited course assessment requirements? 2 Do all assessors involved in the process meet the requirements of the training package assessment guidelines or course assessment requirements and internal standards? If not, are they properly managed as required by the assessment guidelines and standards? 3 Is the assessment process compliant with the internal and external standards and agencies? 4 Are the roles of all parties in the assessment process clearly defined? 5 Is the clustering of units of competency used appropriately in the assessment process? © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 81 Element 5: Report on the contribution made by the assessment system to enterprise training and performance 6 Does clustering confuse or enhance the assessment process for all aspects of each unit of competency? Why or why not? 7 Does the assessment plan faithfully reflect all required components of the unit(s) of competency and the critical aspects of evidence? 8 Do the assessment methods/tools (either developed or adopted) faithfully reflect the assessment plan and the requirements of the units of competency? 9 Is any contextualisation of the unit(s) of competency compliant with training package requirements? 10 Do the assessment methods/tools reflect current industry practices and workplace context? 11 Do the assessment methods/tools reflect the level of complexity, responsibility and autonomy indicated by the employability skills and the standards of the unit(s) of competency? 12 Do assessment simulations reflect current workplace practices and environment, and adequately show evidence of the four dimensions of competency? 13 Can reasonable adjustment be made to the assessment tools? 14 Have limits to reasonable adjustment been set, and are these limits consistent with the training package assessment guidelines or accredited course assessment requirements? 15 Does the judgement process reflect the required performance outcomes and critical aspects of evidence of the unit(s) of competency? 16 Is a form of version control in place to ensure that only current versions of assessment method/tools are being used? 17 Do partnership agreements clearly specify the assessment roles and responsibilities of both parties, including quality assurance and record keeping? 18 Are all the necessary assessment resources readily accessible? 19 Have evidence gatherers been appropriately selected, briefed, managed and monitored to ensure sufficient quality evidence? 20 Are evidence gatherers provided with clear instructions? 21 Are students provided with clear instructions? 22 Is the quality of evidence regularly reviewed for credibility and admissibility? 23 Are judgements based upon sufficient evidence (a range of time, context and application)? 24 Are the assessment records adequate to respond to appeals or complaints? 82 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 5: Report on the contribution made by the assessment system to enterprise training and performance The following questions require assessors to use the assessment processes and compare their results – a process usually described as ‘moderation’. 1 Do different assessors develop similar assessment strategies for the same unit(s) of competency? 2 Do different assessors record the same evidence when observing the same participant’s performance? (The tools and assessment process may need to be adjusted if they do not) 3 Do different assessors make the same judgement based upon the same evidence for a unit of competency? Evidence gatherers Evidence gatherers are people who act on behalf of an assessor to apply an assessment method/tool to gather evidence relating to a participant’s performance. Evidence gatherers do not make an assessment judgement; they simply record what they have observed using an observation checklist and a record sheet provided by the assessor. All assessors are evidence gatherers, accumulating evidence until there is sufficient to make a safe judgement, but anyone referred to as an ‘evidence gatherer’ is probably not an assessor. The following evaluation questions are some that might be put to evidence gatherers. Suggested evaluation questions 1 Were your roles and responsibilities as an evidence gatherer clearly specified? 2 Were you provided with clear instructions relating to gathering the evidence and recording the outcomes? 3 Were occupational health and safety (OHS) and other regulatory issues adequately addressed in your instructions on evidence gathering? 4 Were the assessment methods/tools easy to use and relevant to your role as an evidence gatherer? 5 Were limits to reasonable adjustment clearly defined and applicable? 6 Was there any perceived or actual conflict of interest between your role as an evidence gatherer and your relationship with the participant (as a supervisor, client or colleague)? Comparison task 1 Do different evidence gatherers record the same evidence when observing the performance of the same student? Government authorities Government authorities will from time to time audit compliance of the organisation with various standards relating to OHS and the external requirements. For the most part, they will ask the questions, but their findings can contribute to assessors’ continuous improvement process. The sort of question assessors can ask is: 1 What ‘non-compliance’ or ‘opportunities for improvement’ relating to assessment have been identified? © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 83 Element 5: Report on the contribution made by the assessment system to enterprise training and performance 5.3 Make recommendations for change to the assessment system Introduction After completing a review of the assessment system, the outcomes of the review need to be documented and suggestions or recommendations made. These recommendations need to be based upon evidence that is sufficient and real in order for any changes that are implemented to be effective. Making a recommendation to change the assessment system, based upon insufficient evidence, could negatively impact on the system. Recommending change There are many reasons why change may be needed for an assessment system. As has been seen in previous units, gathering evidence relating to the system’s effectiveness is an important part of a continuous improvement cycle. Once sufficient evidence has been gathered, it is necessary to analyse the results and use them to make recommendations for the assessment system process. These suggestions could relate to many different aspects of the assessment system including: Whether to continue with the existing assessment system with minor alterations or to cease with the current system in favour of an external provider administering an assessment system Making recommendations relating to staff resources, training, payment, costs, assessment content or administration procedures Expanding the existing system or reducing the scope of the existing assessment system Recommending changes to location and timing of the assessment system. 84 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 5: Report on the contribution made by the assessment system to enterprise training and performance Below is an example of an assessment system review with recommendations for change. Activity Read through the chart below and make recommendations for the sections which have not been completed. Discuss your responses with your partner. System area Evidence collection Finding Recommendations Assessment result recording 10 participant result sheets All samples collected were completed in full with no information missing. All participant results were current and up to date Continue with current practice Several assessors’ had not completed the OH&S certification due to lack of time Assessors to complete OH&S training as soon as possible and regular checks implemented to monitor assessor skills Assessor summary of results sheets Interviews with participants and assessors Assessor qualifications Interviews with assessors Qualification checks of all assessors Interviews with participants and managers Assessment validation Validation documents observed Interviews with assessors, evidence gatherers and managers Interviews with participants Assessment staff resources Assessor scheduling documents obtained Participant numbers and assessment delivery schedule checked Administration staff scheduling documents obtained Validation documents completed and up to date Two assessment areas not documented: assessment procedures and assessment moderation Several assessors scheduled to conduct assessment on certain days but no candidates available Assessment details not communicated to participants in a timely manner © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 85 Element 5: Report on the contribution made by the assessment system to enterprise training and performance System area Evidence collection Finding Assessment methods Observation of assessment delivery Assessors used only two varieties of assessment methods Assessor interviews Participant interviews Assessment tools Assessor interviews Assessment tools examined A reliance on summative testing was observed with limited scope of different assessment tools Assessors used paper based tools when online assessment tools are available and computer labs available 86 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Recommendations Element 5: Report on the contribution made by the assessment system to enterprise training and performance 5.4 Produce and circulate a report Introduction The next stage in managing an assessment system is to develop a report and distribute the report to all relevant stakeholders. The report should outline all the necessary information about the assessment system, the results of the review, any recommendations for change or updating of the system, and any future plans for further reviews or checks. The report then needs to be circulated to all of the people that can are connected to the assessment system so that they can provide any other feedback that may be relevant or just to be aware of the results and any changes that may be implemented due to the review process. What should be included in a report? When writing a report it is important to identify the intended audience or readers. What do they need to know about the review of the assessment system? There are many different ways to write a report but there are certain parts or sections that should be included in any report. A suggested format for a report may include: Section 1: Cover Title page Executive Summary Table of contents List of figures. Section 2: Body of the report Introduction Findings and discussions Conclusions Recommendations. Section 3: Appendices References Glossary (If required). © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 87 Element 5: Report on the contribution made by the assessment system to enterprise training and performance Title page A title page should include: The full title of the report which is clear and informative Who the report was developed for Who developed or produced the report The date A footer identifying if it is ‘strictly confidential’ or for ‘public release’ No page number. Table of contents A table of contents allows readers to locate necessary information easily by providing an outline or overview of the report as a whole. All of the main sections and subsections should be included and they should be written the same as in the main body of the report. Executive Summary An executive summary is usually a one page summary of the entire report. It should be no longer than 5% - 10% of the report length. It outlines why the report was developed, what the report is about, how it was developed, the most significant findings or results of the review, and any analysis, conclusions or recommendations. Many stakeholders will only read this section so it is extremely important. Main body In this section an introduction is provided which details any relevant background information, describes the objectives of the report, details the structure of the report and how the review was conducted, and outlines any constraints, assumptions or limitations on the scope of the review or report. Findings or results In this section the main findings are outlined with further discussion and analysis of the results. It is important to provide an analysis of the findings and use the evidence from the review to support the findings. The findings section can be presented in text and/or by using charts, graphs and figures. Conclusion The conclusion summarises all aspects of the report and provides an opportunity to present an overall opinion or reflection from the report. New information should not be included in the conclusion. Recommendations This section is one of the most important parts of the report as it takes the findings and uses the evidence to recommend changes that will improve the assessment system. The recommendations should outline what needs to changed or implemented, who should be responsible for the changes and when they should be completed by. 88 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 5: Report on the contribution made by the assessment system to enterprise training and performance Activity Your trainer will give you some examples of reports. Read through them in small groups and look at all the sections. Are the reports clear? Is there a table of contents, executive summary or recommendations section? Conduct a survey of your class. The survey could be about favourite food, sports, holidays or it could determine the qualifications of the members of your class. Once you have collected the data, create a chart or graph that captures the information, and then write a brief outline of the findings. Circulate the report Once you have completed the report it is necessary to circulate the report to all the relevant stakeholders. It is important to identify all of the stakeholders they may need to receive the report and these could be: Assessors Managers External agencies Training providers and trainers Candidates or staff Administrators Assessment coordinators. The report may be paper-based or it may be distributed electronically. You may use a combination of both paper-based and electronic distribution via email. There should be a mechanism that allows for feedback and you might organise a presentation of the report with all key stakeholders attending. The presentation should allow participants to respond to the findings and then this feedback can form the basis for a further review of the assessment system. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 89 Element 5: Report on the contribution made by the assessment system to enterprise training and performance 5.5 Revise the existing assessment system on the basis of responses to the report Introduction After completing a report and asking for feedback based upon the report findings and recommendations, it may be necessary to change or update some parts of the assessment system. Once the changes have been implemented the quality assurance or continuous improvement cycle begins again as the new changes will have to be monitored to make sure that they are appropriate and beneficial or relevant. What will be revised? As was presented in previous elements, there are five main areas of the assessment system where improvements or revision may be implemented. They include: Assessment systems Assessment processes Assessment methods and tools Assessment evidence Assessment judgments. After recommendations have been made and stakeholders have added their comments and feedback, decisions may be made on what changes or improvements to the assessment system are necessary. These may include: Revising policies and procedures relating to the assessment system Altering the assessment tools to make them more relevant and appropriate Revising the assessors that have been used which may mean increasing or decreasing numbers, choosing assessors with different skill sets and experience, or providing more training for current assessors Revising the assessment times and locations to suit organisational needs Communicating any changes that have been made to the assessment system and making sure that all relevant stakeholders are informed of these changes or updates. 90 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Element 5: Report on the contribution made by the assessment system to enterprise training and performance Work Projects It is a requirement of this Unit you complete Work Projects as advised by your Trainer. You must submit documentation, suitable evidence or other relevant proof of completion of the project to your Trainer by the agreed date. 5.1 Analyse the results of the assessment system at your organisation and develop a short report that outlines the outcomes that the assessment system has achieved. Show the report to your trainer and discuss outcomes. 5.2. Choose one aspect of the assessment system and conduct a review of it. Outline: What was reviewed How the review was conducted The results of the review. 5.3. Based upon the results from the review you conducted, make a list of recommendations for improving the assessment system. Show the list to your trainer. 5.4. Write a short report that discusses the review of the assessment system that you conducted. Make sure that the report follows the guidelines from the trainee manual. Provide the report to your trainer. 5.5 After discussing the assessment review report with your trainer, describe the changes that you would make to the assessment system to improve it. These revises should be based upon feedback from the review. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 91 Element 5: Report on the contribution made by the assessment system to enterprise training and performance Summary Report on the contribution made by the assessment system to enterprise training and performance After conducting relevant reviews and checks relating to all aspects of the assessment system, it is critical to determine if the system is meeting the original goals or purposes that it was designed for. In examining the contribution made by the assessment system, it is important to: Identify the outcomes that the assessment system has achieved Establish mechanisms to review the system Make recommendations bas upon the results of reviews Document the review process and outcomes, and distribute to all relevant stakeholders to provide opportunities for feedback Revise the assessment system based upon the outcomes of the review and feedback received. 92 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Presentation of written work Presentation of written work 1. Introduction It is important for students to present carefully prepared written work. Written presentation in industry must be professional in appearance and accurate in content. If students develop good writing skills whilst studying, they are able to easily transfer those skills to the workplace. 2. Style Students should write in a style that is simple and concise. Short sentences and paragraphs are easier to read and understand. It helps to write a plan and at least one draft of the written work so that the final product will be well organised. The points presented will then follow a logical sequence and be relevant. Students should frequently refer to the question asked, to keep ‘on track’. Teachers recognise and are critical of work that does not answer the question, or is ‘padded’ with irrelevant material. In summary, remember to: Plan ahead Be clear and concise Answer the question Proofread the final draft. 3. Presenting Written Work Types of written work Students may be asked to write: Short and long reports Essays Records of interviews Questionnaires Business letters Resumes. Format All written work should be presented on A4 paper, single-sided with a left-hand margin. If work is word-processed, one-and-a-half or double spacing should be used. Handwritten work must be legible and should also be well spaced to allow for ease of reading. New paragraphs should not be indented but should be separated by a space. Pages must be numbered. If headings are also to be numbered, students should use a logical and sequential system of numbering. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 93 Presentation of written work Cover Sheet All written work should be submitted with a cover sheet stapled to the front that contains: The student’s name and student number The name of the class/unit The due date of the work The title of the work The teacher’s name A signed declaration that the work does not involve plagiarism. Keeping a Copy Students must keep a copy of the written work in case it is lost. This rarely happens but it can be disastrous if a copy has not been kept. Inclusive language This means language that includes every section of the population. For instance, if a student were to write ‘A nurse is responsible for the patients in her care at all times’ it would be implying that all nurses are female and would be excluding male nurses. Examples of appropriate language are shown on the right: Mankind Humankind Barman/maid Bar attendant Host/hostess Host Waiter/waitress Waiter or waiting staff 94 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Recommended reading Recommended reading Making your ideas stick, reporting and program planning Atkinson C. Beyond Bullet Points: Using Microsoft Office PowerPoint 2007 to Create Presentations That Inform, Motivate, and Inspire. Microsoft Press, 2007 Becker HS. Writing for Social Scientist: How to Start and Finish Your Thesis, Book, or Article. Chicago, IL: University of Chicago Press, 2077, 2nd ed Heath C, Heath, D. Made to Stick: Why some Ideas Survive and Others Die, New York, NY: Random House, 2007 Heath C, Heath D. Switch: How to Change Things When Change is Hard. New York, NY: Random House, 2010 Lavinghouze R, Price AW, Smith, KA. The Program Success Story: A Valuable Tool for Program Evaluation. Health Promotion Practice, 2007; 8(4): 323–331 Torres R, Preskill H, Piontek ME. Evaluation Strategies for Communicating and Reporting. Thousand Oaks, CA: Sage Publications, Inc., 2004, 2nd ed Qualitative methods Miles MB, Huberman, MA. Qualitative Data Analysis. Thousand Oaks, CA: Sage Publications, Inc., 1994, 2nd ed Patton M.Q. Qualitative Research and Evaluation Methods. Thousand Oaks, CA: Sage Publications, Inc., 2001, 3rd ed Yin RK. Case Study Research: Design and Methods (Applied Social Research Methods) Thousand Oaks, CA: Sage Publications, Inc., 2008, 4th ed Yin RK. Qualitative Research from Start to Finish. New York, NY: The Guilford Press, 2010 Quantitative methods Kleinbaum DG, Klein M. Logistic Regression: A Self-Learning Text (Statistics for Biology and Health. Springer, New York, NY, 2010, 3rd ed Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. Philadelphia, PA: Lippincott Williams & Wilkins, 2008, 3rd ed Tufte ER. Envisioning Information. Cheshire, CT: Graphics Press, 1990 Tufte ER. The Visual Display of quantitative Information. Cheshire, CT: Graphics Press, 2001 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 95 Recommended reading Evaluation use Butterfoss FD. Coalitions and Partnerships in Community Health. San Francisco, CA: Jossey-Bass, 2007 Mattessich PW. The Manager’s Guide to Program Evaluation: Planning, Contracting, and Managing for Useful Results. St. Paul, Minnesota: Amherst H. Wilder Foundation, 2003 Patton MQ. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York, NY: The Guilford Press, 2010 Patton MQ. Utilisation-Focused Evaluation. Thousand Oaks, CA: Sage Publication, 2008 Useful websites: Assessment systems: http://equella.think.edu.au/lor/file/c5c75dd7-cfb6-63ba-de0274a178fbb10b/7/AssessmentPolicyAndProcedure.pdf Evaluation ideas: http://www.eval.org Impact and Value: Telling Your Program’s Story: www.cdc.gov/oralhealth/publications/library/success_stories_wkbk.htm 96 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes Trainee evaluation sheet Trainee evaluation sheet Manage an assessment system for training outcomes The following statements are about the competency you have just completed. Please tick the appropriate box Agree Don’t Know Do Not Agree Does Not Apply There was too much in this competency to cover without rushing. Most of the competency seemed relevant to me. The competency was at the right level for me. I got enough help from my trainer. The amount of activities was sufficient. The competency allowed me to use my own initiative. My training was well-organised. My trainer had time to answer my questions. I understood how I was going to be assessed. I was given enough time to practice. My trainer feedback was useful. Enough equipment was available and it worked well. The activities were too hard for me. © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes 97 Trainee evaluation sheet The best things about this unit were: ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ The worst things about this unit were: ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ The things you should change in this unit are: ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ 98 © ASEAN 2013 Trainee Manual Manage an assessment system for training outcomes