UQ-VN PDSS ASSURANCE OF ASSESSMENT QUALITY IN HIGHER EDUCATION UQ Vietnam Professional Development Seminar Series 4: April 2012 Ho Chi Minh City: Hanoi: Hue: Danang Dr Clair Hughes clair.hughes@uq.edu.au Senior Lecturer in Higher Education The Teaching and Educational Development Institute (TEDI) Session overview This session will address: 1. 2. 3. 4. 5. 6. The influence of the international standards agenda on quality assurance practices The importance of assessment in providing credible assurance of the quality of student learning Approaches to the assurance of the quality of assessment tasks Characteristics of assessment quality Approaches to the assurance of assessment judgements Implications for assessment quality assurance responsibilities Definitions • Program - a period of study of several years duration leading to the award of a degree • Course - a period of study usually of a semester’s duration – a component of a program • Assessment - the process of forming a judgment about the quality and extent of student achievement or performance (Sadler 2005, p. 177). • Evaluation - the process of forming a judgement about the quality of (in this context) an educational experience or one or more of its components • Criterion – a distinguishing property or characteristic of anything, by which its quality can be judged or estimated. (Sadler 2005, p. 178) • Standard - a definite level of excellence or attainment.(Sadler 2005, p. 189) The assessment task • Your colleague Ms Hoa is new to your department and has asked for your advice on an assessment task she has developed for a new course in English language. The main learning objectives are for students to: – improve their English vocabulary and pronunciation – develop awareness of the type of information discussed in a job interview – present their skills and knowledge well to a future employer. CRICOS Provider No 00025B Activity 1: • Ms Hoa has not taught this type of course before and so she asks your advice on this task: • Her ideas include asking students to: A. think of 3 likely questions they will be asked in an interview and prepare written answers to these questions B. role-play a job interview with a partner using questions provided by the teacher C. write an essay on how to behave in a job interview D. prepare a CV and a letter of application E. watch a video of a job interview and write about what went well and what could have been improved • What feedback would you give her on these suggestions? Why? CRICOS Provider No 00025B Focus of assessment quality assurance (1) Task quality Purpose Does the task allow students to demonstrate intended learning outcomes – it is aligned? Type of text Does the task require the students to produce the type of work common to your discipline? PURPOSE TEXT TYPE Roles and relationships Does the task give students a realistic role or audience – is it authentic? Roles and Relationships Subject Matter Assessment as TEXT production Subject matter or topic Does the task address significant subject matter? Mode & Medium Mode Does the task require students to use forms of communication used in your discipline – spoken, written, visual? Medium Does the task require students to use the media used in your discipline – face-to-face, webpages, posters, journals etc? Hughes 2009 Focus of assessment quality assurance Task quality- Example from Engineering (Project report) Purpose Task requires students to address learning outcomes related to teamwork, engineering construction and project management PURPOSE TEXT TYPE Roles and relationships Report requires students to take the role of a professional engineer working with a client Mode The report contains written and visual material and is also presented orally Roles and Relationships Subject Matter Assessment as TEXT production Subject matter or topic Project requires students to use wide range of engineering skills and reflect on teamwork skills Type of text The report is to be of the same quality as those produced by professional engineers Mode & Medium Medium An electronically produced report is common in the engineering industry Hughes 2009 Activity 3: Task analysis Think of an assessment task with which you are familiar and analyse it as for the example on the previous slide Purpose Type of text PURPOSE TEXT TYPE Roles and relationships Mode Roles and Relationships Subject Matter Assessment as TEXT production Subject matter or topic Mode & Medium Medium Hughes 2009 Focus of assessment quality assurance (2) • Also consider task in relation to: • • • • • • • • CRICOS Provider No 00025B compliance with assessment policy and rules clarity of assessment description other tasks in this course (how they fit together) assessment tasks in other courses studied at the same time (spread, timing, workload) assessment tasks across the program (how they allow for progression in difficulty and skill development) accreditation requirements of professional bodies provision of feedback its contribution to overall grade Activity 2 • After seeking advice from a few people Ms Hoa decides on a 2-part assessment task: 1. Students develop a CV and letter of application that addresses a particular job they have found in a newspaper 2. Students role play a job interview with one student in the role of the interviewer and one in the role of the applicant • • She is anxious not to make any mistakes and asks you for further advice: Select two of the following as being important things for her to do: A. Read the university assessment policy and guidelines B. Talk to the program coordinator C. Not to do anything as the task is already quite good D. Look at the assessment for other courses in this department E. Look at student survey ratings for other courses in this department F. Read some articles on assessment in educational journals G. Other CRICOS Provider No 00025B Approval • The university has an approval process in place for new courses. Ms Hoa gains approval for the task from: – the program coordinator – The university Course and Program Approval Committees CRICOS Provider No 00025B Activity 3 • Ms Hoa wants to collect some evidence of her assessment quality so you advise her to (select two of the following): A. B. C. D. E. F. Make notes while students are doing the role-play Nothing as it is more important to spend time on her research Look at the assessment items on the student survey ratings Ask some students for their opinion in a group interview Ask the program coordinator to sit in on some of the role plays Ask the program coordinator to mark some of the CVs and letters of application G. Analyse what students did well or poorly on their written CVs and letters of application H. Other? CRICOS Provider No 00025B Activity 4 • As there were a lot of students in the class, Ms Hoa had the help of 2 tutors in marking the CVs, letters of application and role plays. Some students complained that one of the tutors gave much lower marks than Ms Hoa and the other tutor. What advice would you offer her so that this does not happen next time the course is offered? Select two of the following. A. B. C. D. E. F. G. H. Keep some of the CVs and letters of application (with student permission) from the first course and use them to train the two tutors before they mark student work Develop a detailed marking guide and give it to the tutors Collect a random sample of marked work to check the marks of both tutors Video all student role-plays and look at a random sample of them to check the marks Sit in on the first couple of role-plays and discuss tutor marks before they go ahead with the rest of the marking Mark a couple of CVs and letters together before doing the rest of the marking Meet with the tutors to discuss examples of high and low marks and any that were difficult to mark Other CRICOS Provider No 00025B Activity 5 Global activity in Quality Assurance reflects a shift of emphasis from teaching to learning • Tuning in Europe, South America • Tuning USA – Lumina Foundation • Learning and Teaching Academic Standards project of the Australian Learning and Teaching Council (now OLT) • As a result, the Ministry of Education is asking universities to collect evidence that they are encouraging teachers to engage in the assurance of assessment quality. Your University conducts a survey and Ms Hoa is one of a group of randomly selected teachers who is asked to write a short report on what she has done to assure assessment quality in her course. List all the activities she could include in her report. CRICOS Provider No 00025B SUMMARY CRICOS Provider No 00025B The international standards agenda Assuring the quality of student learning outcomes through: • articulation of minimum or threshold standards • standardised testing (AHELO, CLA) • external examining and moderation - comparison of standards within and between institutions • development or revision of student satisfaction surveys Funded projects to identify and support effective practice and explore emerging issues http://www.itl.usyd.edu.au/projects/aaglo/ Assessment – a key curriculum component Learning Objectives What do we want students to learn? Assessment How will students demonstrate their learning? Credible evidence of student learning outcomes Learning activities What will students do to develop their learning? Teaching activities How will teachers facilitate student learning? Assessment quality is the responsibility of..... Senior management • VC • DVC (Academic) • President of academic Board • Chair of Teaching and Learning Committee • Chair of Teaching and Learning Sub-Committee • Chair of Program Approval Committee • Chairs of relevant Working Parties (e.g. policy on Assessment, Academic Integrity, Curriculum and Teaching Quality Assurance, Student Charter) CRICOS Provider No 00025B Faculty and school leaders • Faculty Dean • Heads of School • Assistant Deans (Academic) • Chief Examiners • Chair of Faculty/School Teaching and Learning Committee • Chair of faculty or school program approval committees Program and course staff • Program coordinators • Course coordinators • Teaching staff (including casuals) • Tutors • Administration staff Other • Members of working parties • Academic Development Unit staff Different roles – different QA responsibilities External, government agencies Monitor institutional processes and outcomes Professional bodies and employers Institutional leaders Support and assure industry standards Establish policies a supporting infrastructure Faculty leaders Support &monitor institutional policies Program &course coordinators Comply with assessment and QA policies Teachers &admini. staff Students CRICOS Provider No 00025B Deliver program and course assessment Undertake assessment with integrity Quality assurance of assessment Policy and infrastructure Create appropriate roles with descriptions that refer to responsibility for assessment quality Develop and cross-reference policy, guidelines and rules for: • Assessment •Quality assurance •Academic Integrity •Student Charter •Appointment and promotion •Academic roles Create infrastructure required for effective implementation , e.g. •Data collection strategies •Support technology •Professional development •Staffing CRICOS Provider No 00025B Approval Take a whole of program approach to assessment planning (e.g. map learning objectives and/or assessment across program) Put approval processes in place for new or revised assessment Make the approval focus appropriate for level e.g. •Course – detail of course quality •Institutional – policy compliance Make approval consequential (i.e. a response is required when proposed assessment needs improvement) Review/evaluation Put in place regular and systematic review or evaluation Collect data on assessment quality (e.g. student satisfaction, student outcomes, grade distributions, benchmarking with comparable programs) and make it available Make evaluation formal and consequential (i.e. systematic processes are guided by checklists or templates, it is clear who is responsible, findings must be addressed and recommendations and responses are reported) Streamline institutional QA and accreditation processes – no gaps, no duplications Assurance of the quality of judgements Calibration – activities that develop shared understanding of standards before judgements are made (Consensus) moderation – activities to compare, confirm, defend or adjust judgements and standards • Develop activities to induct new academics – e.g. provide workshops to make and discuss judgements on samples of student work from previous years (video used for performance or presentation) • Hold regular meetings of tutors to discuss expectations and agree on advice provided to students • Have all assessors mark some samples of student work and then agree on standards before marking the rest • Coordinators can check random samples of all assessors judgements and negotiate necessary adjustments • Coordinators can attend oral presentations or performance and compare their judgements with those made by tutors • Student work can be double-marked and then discussed • Chief examiners or school committees can monitor grade distributions across courses and ask academics for an explanation where necessary • Colleagues from other institutions can review and comment on standards CRICOS Provider No 00025B External assurance of quality of student learning outcomes • Development of discipline-specific graduate learning outcomes (QAA in the UK: Tuning in Europe: ALTC in Australia) • Quality audits (AUQA and now TEQSA in Australia: QAA in the UK) • Standardised testing – Generic (AHELO: CLA) – Discipline specific (AHELO Engineering and Economics: Under consideration by national discipline communities – chemistry, mathematics, medicine) • External examining or review (Well established external examining system in UK: QVS and other funded projects trialled in Australia) • Accreditation panels of professional associations • National student surveys CRICOS Provider No 00025B Emerging issues • Variable or ‘hidden’ standards • Modular or otherwise fragmented approaches to curriculum and assessment • Increasing class sizes, student diversity and academic workloads • Inconsistent or limiting policies • Debatable validity of ‘generic’ approaches to assessment – strong arguments for discipline-specific approaches and instruments for standardised testing • Other criticisms of generic, standard instruments – e.g. Does CLA measure aptitude or learning gains? • Doubtful value of external examining processes based on partial information • Emphasis on QA rather than QI CRICOS Provider No 00025B Effective practice • Characteristics of effective quality assurance (QA) and improvement (QI) of assessment quality processes – Clear standards – process and learning outcomes – Systematic and formal processes coordinated across all administrative levels and policies, integrated with professional development and appropriately resources – Whole of program approaches – avoidance of fragmentation – Widespread participation – assessment quality the responsibility of all – Balance of efficiency (e.g. automation) with meaningful involvement – avoidance of perfunctory or ritualistic compliance – Consequential – assessment strengths are recognised and rewarded and weaknesses are addressed – Appropriate use and interpretation of standardised processes and instruments – Collaboration and peer review – both internal and external – as encouragement of interaction and dialogue that support QA and QI CRICOS Provider No 00025B Activity 6: Ongoing quality improvement • Thinking about your role in the university: – – – – What have you learned? What workshop ideas are most relevant to your role? How has the workshop affirmed what you are already doing? What are the implications for your future practice? CRICOS Provider No 00025B References • • • • • • • • • • AAGLO Project. (2011). AAGLO Summary 2: Assurance of graduate learning outcomes through external review. Retrieved from http://www.itl.usyd.edu.au/projects/aaglo/pdf/AAGLO_Summary%202_Final.pdf. Barr, R. B., & Tagg, J. (2004). From Teaching to Learning - A New Paradigm for Undergraduate Education. Retrieved from http://ilte.ius.edu/pdf/BarrTagg.pdf Hughes, C. (2009). Assessment as text production: drawing on systemic functional linguistics to frame the design and analysis of assessment tasks. Assessment and Evaluation in Higher Education. 34(5), 553-563. Krause, K., Scott, G., & Colleagues. (2011). Learning and Teaching Standards Project: Inter-university peer review and moderation of coursework and assessment. Retrieved from http://www.uws.edu.au/LATS Lumina Foundation. (n.d.). Tuning USA. Retrieved from http://www.luminafoundation.org/?s=tuning OECD. (2011). Assessment of Higher Education Learning Outcomes. AHELO Project Update - May 2011. Retrieved from http://www.oecd.org/dataoecd/8/26/48088270.pdf The Quality Assurance Agency for Higher Education. (2012). UK Quality Code for Higher Education Part B: Assuring and enhancing academic quality. Retrieved from http://www.qaa.ac.uk/AssuringStandardsAndQuality/qualitycode/Pages/Quality-Code-Part-B.aspx Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education, 30(2), 175–194 Tertiary Education Quality and Standards Agency. (2011). Developing a Framework for Teaching and Learning Standards in Australian Higher Education and the Role of TEQSA: Discussion Paper. Retrieved from http://www.deewr.gov.au/HigherEducation/Policy/teqsa/Documents/Teaching_Learning_Discussion_Paper.pdf. Tuning Educational Structures in Europe. (n.d.). Tuning Educational Structures in Europe: What is Tuning? Retrieved from http://www.unideusto.org/tuningeu/