FINAL Project Title: A sector-wide model for assuring final year subject and program achievement standards through inter-university moderation 1. Outcomes This project will yield a validated, robust approach for assuring subject achievement standards through inter-university moderation in common final year subjects across disciplines. It will also trial approaches for moderating and assuring program achievement standards, building on ALTC discipline standards. Project resources will assist universities to implement sustainable, self-regulatory moderation processes for monitoring subject and program standards. 2. Abstract This proof-of-concept project will produce resources to guide inter-institutional moderation for assuring final year subject and program achievement standards. Eight universities will identify common final year subjects across eight disciplines aligned with the ALTC discipline standards project. Subject convenors will share subject outlines and selected assessment artefacts for review by at least two other project universities. The moderation process includes inputs (e.g., subject outlines, assessment tasks, marking criteria) and outcomes (i.e., assessment samples). External blind peer review of both inputs and outcomes will determine the consistency of subject-level standards and how these compare with comparable final year subjects in other universities. Where relevant, capstone subjects will be used and program learning outcomes considered to identify approaches for assuring program achievement standards through inter-university moderation. The project addresses the TEQSA imperative to demonstrate sector-level, self-regulated, robust approaches for assuring quality and standards and highlights the role of peer review. Guidelines for practice will be sustainable and owned by academic disciplinary communities. External moderation and coursework 1 assessment Project Title: A sector-wide model for assuring final year subject and program achievement standards through inter-university moderation Project Team: Prof Kerri-Lee Krause1 and Prof Geoff Scott2 (co-leaders) Prof Stuart Campbell2, A/Prof Martin Carroll3, Professor Liz Deane4, Dr Duncan Nulty1, Prof Pip Pattison5, Prof Belinda Probert6, Prof Judyth Sachs7, Prof Stephen Towers8 1Griffith University 2University of Western Sydney 3Charles Darwin University 4Australian National University 5The University of Melbourne 6La Trobe University 7Macquarie University University of Technology 8Queensland 1. Project Rationale and Outcomes 1.1 Context This project is proposed at a time of national and international focus on higher education assessment outcomes, quality and standards, with an emphasis on universities’ capacity to demonstrate how they set and meet standards across discipline areas. A number of factors are acting in combination to account for the significant increase in focus on this area over the last few years: • the Australian Universities Quality Agency (AUQA, 2009) expert working party, discussion paper and sector-wide consultation on setting and monitoring academic standards in Australian higher education; • introduction of the Tertiary Education Quality and Standards Agency (TEQSA) and an increased policy focus on standards, including the development of a revised regulatory standards framework for higher education; • introduction of the Collegiate Learning Assessment as a new national indicator to gauge the value added by universities to student learning outcomes and skill development; • internationally, the OECD’s (2009) International Assessment of Higher Education Learning Outcomes (AHELO) project is examining the feasibility of assessing and comparing graduate learning, including the value added during higher education, across nations, systems, languages and cultures; • evidence of a significant growth of interest in proactive action with regard to measuring and evidencing student outcomes by Vice-Chancellors and higher education networks like the ATN and Go8; • a growing number of cases of litigation concerning the quality of assessment in universities, both within and beyond Australia; • emphasis on the portability and equivalence of qualifications within and beyond Australia, including revisions to the Australian Qualifications Framework (AQF); • international competition in the higher education market as a consequence of the Bologna process. The imperative now is for every university to focus on both quality assurance and quality enhancement in assessment and standards. The focus needs to be on assuring the quality and standards of not only the inputs but also the outcomes of assessment in ways that are relevant to specific disciplines and professions. Importantly, the sector also needs to demonstrate ways in which it is supporting diversity and responsiveness, along with peer review and local ownership in self-regulation of standards and quality. 1.2 The proposed project and outcomes The project will develop and validate a proof-of-concept inter-institutional model for external validation of assessment processes and outcomes at subject level, while taking into account whole-of-degree program outcomes. It will build on a self-initiated pilot undertaken during 2010, involving three final year subjects and discipline areas across six of the universities in the project team. The project will result in a scaleable model and resources to enable self-regulated external moderation and benchmarking of final year subject-level outcomes across the sector and across disciplines. Mindful of the importance of a whole-of-program 1 Definition of terms: i) coursework: curriculum covered at unit/subject and course/program level. ii) program/course: whole-of-degree program. iii) subject/unit: an individual unit taken as part of a whole-of-degree program. iv) capstone subject: a final year culminating subject taken at the end of the program in which students showcase capabilities and competencies developed during the program. 1 perspective on standards and outcomes, the project will also pilot approaches for moderating program-level achievement standards, based on work with program leaders responsible for capstone courses in selected universities. The challenges of such a process are fully acknowledged (Brown, 2010), nevertheless there is an urgent need to pursue a range of robust approaches to demonstrate and further enhance how standards are being assured at both subject and program level. The proposed project will facilitate external validation across selected common final year subjects in the 8 Fields of Education (FOE) used in the ALTC Discipline Standards project. The ALTC Discipline Standards project focuses on the threshold learning outcomes for different FOEs. The proposed project complements this approach by focussing on evidence of ‘how’ these standards are evidenced through coursework and assessment artefacts at the subject level. While inter-institutional subject level moderation is the primary focus, the project team will have the opportunity to conduct a stocktake of capstone subjects as a basis for investigating program-level outcomes and standards. It is expected that the project outcomes will include recommendations for a comprehensive program-level moderation project, based on the pilot activities and the capstone stocktake results of the proposed project. The subject level moderation process will involve peer review of subject inputs, along with blind peer marking of assessment outcomes. The focus on final year subject outcomes is key to addressing the policy imperative to demonstrate exit standards at the discipline level. Rather than relying on a generic graduate skills assessment test, this project focuses on evidence of actual student achievement in disciplinary contexts as they prepare to conclude their undergraduate study. In each of the 8 FOEs, final year subjects that are common to at least three universities in the project team will be selected, with a view to reviewing outcome standards as well as the quality of inputs including: • the demonstrable relevance of the capabilities and competencies being assessed; • how these capabilities and competencies are being assessed; • how marks and grades are being determined; and • how consistency in assessment inputs for the same subject is being achieved between institutions. Based on an evaluation of the 2010 self-initiated moderation pilot, the project team is confident that the proposed project will enhance the quality of learning, teaching and assessment in the sector in several ways, including: • producing a validated proof-of-concept inter-institutional subject-level moderation model that is sustainable and scaleable for use across the sector and across disciplines; • an evaluation of relative benefits of consensus moderation and the G08 Quality Verification System for assuring standards, based on feedback from the two Go8 university representatives; • a stocktake of capstone subjects (in collaboration with existing ALTC capstone projects) with a view to recommending approaches for addressing program-level outcomes and standards; • networks of institution- and discipline-based academic staff networks to share good practice in assessment and moderation across disciplines; • capacity-building among subject and program leaders who develop insights into the assessment and grading practices of colleagues teaching comparable final year subjects, including sharing of subject outlines and grading guides; • practical, on-the-job experience for academic staff in standards-based assessment and interinstitutional moderation practices; • development of a common language around issues of moderation and standards; • resources and capacity-building opportunities for academic staff and senior university leaders in relation to quality and standards in anticipation of TEQSA requirements for self-regulated accountability and quality assurance mechanisms that are transparent, rigorous and sustainable. Information on how these outcomes will be disseminated is included in 2.5 below. 1.3 Rationale for the project The rationale for this project is threefold. a. It addresses ethical and educational imperatives to ensure that Australian universities have demonstrably effective processes in place to assure the quality of assessment processes and outcomes for students 2 across the sector. b. It supports social and economic imperatives associated with demonstrating to key stakeholders, such as industry and community members, that universities have robust approaches for assuring assessment quality and standards that are aligned to achievement of whole-of-program graduate capabilities and learning outcomes. c. It takes account of the higher education policy and regulatory environment which requires the sector to make explicit its processes for assuring quality in assessment processes and outcomes through a combination of institution-level systems and policies, as well as inter-institutional benchmarking and assessment validation practices. Explication of this kind adds value to the Australian sector, both nationally and internationally, as student mobility increases and as credit transfer arrangements in the context of processes such as Bologna become increasingly important. 2. Value/Need for the Project The need for this project is argued on several grounds which include a range of value add dimensions. i. In the context of a new national regulatory framework it is important for the sector to show that it is willing to address assessment quality and standards issue by: a. expanding the notion of standards to include inputs (e.g., quality of assessment tasks and grading procedures) and students’ academic achievement outcomes; b. developing, testing and evaluating a model of external validation; c. adopting a rigorous, validated and feasible approach to self-regulated, transparent strategies for assuring standards and the consistent quality of assessment. ii. The model that will result will provide Australia’s university sector with a viable alternative model for assure standards whilst maintaining system flexibility, responsiveness and diversity. iii. The project team comprises experts who are actively involved in sector-wide projects and policy discussions regarding quality and standards in higher education. Their combined expertise will inform the development of an integrating framework that will guide the project and within which to locate the range of sector-wide quality and standards-related projects underway. iv. The project outcomes will add value to the sector by producing a model for constructive steered engagement with an enhancement focus in relation to assuring quality and standards in the sector. v. The focus on the nature and validity of subject-level assessment standards and outcomes in the context of program-level outcomes is a further fundamentally important dimension. 3. Project Approach 3.1 Theoretical framework Assessment and grading: a social, discipline-based practice Assessment and grading are acknowledged as social practices (Yorke, 2008) into which ‘new colleagues are inducted’ (p.6). The epistemological characteristics of the discipline shape assessment practices (Kreber, 2009; Trowler, 2009), hence the importance of ensuring disciplinary representation in this project. It will be particularly important to ensure that the guidelines and resources emerging from the project are sufficiently flexible to accommodate disciplinary differences. Yorke (2008) found that a majority of academic staff developed their understanding of marking students’ work from talking with and observing colleagues, while some said they adopted assessment practices based on their experience as students. Academics undertake relatively little developmental work in the area of assessment (Rosovsky & Hartley, 2002, p.14), yet there are significant technical considerations and skills underpinning effective assessment practices. For instance, the principles of validity (Messick, 1989) and reliability (Feldt & Brennan, 1989) are core and they underpin the theoretical framework for this project. Reliability, in the sense of replicability of performance, is particularly problematic for universities typically do not have the resources to conduct assessment on a scale sufficient to ensure reliability (Yorke, 2009). Yorke (2008, p.24) acknowledges that grading of student work is subject to considerable variability that may result from the assessor’s interpretation of the assigned task and grading criteria. Knight (2006) also observes that grading is a ‘local’ process which means that grading in one institution may not be replicated in another. Such variability poses considerable risk to the sector for, as Smith (1992) argues, for grades to be considered reliable, they should be reasonably stable across courses in the disciplines concerned. Inter-institutional moderation is recognized as a valuable strategy for addressing this 3 challenge (Brown & Knight, 1994). Assuring standards in disciplinary communities: consensus moderation and peer review Moderation is a peer review process for developing consistency or comparability of assessment judgements. Sadler (2010) argues for the value of consensus moderation for assuring grading standards and academic achievement standards to attain quality assurance of the summative grading process. Likewise Boud (ALTC Fellowship, 2010) and a team of national and international assessment experts emphasise the importance of assessing student achievements that are judged against ‘consistent national and international standards that are subject to continuing dialogue, review and justification within disciplinary and professional communities’. Further, Boud argues for the value of ‘ongoing collaboration and dialogue to determine, review and moderate academic achievement standards’. A commitment to within-subject moderation practices is core to assessment policies across the Australian higher education sector. However, in a national issues paper on higher education - ‘Striving for Quality’ (DEST, 2002) - it was observed that: ‘there is not a strong tradition of systematic moderation of assessment and evaluation of performance within Australian universities … either between different markers in the same subject, across subjects, across courses or across institutions’ (p.28). This project is informed by Sadler’s approach to consensus moderation as a practical means of engaging academic staff in assuring academic achievement standards or ‘grade integrity’ (Sadler, 2009). While institution-level moderation at subject and program level is important, Sadler contends for the value of ‘calibrating’ standards across universities in common subjects and programs, using first-hand, primary evidence of academic standards in the form of assessment samples within the discipline. This project adopts Sadler’s notion of ‘calibrating’ academic staff as markers, graders and standards-keepers in order to calibrate standards at subject and program level. This is further supported by Maxwell (2002) and Harlen (2005) who show that development of a standards framework is not sufficient to ensure comparable judgement. Moderation techniques, for example face to face meetings, are an essential support for the calibration of these judgements (Klenowski, 2007). Boud (2010) reinforces the importance of staff development in developing expertise required for subject and program assessment responsibilities. This literature has informed the project methodology. Professional development and ‘calibration’ activities will take place during pre- and postmoderation discussion sessions, during exchanges among academics between participating universities, and during plenary gatherings and workshops. While the focus of this project is on subject-level moderation, consideration will also be given to approaches for program-level moderation, in light of the ALTC threshold learning outcomes across disciplines. Sadler (2010) argues for the value of peer review of grading and academic achievement standards leading to external calibration of end-of-degree standards. Such calibration exercises may take place in collaboration with other universities, professional accrediting agencies and employers. Principles of Sadler’s work will be applied in piloting these approaches with a view to informing a comprehensive follow-up project in the area. Standards and standards setting This project aims to develop an approach that may be adopted as part of a suite of strategies to demonstrate the sector’s commitment to self-regulating the assurance of academic standards. Empirical research supports the need for ‘standards-setters’ to be knowledgeable and credible, with a range of stakeholders across universities collectively engaged in the process (Yorke, 2008); standards should be based on the judgements of disciplinary experts using a method that provides examples of student achievement, and the method of setting standards should be supported by research (Norcini & Shea, 1997). The UK QAA supports the use of student achievement data to allow ‘cross-institutional and inter-institutional comparisons of student performance'. (QAA, 2008). Similarly, the 2009 AUQA discussion paper on standards (p.13) argues for the value of a ‘standards-referenced system of moderation’ requiring ‘moderators within a discipline to work together and to make reference to the specified standards in addition to grading processes and outcomes.’ While the UK external examiner system has been seen to provide assurance that students are performing at an appropriate level, the AUQA paper points out that ‘ensuring the comparability of assessment processes is not sufficient to ensure the assessment of achievement against specified standards. In the UK system, external examiners operate individually in relation to the institution, and not explicitly within the authority of the discipline.’ 4 While academic achievement standards and associated grading standards are of primary interest in this project, it is also informed by the valuing of other forms of standards, including input, process and delivery standards in the form of subject outlines, assessment tasks, grading rubrics and the like. The European Higher Education Area standards and guidelines for quality assurance (2009) highlights the importance of focusing on both input and outcome standards and Scott (2010) is adopting a similar process at UWS in relation to course design standards, support standards and delivery standards. Leading and managing change in relation to standards and quality In the ALTC program leaders project (forthcoming), Krause and colleagues have highlighted the pivotal role of program leaders in assuming responsibility for whole-of-program standards and quality. This builds on the ALTC Learning Leaders project (Scott et al., 2008) which found that program leaders were pivotal to effecting change and improvement at the discipline level. The project will be informed by the outcomes of this research, along with that of Fullan and Scott (2009, see also Lillis, 2007) who argue for the importance of steered engagement in which ‘top-down and bottom-up approaches are used together’ (p.85). This project combines the national and institutional imperatives (top-down) to assure quality and standards at subject and program level with a bottom-up strategy that engages academic staff in disciplinary communities (Radbourne & Nulty, 2002) in self-regulatory process that are at once robust and sustainable, and owned by those responsible for enacting curriculum and assessment processes (Nulty, forthcoming). 3.2 Project Research Questions The following research questions are informed by the theoretical framework and will guide the project methodology: 1. To what extent might a robust and validated inter-institutional moderation strategy contribute to the need for universities to demonstrate self-regulated approaches for monitoring and maintaining standards across disciplines? 2. To what extent can consensus be reached on input (e.g., assessment focus, criteria, valid assessment tasks and guidelines) and outcome (i.e., student achievement in subject-level assessment as evidenced in assessment samples) standards in disciplines? 3. What processes and resources are required for engaging academic staff in final year subject level moderation across universities and disciplines? 4. What disciplinary and institutional differences are evident in processes for managing inter-institutional moderation at subject level and what are the implications for accommodating such diversity in ongoing implementation? 5. In developing a proof of concept for assurance of program-level achievement standards, how might inter-institutional final year subject level moderation practices contribute to peer review of whole-ofprogram threshold learning outcomes in disciplines? 3.3 Methodology The methodology adopted in this project has been piloted during 2010 in a project involving six of the project universities. The proposed project will compare and evaluate the validity of a range of assessment inputs (subject outlines, assessment items, marking guides etc), along with the reliability of marking for the outcomes - at the final year level and across a range of FOEs. The project will expand and validate the 2010 pilot by including subjects that do and do not have external accreditation and high levels of employability. It will also include performance-based subjects in the creative arts, along with common final year subjects in Education as well as the more traditional disciplines. The project takes account of the Go8 Quality Verification System and its focus on a form of external peer review of final year undergraduate student outcomes. The focus of the latter is primarily on outcomes judged with respect to learning objectives, so it will be complementary to the proposed project which focuses on artefacts of student achievement and also includes input standards. Underpinning the methodology is the process of ‘steered engagement’ Fullan and Scott (2009, pp. 85-88), a proven approach for managing change in higher education. The senior university leaders on the project team will provide top level direction for project implementation, while academic staff in the disciplines will be supported by linking them up with ‘fellow travellers’ who are teaching and assessing the same subjects in 5 different universities. In this way the project combines a focus on summative assessment outcomes along with developmental professional development for academic staff. A project plan outlining key tasks for this three-phase, two-year project is summarised below. Phase 1: Project establishment and Program Leader recruitment (Dec 2010-Feb 2011) • recruit project personnel, including external evaluator; • develop communication and dissemination plan to ensure regular communication among project universities and across the sector during the project; • establish project Reference Group membership, including representatives from key stakeholder groups including: ALTC, TEQSA, DEEWR, Universities Australia • establish project Advisory Group membership, including national and international experts and project leaders of relevant ALTC projects to which this is linked, including Moderation for Fair Assessment in Transnational Learning and Teaching • liaise closely with ALTC Discipline Standards project leader to ensure close alignment between units selected for moderation and areas addressed in the threshold standards project; • agree on protocols and processes for moderation of inputs (unit outlines, assessment tasks and marking guides) and outcomes (student assessment tasks), including ethics clearance requirements; • determine the subjects to be used as the basis for Semester 1 2011 inter-institutional moderation activities, including analysis of unit equivalence; • consult with team members on criteria for selecting disciplines and sub-disciplines for moderation, along with a workplan for progressive moderation at subject level across 8 FOEs at final year undergraduate level. Where possible, moderation will include final year transdisciplinary and capstone subjects. An optin approach will be used - at least three universities will moderate in each common subject for each FOE; • establish contact with institutional representatives, program and unit leaders to discuss moderation protocols, and agree on processes for Phase 2 moderation activities within participating universities – this will involve at least one workshop within each participating university; • commence desk audit of inter-institutional moderation practices nationally and internationally to identify models of good practice; • optional (if funded): appoint two university level Assessment Fellows (at UWS and Griffith respectively, for 1-2 years) – members of academic staff who will support implementation and capacity-building as a proof of concept that will be evaluated as part of the project implementation. Phase 2: Implementation, consultation and benchmarking phase (March 2011- July 2012) • Phase 2 will involve implementation, documentation and review of inter-institutional moderation activities progressively over 3 semesters across project universities in the 8 discipline areas addressed in the ALTC Discipline standards project. This will include: o convening Reference Group and Advisory Group meetings at key milestone points – this will be particularly important for ensuring that the project implementation aligns with developments relating to TEQSA and outcomes from the Discipline Standards project; o establishing networking opportunities for subject and program coordinators involved in the project across universities – one institution-level summit is suggested within each project university. • Phase 2 will be subdivided by semesters as follows:o In Phase 2a (Sem 1 2011), significant scoping activity will be required to establish unit equivalence among participating universities and to agree on process, given the large number of institutions involved. A small number of subjects from among the pilot 2010 project team will be involved in interinstitutional moderation in Phase 2a. o Phase 2a will also include developing ‘how-to’ guidelines and resources on strategies for interinstitutional unit level moderation. These will be tested and validated by universities beyond the project team during semester 2 2011 and semester 1 2012. Advice on the guidelines will be sought from international advisors and the project reference group; o project universities will receive professional development in the form of i) at least one pre-moderation training workshop and one post-moderation debrief meeting with discipline-based unit and program coordinators involved in the project in each university; ii) professional development support from 6 University Assessment Fellows (if funded) for subject leaders in the two lead universities to test a proof-of-concept model of local-level capacity-building o optional (if funded): clarify roles and responsibilities of 2 uni-level Assessment Fellows (UWS & Griffith) o In Phase 2b (Sem 2 2011, Sem 1 2012) the majority of moderation will take place in this phase. Refine and repeat moderation activities from semester 1, 2011 with participating unit convenors and institutional representatives to validate processes, build on the initial process, and establish strategies for sustainability and scaleability across the sector; o convene sector-wide Summit on Assessment Quality and Standards at HERDSA July 2011. This will include presentations from other ALTC-funded and institution-level projects on assessment quality assurance and standards; o conduct formative evaluation – Sept 2011; submit progress report – Nov 2011. Phase 3: Reporting, dissemination and summative evaluation (Aug-Nov 2012) • complete online resource development and coursework moderation website; • evaluate and report on impact of moderation initiative on learning, teaching and capacity building via focus groups and interviews; • review the success and impact of the uni-level assessment fellows as an integral part of enhancing assessment quality and building capacity in the area of moderation; • convene second sector-wide Summit on Assessment Quality and Standards at HERDSA July 2012; • prepare and disseminate project report, including summative evaluation; • develop and launch project Assessment Quality and Standards resource website (see below); • liaise with TEQSA and ALTC on the role of project outcomes as a possible model among a suite of options for assuring quality and standards in assessment processes and outcomes across the sector; • disseminate and ‘workshop’ outcomes through four state-based forums, hosted by universities in Brisbane, Sydney, Melbourne and Perth. Further information about dissemination strategies is included in 2.5 below. 3.4 Change management and embedding strategies to achieve measurable outcomes The seniority and expertise of project team members ensures that change management and embedding strategies will be supported by members of the senior leadership teams in the participating universities. The key to success will be implementation of steered engagement strategies during the life of the project. Steered engagement, capacity-building and change management will be facilitated through such activities as:• institutional capacity-building and professional development among local academic staff networks across and within participating FOEs in each institution. The project leader/national coordinator (if resourced) will foster these networks through pre- and post-moderation briefing sessions. • optional (if funded): in the two lead institutions, the University Assessment Fellow, in collaboration with project team members and the respective Griffith and UWS academic development units, will implement a model for supporting staff capacity-building and change management at institution level; • the annual AQS Summit, along with State-based Fora. Project team members will also investigate ways of embedding this process in daily practice within universities’ standards operating processes, including an exploration of: • embedding cyclical external moderations of particular FOEs using the project’s methodology • a potentially new and significant role for Academic Senates and Academic Senate Chairs – the notion of giving the processes being used to implement the moderation of assessment inputs and outcomes as being effective and consistent – ‘ a heart foundation’ stamp. 3.5 Evaluation framework A range of formative and summative evaluation strategies will be used to guide the project. These include: • regular project meetings via tele- and video-conference and online to review and reflect on the progress of the project; • regular reports to the Reference Group on the project’s direction, milestones and budget expenditure; 7 feedback from the Reference and Advisory Groups will be sought as part of the evaluation strategy; evaluation of the implementation and its outcomes in the form of surveys and focus group interviews of participants and senior leaders across all participating universities; • a formative evaluation report (end Year 1) and a summative evaluation report as part of the final report (end Year 2) for the ALTC. • • During Phase 2 (Implementation, consultation, benchmarking), workshop and online resources will be piloted and evaluated by members of the project team and selected relevant stakeholders. During Phase 3 a formal evaluation will be conducted on the impact of the initiative on learning, teaching, assessment and moderation practices and capacity building in participating universities. Findings will be included in the final project report. A formal independent evaluation will be a key component of this project’s evaluation strategy. A suitably qualified external evaluator will be commissioned by the project team and the Reference Group to conduct an expert review of how the project was conducted and to review the outcomes of the project. The project team will provide initial frames of reference for the external evaluator and will also supply him/her with documentation (meeting minutes, reports, etc) to facilitate the evaluation. 3.6 Deliverables and dissemination Deliverables for the project will include: i. a website comprising ready-to-use, downloadable moderation guides, step-by-step strategies for replicating and implementing the project across the sector, including FOE-specific user guides and guidelines for initiating subject-level moderation practices at university and sector-wide level; ii. guidelines for reviewing subject-level assessment quality and practices in the context of program-level outcomes and standards; iii. investigation of the potential to use IT-enabled systems to gather and review sample assessment items to assist in the standards-setting process and capacity-building for academic staff across disciplines – e.g. the potential to use tools like Confluence; iv. evaluation of user views on utility, feasibility and validity of the proposed inter-institutional moderation model in assuring quality in different fields of education. Key motivators will be explored through a survey of participants including views about the connection between inter-institutional moderation activities and leadership in learning and teaching, integration into promotion criteria etc.; v. a database of existing examples of assessment (i.e., actual assessment ‘artefacts’ at different grade levels x FOE with explanations of why they attracted the grade received) vi. a report on key findings, with recommendations for systemic implementation. It will include an overall framework within which to locate this project and show how it links to and complements parallel projects and initiatives including ALTC-funded projects, TEQSA developments, institutional consortia moderation activities and AQF developments; vii. recommendations regarding the feasibility of a National Assessment Quality and Standards Fellow or Advisor to connect the various important initiatives now underway (e.g., Go8 project, ALTC Discipline Standards project, TEQSA focus on evidencing and assuring standards). This person may be an international scholar who can provide a ‘critical friend’ and evaluator role for the sector in light of current initiatives and policy developments in relation to assessment and standards; viii. two national Assessment Quality and Standards (AQS) Summits, including senior university leaders, state and national government policy makers, ALTC and AUQA/TEQSA representatives, school and VET representatives. The AQS Summits will include the following elements: • identify and discuss models for assuring quality and standards in assessment and program-level outcomes across the sector; • examine issues relating to operationalising these models and their implications for cross-sectoral collaborations; and • disseminate examples of good practice. Dissemination strategies A range of strategies will be used to disseminate project outcomes and deliverables. Many of these strategies have been built into Dissemination Phase 3 as outlined in 2.2 above. Key dissemination strategies are: • formative dissemination through institution-level networks for subject and program leaders involved in the 8 • • • • • moderation process; State-based Fora to test the proof-of-concept model and the veracity of processes developed and data gathered; identify implications for action across the sector; share resources and guidelines for interinstitutional moderation. Additional funding is sought to expand the number of Fora for the purposes of widespread sector-wide engagement (see Attachment 1 – Budget, p.13); one national AQS Summits at HERDSA (2011). It is expected that the AQS Summit will convene on an annual basis over the next 3-5 years – possibly using the HERDSA mid-year conference as a meeting point. International advisors may also be included, depending on budget available. Additional funding is sought for a second Summit at the close of the project (see Attachment 1 – Budget, p.13); web-based dissemination of resources and guidelines; ALTC Final Report: A report documenting the approach and outcomes of the project will be prepared and published for the ALTC. This report will contain an executive summary of the project outcomes, a description of the methodologies and approaches used, the results of the investigation and benchmarking activities, a description of dissemination strategies employed and a discussion of policy implications for the higher education sector in Australia. academic publications: The findings and outcomes of this project will be presented in national and international conferences and published in international journals. References AUQA. (2009, May). Setting and monitoring academic standards for Australian higher education: A discussion paper. Available online: www.auqa.edu.au/qualityenhancement/academicstandards/ Boud, D. & Associates. (2010). Assessment 2020: Seven propositions for assessment reform in higher education. Sydney: Australian Learning and Teaching Council. Brown, R. (2010). Comparability of degree standards? Available online: www.hepi.ac.uk. Brown, S., & Knight, P. (1994). Assessing learners in higher education. London: Kogan Page. DEST. (2002). Striving for quality: Learning, teaching and scholarship. Canberra: AGPS. ENQA. (2009). Standards and guidelines for quality assurance. Available online: www.enqa.eu/pubs.lasso Fullan, M., & Scott, G. (2009). Turnaround leadership for higher education. San Francisco: Jossey Bass. Feldt, L., & Brennan, R. (1989). Reliability. In R. Linn (Ed.). Educational measurement (3rd ed.). (pp.105-146). New York: Macmillan. Harlen, W. (2005). Trusting teachers’ judgement: research evidence of the reliability and validity of teachers’ assessment used for summative purposes. Research Papers in Education, 20(3), 245-270. Klenowski, V. (2007). Evaluation of the effectiveness of the consensus-based standards validation process, Brisbane Qld: DETA. Knight, P. (2006). The local practices of assessment. Assessment and Evaluation in Higher Education, 31(4), 435-452. Kreber, C. (Ed.). (2009). The university and its disciplines. New York: Routledge. Lillis, D. (2007, May). Steered by engagement. In L. Bollaert et al. Embedding quality culture in higher education. EUA. Maxwell, G. (2002). Moderation of teacher judgements in student assessment. Brisbane: Queensland School Curriculum Council. Messick, S. (1989). Validity. In R. Linn (Ed.). Educational measurement (3rd ed.). (pp.13-103). New York: Macmillan. Norcini, J. & Shea, J. (1997). The credibility and comparability of standards. Applied Measurement in Education, 10(1), 39-59. Nulty, D. (forthcoming). Peer and self-assessment in the first year of university. Assessment and Evaluation in Higher Education. Radbourne, J., & Nulty, D. (2002). Immersing a faculty in assessment: A case study of an organisational Culture Shift. In Assessing Learning in Australian Universities, See: http://www.cshe.unimelb.edu.au/assessinglearning/ Rosovsky, H., & Hartley, M. (2002). Evaluation and the academy. Cambridge, MA: AAAS. Sadler, R. (2009). Grade integrity and the representation of academic achievement. Studies in Higher Education, 34(7), 807-826. Sadler, R. (2010). Operational guidelines for assuring academic achievement standards at Griffith. [internal document] Scott, G., Coates, H., & Anderson, M. (2008). Learning leaders in times of change. Camberwell: ACER. Smith,D. (1992). Validity of faculty judgements of student performance. The Journal of Higher Education, 63(3), 329-340. Trowler, P. (2009). Beyond epistemological essentialism: Academic tribes in the 21st century. In C. Kreber (Ed.) (pp. 181-196). The university and its disciplines. New York: Routledge. Yorke, M. (2008). Grading student achievement in higher education. London: Routledge. Yorke, M. (2009). Assessment for career and citizenship. In C. Kreber (Ed.) (pp. 221-230). The university and its disciplines. New York: Routledge. 9