FINAL
Project Title:
A sector-wide model for assuring final year subject and program achievement standards
through inter-university moderation
1. Outcomes
This project will yield a validated, robust approach for assuring subject achievement standards through
inter-university moderation in common final year subjects across disciplines. It will also trial
approaches for moderating and assuring program achievement standards, building on ALTC discipline
standards. Project resources will assist universities to implement sustainable, self-regulatory
moderation processes for monitoring subject and program standards.
2. Abstract
This proof-of-concept project will produce resources to guide inter-institutional moderation for assuring
final year subject and program achievement standards. Eight universities will identify common final
year subjects across eight disciplines aligned with the ALTC discipline standards project. Subject
convenors will share subject outlines and selected assessment artefacts for review by at least two
other project universities. The moderation process includes inputs (e.g., subject outlines, assessment
tasks, marking criteria) and outcomes (i.e., assessment samples). External blind peer review of both
inputs and outcomes will determine the consistency of subject-level standards and how these
compare with comparable final year subjects in other universities. Where relevant, capstone subjects
will be used and program learning outcomes considered to identify approaches for assuring program
achievement standards through inter-university moderation. The project addresses the TEQSA
imperative to demonstrate sector-level, self-regulated, robust approaches for assuring quality and
standards and highlights the role of peer review. Guidelines for practice will be sustainable and owned
by academic disciplinary communities.
External moderation and coursework 1 assessment
Project Title:
A sector-wide model for assuring final year subject and program achievement standards
through inter-university moderation
Project Team: Prof Kerri-Lee Krause1 and Prof Geoff Scott2 (co-leaders)
Prof Stuart Campbell2, A/Prof Martin Carroll3, Professor Liz Deane4, Dr Duncan Nulty1, Prof
Pip Pattison5, Prof Belinda Probert6, Prof Judyth Sachs7, Prof Stephen Towers8
1Griffith
University
2University of Western Sydney
3Charles Darwin University
4Australian
National University
5The University of Melbourne
6La Trobe University
7Macquarie
University
University of Technology
8Queensland
1.
Project Rationale and Outcomes
1.1 Context
This project is proposed at a time of national and international focus on higher education assessment
outcomes, quality and standards, with an emphasis on universities’ capacity to demonstrate how they set and
meet standards across discipline areas. A number of factors are acting in combination to account for the
significant increase in focus on this area over the last few years:
• the Australian Universities Quality Agency (AUQA, 2009) expert working party, discussion paper and
sector-wide consultation on setting and monitoring academic standards in Australian higher education;
• introduction of the Tertiary Education Quality and Standards Agency (TEQSA) and an increased policy
focus on standards, including the development of a revised regulatory standards framework for higher
education;
• introduction of the Collegiate Learning Assessment as a new national indicator to gauge the value
added by universities to student learning outcomes and skill development;
• internationally, the OECD’s (2009) International Assessment of Higher Education Learning Outcomes
(AHELO) project is examining the feasibility of assessing and comparing graduate learning, including
the value added during higher education, across nations, systems, languages and cultures;
• evidence of a significant growth of interest in proactive action with regard to measuring and evidencing
student outcomes by Vice-Chancellors and higher education networks like the ATN and Go8;
• a growing number of cases of litigation concerning the quality of assessment in universities, both
within and beyond Australia;
• emphasis on the portability and equivalence of qualifications within and beyond Australia, including
revisions to the Australian Qualifications Framework (AQF);
• international competition in the higher education market as a consequence of the Bologna process.
The imperative now is for every university to focus on both quality assurance and quality enhancement in
assessment and standards. The focus needs to be on assuring the quality and standards of not only the inputs
but also the outcomes of assessment in ways that are relevant to specific disciplines and professions.
Importantly, the sector also needs to demonstrate ways in which it is supporting diversity and responsiveness,
along with peer review and local ownership in self-regulation of standards and quality.
1.2 The proposed project and outcomes
The project will develop and validate a proof-of-concept inter-institutional model for external validation of
assessment processes and outcomes at subject level, while taking into account whole-of-degree program
outcomes. It will build on a self-initiated pilot undertaken during 2010, involving three final year subjects and
discipline areas across six of the universities in the project team. The project will result in a scaleable model
and resources to enable self-regulated external moderation and benchmarking of final year subject-level
outcomes across the sector and across disciplines. Mindful of the importance of a whole-of-program
1 Definition of terms: i) coursework: curriculum covered at unit/subject and course/program level. ii) program/course: whole-of-degree
program. iii) subject/unit: an individual unit taken as part of a whole-of-degree program. iv) capstone subject: a final year culminating
subject taken at the end of the program in which students showcase capabilities and competencies developed during the program.
1
perspective on standards and outcomes, the project will also pilot approaches for moderating program-level
achievement standards, based on work with program leaders responsible for capstone courses in selected
universities. The challenges of such a process are fully acknowledged (Brown, 2010), nevertheless there is an
urgent need to pursue a range of robust approaches to demonstrate and further enhance how standards are
being assured at both subject and program level.
The proposed project will facilitate external validation across selected common final year subjects in the 8
Fields of Education (FOE) used in the ALTC Discipline Standards project. The ALTC Discipline Standards
project focuses on the threshold learning outcomes for different FOEs. The proposed project complements this
approach by focussing on evidence of ‘how’ these standards are evidenced through coursework and
assessment artefacts at the subject level. While inter-institutional subject level moderation is the primary focus,
the project team will have the opportunity to conduct a stocktake of capstone subjects as a basis for
investigating program-level outcomes and standards. It is expected that the project outcomes will include
recommendations for a comprehensive program-level moderation project, based on the pilot activities and the
capstone stocktake results of the proposed project.
The subject level moderation process will involve peer review of subject inputs, along with blind peer marking
of assessment outcomes. The focus on final year subject outcomes is key to addressing the policy imperative
to demonstrate exit standards at the discipline level. Rather than relying on a generic graduate skills
assessment test, this project focuses on evidence of actual student achievement in disciplinary contexts as
they prepare to conclude their undergraduate study. In each of the 8 FOEs, final year subjects that are
common to at least three universities in the project team will be selected, with a view to reviewing outcome
standards as well as the quality of inputs including:
• the demonstrable relevance of the capabilities and competencies being assessed;
• how these capabilities and competencies are being assessed;
• how marks and grades are being determined; and
• how consistency in assessment inputs for the same subject is being achieved between institutions.
Based on an evaluation of the 2010 self-initiated moderation pilot, the project team is confident that the
proposed project will enhance the quality of learning, teaching and assessment in the sector in several ways,
including:
• producing a validated proof-of-concept inter-institutional subject-level moderation model that is
sustainable and scaleable for use across the sector and across disciplines;
• an evaluation of relative benefits of consensus moderation and the G08 Quality Verification System for
assuring standards, based on feedback from the two Go8 university representatives;
• a stocktake of capstone subjects (in collaboration with existing ALTC capstone projects) with a view to
recommending approaches for addressing program-level outcomes and standards;
• networks of institution- and discipline-based academic staff networks to share good practice in
assessment and moderation across disciplines;
• capacity-building among subject and program leaders who develop insights into the assessment and
grading practices of colleagues teaching comparable final year subjects, including sharing of subject
outlines and grading guides;
• practical, on-the-job experience for academic staff in standards-based assessment and interinstitutional moderation practices;
• development of a common language around issues of moderation and standards;
• resources and capacity-building opportunities for academic staff and senior university leaders in
relation to quality and standards in anticipation of TEQSA requirements for self-regulated
accountability and quality assurance mechanisms that are transparent, rigorous and sustainable.
Information on how these outcomes will be disseminated is included in 2.5 below.
1.3 Rationale for the project
The rationale for this project is threefold.
a. It addresses ethical and educational imperatives to ensure that Australian universities have demonstrably
effective processes in place to assure the quality of assessment processes and outcomes for students
2
across the sector.
b. It supports social and economic imperatives associated with demonstrating to key stakeholders, such as
industry and community members, that universities have robust approaches for assuring assessment
quality and standards that are aligned to achievement of whole-of-program graduate capabilities and
learning outcomes.
c. It takes account of the higher education policy and regulatory environment which requires the sector to
make explicit its processes for assuring quality in assessment processes and outcomes through a
combination of institution-level systems and policies, as well as inter-institutional benchmarking and
assessment validation practices. Explication of this kind adds value to the Australian sector, both
nationally and internationally, as student mobility increases and as credit transfer arrangements in the
context of processes such as Bologna become increasingly important.
2.
Value/Need for the Project
The need for this project is argued on several grounds which include a range of value add dimensions.
i. In the context of a new national regulatory framework it is important for the sector to show that it is willing
to address assessment quality and standards issue by:
a. expanding the notion of standards to include inputs (e.g., quality of assessment tasks and grading
procedures) and students’ academic achievement outcomes;
b. developing, testing and evaluating a model of external validation;
c. adopting a rigorous, validated and feasible approach to self-regulated, transparent strategies for
assuring standards and the consistent quality of assessment.
ii. The model that will result will provide Australia’s university sector with a viable alternative model for assure
standards whilst maintaining system flexibility, responsiveness and diversity.
iii. The project team comprises experts who are actively involved in sector-wide projects and policy
discussions regarding quality and standards in higher education. Their combined expertise will inform the
development of an integrating framework that will guide the project and within which to locate the range of
sector-wide quality and standards-related projects underway.
iv. The project outcomes will add value to the sector by producing a model for constructive steered
engagement with an enhancement focus in relation to assuring quality and standards in the sector.
v. The focus on the nature and validity of subject-level assessment standards and outcomes in the context of
program-level outcomes is a further fundamentally important dimension.
3.
Project Approach
3.1 Theoretical framework
Assessment and grading: a social, discipline-based practice
Assessment and grading are acknowledged as social practices (Yorke, 2008) into which ‘new colleagues are
inducted’ (p.6). The epistemological characteristics of the discipline shape assessment practices (Kreber,
2009; Trowler, 2009), hence the importance of ensuring disciplinary representation in this project. It will be
particularly important to ensure that the guidelines and resources emerging from the project are sufficiently
flexible to accommodate disciplinary differences. Yorke (2008) found that a majority of academic staff
developed their understanding of marking students’ work from talking with and observing colleagues, while
some said they adopted assessment practices based on their experience as students. Academics undertake
relatively little developmental work in the area of assessment (Rosovsky & Hartley, 2002, p.14), yet there are
significant technical considerations and skills underpinning effective assessment practices. For instance, the
principles of validity (Messick, 1989) and reliability (Feldt & Brennan, 1989) are core and they underpin the
theoretical framework for this project. Reliability, in the sense of replicability of performance, is particularly
problematic for universities typically do not have the resources to conduct assessment on a scale sufficient to
ensure reliability (Yorke, 2009). Yorke (2008, p.24) acknowledges that grading of student work is subject to
considerable variability that may result from the assessor’s interpretation of the assigned task and grading
criteria. Knight (2006) also observes that grading is a ‘local’ process which means that grading in one
institution may not be replicated in another. Such variability poses considerable risk to the sector for, as Smith
(1992) argues, for grades to be considered reliable, they should be reasonably stable across courses in the
disciplines concerned. Inter-institutional moderation is recognized as a valuable strategy for addressing this
3
challenge (Brown & Knight, 1994).
Assuring standards in disciplinary communities: consensus moderation and peer review
Moderation is a peer review process for developing consistency or comparability of assessment judgements.
Sadler (2010) argues for the value of consensus moderation for assuring grading standards and academic
achievement standards to attain quality assurance of the summative grading process. Likewise Boud (ALTC
Fellowship, 2010) and a team of national and international assessment experts emphasise the importance of
assessing student achievements that are judged against ‘consistent national and international standards that
are subject to continuing dialogue, review and justification within disciplinary and professional communities’.
Further, Boud argues for the value of ‘ongoing collaboration and dialogue to determine, review and moderate
academic achievement standards’. A commitment to within-subject moderation practices is core to
assessment policies across the Australian higher education sector. However, in a national issues paper on
higher education - ‘Striving for Quality’ (DEST, 2002) - it was observed that: ‘there is not a strong tradition of
systematic moderation of assessment and evaluation of performance within Australian universities … either
between different markers in the same subject, across subjects, across courses or across institutions’ (p.28).
This project is informed by Sadler’s approach to consensus moderation as a practical means of engaging
academic staff in assuring academic achievement standards or ‘grade integrity’ (Sadler, 2009). While
institution-level moderation at subject and program level is important, Sadler contends for the value of
‘calibrating’ standards across universities in common subjects and programs, using first-hand, primary
evidence of academic standards in the form of assessment samples within the discipline. This project adopts
Sadler’s notion of ‘calibrating’ academic staff as markers, graders and standards-keepers in order to calibrate
standards at subject and program level. This is further supported by Maxwell (2002) and Harlen (2005) who
show that development of a standards framework is not sufficient to ensure comparable judgement.
Moderation techniques, for example face to face meetings, are an essential support for the calibration of these
judgements (Klenowski, 2007). Boud (2010) reinforces the importance of staff development in developing
expertise required for subject and program assessment responsibilities. This literature has informed the project
methodology. Professional development and ‘calibration’ activities will take place during pre- and postmoderation discussion sessions, during exchanges among academics between participating universities, and
during plenary gatherings and workshops.
While the focus of this project is on subject-level moderation, consideration will also be given to approaches
for program-level moderation, in light of the ALTC threshold learning outcomes across disciplines. Sadler
(2010) argues for the value of peer review of grading and academic achievement standards leading to external
calibration of end-of-degree standards. Such calibration exercises may take place in collaboration with other
universities, professional accrediting agencies and employers. Principles of Sadler’s work will be applied in
piloting these approaches with a view to informing a comprehensive follow-up project in the area.
Standards and standards setting
This project aims to develop an approach that may be adopted as part of a suite of strategies to demonstrate
the sector’s commitment to self-regulating the assurance of academic standards. Empirical research supports
the need for ‘standards-setters’ to be knowledgeable and credible, with a range of stakeholders across
universities collectively engaged in the process (Yorke, 2008); standards should be based on the judgements
of disciplinary experts using a method that provides examples of student achievement, and the method of
setting standards should be supported by research (Norcini & Shea, 1997). The UK QAA supports the use of
student achievement data to allow ‘cross-institutional and inter-institutional comparisons of student
performance'. (QAA, 2008). Similarly, the 2009 AUQA discussion paper on standards (p.13) argues for the
value of a ‘standards-referenced system of moderation’ requiring ‘moderators within a discipline to work
together and to make reference to the specified standards in addition to grading processes and outcomes.’
While the UK external examiner system has been seen to provide assurance that students are performing at
an appropriate level, the AUQA paper points out that ‘ensuring the comparability of assessment processes is
not sufficient to ensure the assessment of achievement against specified standards. In the UK system,
external examiners operate individually in relation to the institution, and not explicitly within the authority of the
discipline.’
4
While academic achievement standards and associated grading standards are of primary interest in this
project, it is also informed by the valuing of other forms of standards, including input, process and delivery
standards in the form of subject outlines, assessment tasks, grading rubrics and the like. The European Higher
Education Area standards and guidelines for quality assurance (2009) highlights the importance of focusing on
both input and outcome standards and Scott (2010) is adopting a similar process at UWS in relation to course
design standards, support standards and delivery standards.
Leading and managing change in relation to standards and quality
In the ALTC program leaders project (forthcoming), Krause and colleagues have highlighted the pivotal role of
program leaders in assuming responsibility for whole-of-program standards and quality. This builds on the
ALTC Learning Leaders project (Scott et al., 2008) which found that program leaders were pivotal to effecting
change and improvement at the discipline level. The project will be informed by the outcomes of this research,
along with that of Fullan and Scott (2009, see also Lillis, 2007) who argue for the importance of steered
engagement in which ‘top-down and bottom-up approaches are used together’ (p.85). This project combines
the national and institutional imperatives (top-down) to assure quality and standards at subject and program
level with a bottom-up strategy that engages academic staff in disciplinary communities (Radbourne & Nulty,
2002) in self-regulatory process that are at once robust and sustainable, and owned by those responsible for
enacting curriculum and assessment processes (Nulty, forthcoming).
3.2 Project Research Questions
The following research questions are informed by the theoretical framework and will guide the project
methodology:
1. To what extent might a robust and validated inter-institutional moderation strategy contribute to the
need for universities to demonstrate self-regulated approaches for monitoring and maintaining
standards across disciplines?
2. To what extent can consensus be reached on input (e.g., assessment focus, criteria, valid assessment
tasks and guidelines) and outcome (i.e., student achievement in subject-level assessment as
evidenced in assessment samples) standards in disciplines?
3. What processes and resources are required for engaging academic staff in final year subject level
moderation across universities and disciplines?
4. What disciplinary and institutional differences are evident in processes for managing inter-institutional
moderation at subject level and what are the implications for accommodating such diversity in ongoing
implementation?
5. In developing a proof of concept for assurance of program-level achievement standards, how might
inter-institutional final year subject level moderation practices contribute to peer review of whole-ofprogram threshold learning outcomes in disciplines?
3.3 Methodology
The methodology adopted in this project has been piloted during 2010 in a project involving six of the project
universities. The proposed project will compare and evaluate the validity of a range of assessment inputs
(subject outlines, assessment items, marking guides etc), along with the reliability of marking for the outcomes
- at the final year level and across a range of FOEs. The project will expand and validate the 2010 pilot by
including subjects that do and do not have external accreditation and high levels of employability. It will also
include performance-based subjects in the creative arts, along with common final year subjects in Education
as well as the more traditional disciplines. The project takes account of the Go8 Quality Verification System
and its focus on a form of external peer review of final year undergraduate student outcomes. The focus of the
latter is primarily on outcomes judged with respect to learning objectives, so it will be complementary to the
proposed project which focuses on artefacts of student achievement and also includes input standards.
Underpinning the methodology is the process of ‘steered engagement’ Fullan and Scott (2009, pp. 85-88), a
proven approach for managing change in higher education. The senior university leaders on the project team
will provide top level direction for project implementation, while academic staff in the disciplines will be
supported by linking them up with ‘fellow travellers’ who are teaching and assessing the same subjects in
5
different universities. In this way the project combines a focus on summative assessment outcomes along with
developmental professional development for academic staff.
A project plan outlining key tasks for this three-phase, two-year project is summarised below.
Phase 1: Project establishment and Program Leader recruitment (Dec 2010-Feb 2011)
• recruit project personnel, including external evaluator;
• develop communication and dissemination plan to ensure regular communication among project
universities and across the sector during the project;
• establish project Reference Group membership, including representatives from key stakeholder groups
including: ALTC, TEQSA, DEEWR, Universities Australia
• establish project Advisory Group membership, including national and international experts and project
leaders of relevant ALTC projects to which this is linked, including Moderation for Fair Assessment in
Transnational Learning and Teaching
• liaise closely with ALTC Discipline Standards project leader to ensure close alignment between units
selected for moderation and areas addressed in the threshold standards project;
• agree on protocols and processes for moderation of inputs (unit outlines, assessment tasks and marking
guides) and outcomes (student assessment tasks), including ethics clearance requirements;
• determine the subjects to be used as the basis for Semester 1 2011 inter-institutional moderation
activities, including analysis of unit equivalence;
• consult with team members on criteria for selecting disciplines and sub-disciplines for moderation, along
with a workplan for progressive moderation at subject level across 8 FOEs at final year undergraduate
level. Where possible, moderation will include final year transdisciplinary and capstone subjects. An optin approach will be used - at least three universities will moderate in each common subject for each FOE;
• establish contact with institutional representatives, program and unit leaders to discuss moderation
protocols, and agree on processes for Phase 2 moderation activities within participating universities –
this will involve at least one workshop within each participating university;
• commence desk audit of inter-institutional moderation practices nationally and internationally to identify
models of good practice;
• optional (if funded): appoint two university level Assessment Fellows (at UWS and Griffith respectively,
for 1-2 years) – members of academic staff who will support implementation and capacity-building as a
proof of concept that will be evaluated as part of the project implementation.
Phase 2: Implementation, consultation and benchmarking phase (March 2011- July 2012)
• Phase 2 will involve implementation, documentation and review of inter-institutional moderation activities
progressively over 3 semesters across project universities in the 8 discipline areas addressed in the
ALTC Discipline standards project. This will include:
o convening Reference Group and Advisory Group meetings at key milestone points – this will be
particularly important for ensuring that the project implementation aligns with developments relating
to TEQSA and outcomes from the Discipline Standards project;
o establishing networking opportunities for subject and program coordinators involved in the project
across universities – one institution-level summit is suggested within each project university.
• Phase 2 will be subdivided by semesters as follows:o In Phase 2a (Sem 1 2011), significant scoping activity will be required to establish unit equivalence
among participating universities and to agree on process, given the large number of institutions
involved. A small number of subjects from among the pilot 2010 project team will be involved in interinstitutional moderation in Phase 2a.
o Phase 2a will also include developing ‘how-to’ guidelines and resources on strategies for interinstitutional unit level moderation. These will be tested and validated by universities beyond the
project team during semester 2 2011 and semester 1 2012. Advice on the guidelines will be sought
from international advisors and the project reference group;
o project universities will receive professional development in the form of i) at least one pre-moderation
training workshop and one post-moderation debrief meeting with discipline-based unit and program
coordinators involved in the project in each university; ii) professional development support from
6
University Assessment Fellows (if funded) for subject leaders in the two lead universities to test a
proof-of-concept model of local-level capacity-building
o optional (if funded): clarify roles and responsibilities of 2 uni-level Assessment Fellows (UWS &
Griffith)
o In Phase 2b (Sem 2 2011, Sem 1 2012) the majority of moderation will take place in this phase.
Refine and repeat moderation activities from semester 1, 2011 with participating unit convenors and
institutional representatives to validate processes, build on the initial process, and establish
strategies for sustainability and scaleability across the sector;
o convene sector-wide Summit on Assessment Quality and Standards at HERDSA July 2011. This will
include presentations from other ALTC-funded and institution-level projects on assessment quality
assurance and standards;
o conduct formative evaluation – Sept 2011; submit progress report – Nov 2011.
Phase 3: Reporting, dissemination and summative evaluation (Aug-Nov 2012)
• complete online resource development and coursework moderation website;
• evaluate and report on impact of moderation initiative on learning, teaching and capacity building via
focus groups and interviews;
• review the success and impact of the uni-level assessment fellows as an integral part of enhancing
assessment quality and building capacity in the area of moderation;
• convene second sector-wide Summit on Assessment Quality and Standards at HERDSA July 2012;
• prepare and disseminate project report, including summative evaluation;
• develop and launch project Assessment Quality and Standards resource website (see below);
• liaise with TEQSA and ALTC on the role of project outcomes as a possible model among a suite of
options for assuring quality and standards in assessment processes and outcomes across the sector;
• disseminate and ‘workshop’ outcomes through four state-based forums, hosted by universities in
Brisbane, Sydney, Melbourne and Perth.
Further information about dissemination strategies is included in 2.5 below.
3.4 Change management and embedding strategies to achieve measurable outcomes
The seniority and expertise of project team members ensures that change management and embedding
strategies will be supported by members of the senior leadership teams in the participating universities. The
key to success will be implementation of steered engagement strategies during the life of the project. Steered
engagement, capacity-building and change management will be facilitated through such activities as:• institutional capacity-building and professional development among local academic staff networks
across and within participating FOEs in each institution. The project leader/national coordinator (if
resourced) will foster these networks through pre- and post-moderation briefing sessions.
• optional (if funded): in the two lead institutions, the University Assessment Fellow, in collaboration with
project team members and the respective Griffith and UWS academic development units, will
implement a model for supporting staff capacity-building and change management at institution level;
• the annual AQS Summit, along with State-based Fora.
Project team members will also investigate ways of embedding this process in daily practice within universities’
standards operating processes, including an exploration of:
• embedding cyclical external moderations of particular FOEs using the project’s methodology
• a potentially new and significant role for Academic Senates and Academic Senate Chairs – the notion
of giving the processes being used to implement the moderation of assessment inputs and outcomes
as being effective and consistent – ‘ a heart foundation’ stamp.
3.5 Evaluation framework
A range of formative and summative evaluation strategies will be used to guide the project. These include:
• regular project meetings via tele- and video-conference and online to review and reflect on the progress
of the project;
• regular reports to the Reference Group on the project’s direction, milestones and budget expenditure;
7
feedback from the Reference and Advisory Groups will be sought as part of the evaluation strategy;
evaluation of the implementation and its outcomes in the form of surveys and focus group interviews of
participants and senior leaders across all participating universities;
• a formative evaluation report (end Year 1) and a summative evaluation report as part of the final report
(end Year 2) for the ALTC.
•
•
During Phase 2 (Implementation, consultation, benchmarking), workshop and online resources will be piloted
and evaluated by members of the project team and selected relevant stakeholders. During Phase 3 a formal
evaluation will be conducted on the impact of the initiative on learning, teaching, assessment and moderation
practices and capacity building in participating universities. Findings will be included in the final project report.
A formal independent evaluation will be a key component of this project’s evaluation strategy. A suitably
qualified external evaluator will be commissioned by the project team and the Reference Group to conduct an
expert review of how the project was conducted and to review the outcomes of the project. The project team
will provide initial frames of reference for the external evaluator and will also supply him/her with
documentation (meeting minutes, reports, etc) to facilitate the evaluation.
3.6 Deliverables and dissemination
Deliverables for the project will include:
i. a website comprising ready-to-use, downloadable moderation guides, step-by-step strategies for
replicating and implementing the project across the sector, including FOE-specific user guides and
guidelines for initiating subject-level moderation practices at university and sector-wide level;
ii. guidelines for reviewing subject-level assessment quality and practices in the context of program-level
outcomes and standards;
iii. investigation of the potential to use IT-enabled systems to gather and review sample assessment
items to assist in the standards-setting process and capacity-building for academic staff across
disciplines – e.g. the potential to use tools like Confluence;
iv. evaluation of user views on utility, feasibility and validity of the proposed inter-institutional moderation
model in assuring quality in different fields of education. Key motivators will be explored through a
survey of participants including views about the connection between inter-institutional moderation
activities and leadership in learning and teaching, integration into promotion criteria etc.;
v. a database of existing examples of assessment (i.e., actual assessment ‘artefacts’ at different grade
levels x FOE with explanations of why they attracted the grade received)
vi. a report on key findings, with recommendations for systemic implementation. It will include an overall
framework within which to locate this project and show how it links to and complements parallel
projects and initiatives including ALTC-funded projects, TEQSA developments, institutional consortia
moderation activities and AQF developments;
vii. recommendations regarding the feasibility of a National Assessment Quality and Standards Fellow or
Advisor to connect the various important initiatives now underway (e.g., Go8 project, ALTC Discipline
Standards project, TEQSA focus on evidencing and assuring standards). This person may be an
international scholar who can provide a ‘critical friend’ and evaluator role for the sector in light of
current initiatives and policy developments in relation to assessment and standards;
viii. two national Assessment Quality and Standards (AQS) Summits, including senior university leaders,
state and national government policy makers, ALTC and AUQA/TEQSA representatives, school and
VET representatives. The AQS Summits will include the following elements:
• identify and discuss models for assuring quality and standards in assessment and program-level
outcomes across the sector;
• examine issues relating to operationalising these models and their implications for cross-sectoral
collaborations; and
• disseminate examples of good practice.
Dissemination strategies
A range of strategies will be used to disseminate project outcomes and deliverables. Many of these strategies
have been built into Dissemination Phase 3 as outlined in 2.2 above. Key dissemination strategies are:
• formative dissemination through institution-level networks for subject and program leaders involved in the
8
•
•
•
•
•
moderation process;
State-based Fora to test the proof-of-concept model and the veracity of processes developed and data
gathered; identify implications for action across the sector; share resources and guidelines for interinstitutional moderation. Additional funding is sought to expand the number of Fora for the purposes of
widespread sector-wide engagement (see Attachment 1 – Budget, p.13);
one national AQS Summits at HERDSA (2011). It is expected that the AQS Summit will convene on an
annual basis over the next 3-5 years – possibly using the HERDSA mid-year conference as a meeting
point. International advisors may also be included, depending on budget available. Additional funding is
sought for a second Summit at the close of the project (see Attachment 1 – Budget, p.13);
web-based dissemination of resources and guidelines;
ALTC Final Report: A report documenting the approach and outcomes of the project will be prepared and
published for the ALTC. This report will contain an executive summary of the project outcomes, a
description of the methodologies and approaches used, the results of the investigation and benchmarking
activities, a description of dissemination strategies employed and a discussion of policy implications for the
higher education sector in Australia.
academic publications: The findings and outcomes of this project will be presented in national and
international conferences and published in international journals.
References
AUQA. (2009, May). Setting and monitoring academic standards for Australian higher education: A discussion paper. Available
online: www.auqa.edu.au/qualityenhancement/academicstandards/
Boud, D. & Associates. (2010). Assessment 2020: Seven propositions for assessment reform in higher education. Sydney:
Australian Learning and Teaching Council.
Brown, R. (2010). Comparability of degree standards? Available online: www.hepi.ac.uk.
Brown, S., & Knight, P. (1994). Assessing learners in higher education. London: Kogan Page.
DEST. (2002). Striving for quality: Learning, teaching and scholarship. Canberra: AGPS.
ENQA. (2009). Standards and guidelines for quality assurance. Available online: www.enqa.eu/pubs.lasso
Fullan, M., & Scott, G. (2009). Turnaround leadership for higher education. San Francisco: Jossey Bass.
Feldt, L., & Brennan, R. (1989). Reliability. In R. Linn (Ed.). Educational measurement (3rd ed.). (pp.105-146). New York: Macmillan.
Harlen, W. (2005). Trusting teachers’ judgement: research evidence of the reliability and validity of teachers’ assessment used for
summative purposes. Research Papers in Education, 20(3), 245-270.
Klenowski, V. (2007). Evaluation of the effectiveness of the consensus-based standards validation process, Brisbane Qld: DETA.
Knight, P. (2006). The local practices of assessment. Assessment and Evaluation in Higher Education, 31(4), 435-452.
Kreber, C. (Ed.). (2009). The university and its disciplines. New York: Routledge.
Lillis, D. (2007, May). Steered by engagement. In L. Bollaert et al. Embedding quality culture in higher education. EUA.
Maxwell, G. (2002). Moderation of teacher judgements in student assessment. Brisbane: Queensland School Curriculum Council.
Messick, S. (1989). Validity. In R. Linn (Ed.). Educational measurement (3rd ed.). (pp.13-103). New York: Macmillan.
Norcini, J. & Shea, J. (1997). The credibility and comparability of standards. Applied Measurement in Education, 10(1), 39-59.
Nulty, D. (forthcoming). Peer and self-assessment in the first year of university. Assessment and Evaluation in Higher Education.
Radbourne, J., & Nulty, D. (2002). Immersing a faculty in assessment: A case study of an organisational Culture Shift. In Assessing
Learning in Australian Universities, See: http://www.cshe.unimelb.edu.au/assessinglearning/
Rosovsky, H., & Hartley, M. (2002). Evaluation and the academy. Cambridge, MA: AAAS.
Sadler, R. (2009). Grade integrity and the representation of academic achievement. Studies in Higher Education, 34(7), 807-826.
Sadler, R. (2010). Operational guidelines for assuring academic achievement standards at Griffith. [internal document]
Scott, G., Coates, H., & Anderson, M. (2008). Learning leaders in times of change. Camberwell: ACER.
Smith,D. (1992). Validity of faculty judgements of student performance. The Journal of Higher Education, 63(3), 329-340.
Trowler, P. (2009). Beyond epistemological essentialism: Academic tribes in the 21st century. In C. Kreber (Ed.) (pp. 181-196). The
university and its disciplines. New York: Routledge.
Yorke, M. (2008). Grading student achievement in higher education. London: Routledge.
Yorke, M. (2009). Assessment for career and citizenship. In C. Kreber (Ed.) (pp. 221-230). The university and its disciplines. New
York: Routledge.
9