March 2010
User Guide
Online Peer Review System
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
1
Contents
1. Introduction
page 2
2. Using This Guide
page 3
3. Overview
page 5
4. Getting Started
page 6
5. Filling out a Peer Review
page 7
6. Creating a Review
page 9
7. Content Creation, Criteria and Banks
page 14
8. Administrator Guides and Controls
page 21
9. References
page 23
10. Appendix
page 24
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
1
1.0 Introduction
While there is little debate about the importance of quality teaching (Murphy, MacLare and
Flynn, 2009), there are varying opinions about what constitutes ‘quality teaching’ and even
less agreement about appropriate strategies for evaluating teaching quality (Ingvarson and
Rowe, 2007; Kohut, Burnap and Yon, 2007).
One of the critical aspects of peer review is that it seeks to achieve scholarly outcomes in
course development through processes which are, of themselves, scholarly. Such an
approach to scholarship is based on an understanding of the communal basis of scholarly
activity that fosters the development of the individual academic as well as the profession
through scholarly activity that is open to critique. Peer review serves both a formative and
summative function and peer review processes should aim to achieve multiple outcomes
including:
 quality assurance of teaching;
 identification of areas requiring improvement;
 preparing academic staff for internal and/or external reviews;
 summative review for promotion or awards and
 assuring the quality of teaching and learning outcomes (Gosling, 2005).
This has been achieved in this online peer review system through the design and
development of checklists of agreed good practice incorporated into a comprehensive,
integrated online peer review system designed to engage academic staff course
development through the kind of reflective processes advocated by Schon (1987, 1991).
As described by Wood and Friedel (2008, 2009), the principles underlying the development
of this online peer review system were as follows:
 The criteria for the standards of development are gathered from the full range of
relevant academic literature surrounding teaching and learning. This affirms the work
of academics in the area in a highly practical form, which is accessible to a broadlybased audience.
 The approach locates responsibility for the quality of learning and teaching with the
academic staff responsible. Staff can use the items to guide the development or
redevelopment of their own courses through reflective processes.
 Academics are empowered to construct their own tailored evaluation checklists and
to contribute to developing the database of criteria.
 The online peer review system and its associated website provide an opportunity for
just-in-time academic staff development by providing the accepted standards,
information about how to meet these and exemplars contributed by academics
themselves.
 The online peer review system is flexible and adaptable to accommodate changing
technologies and priorities.
 The supporting website is designed to provide a model of best practice, utilises latest
Web 2.0 and database technologies, and complies with W3C Web Content
Accessibility Guidelines.
 The online peer review system has been developed as an open source platform, to
enable it to be adapted by other institutions to suit their learning and teaching and
technical contexts.
The online peer review system is designed to incorporate dynamic functionality enabling
academics to construct their own customised peer review templates. This flexibility of the
online peer review system means that it is possible to create customised review templates
for any aspect of teaching and learning. Thus the online peer review system has been
trialled on the curriculum of online courses, delivery of face to face courses and on individual
teaching resources such as a self-directed student learning package.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
2
Initial feedback from BETA testing suggested that even though academics appreciated the
opportunity to construct reviews for a variety of purposes, they found the complexity of the
system overwhelming. Thus, the system was redesigned to incorporate banks of criteria
focusing on different areas of the scholarship of learning and teaching as built-in templates,
while also retaining the option for academics to custom design their own review templates.
The online peer review system has also been adapted as a learning scaffold for students
undertaking courses in which they are required to make evaluative judgments about their
work and the work of their peers (Wood, 2009). Trials have demonstrated the potential of the
online peer review system as a scaffold for learners, providing structured opportunities for
reflection on their work and formative feedback prior to summative assessment. The
preliminary findings also show the potential of the online peer review system in facilitating
reflection in action by the teacher informed by students’ self-reviews and by monitoring how
students respond to and act on that feedback. Such reflection enables the teacher to adjust
their feedback and also the structure and detail provided in the review and assessment
templates for subsequent assignments.
This guide outlines the online peer review system and its features. It explains how to create a
review and provides detailed tutorials and helpful information on processes. The structure of
the guide is based on the steps taken when creating a review. It is recommended that you
start at the beginning of the guide and work your way through. An online version of this guide
can be found at http://peerreview.unisa.edu.au/wiki/index.php/Instructions
2.0 Using This Guide
Terminology
There are varying opinions about the role and purposes of peer review. Similarly, there are a
range of different terms used to describe ‘peer review’ processes within the Australian Higher
Education sector. For example, university guidelines for academic promotion may use the
term ‘peer evaluation’, while performance management process (PMP) documentation
makes use of terms such as ‘peer assessment’. On the other hand, university policies
dealing with the evaluation of teaching may refer to terms such as ‘peer review’, while
academic development resources that support the improvement of teaching speak of ‘peer
observation’. Often, specific terms are used to denote ‘what’ teaching is being reviewed,
‘how’ that teaching is being reviewed or by ‘whom’ the teaching is being reviewed.
The varied use of terms relating to ‘peer review’ is further complicated by applying formative
or summative aspects to the process. Taking a formative or summative approach to peer
review indicates the ‘purpose’ or intended use of the feedback generated by the peer review
process. For example, formative peer review relates to seeking feedback and constructive
suggestions from peers that will support the development of teaching or improvement of
teaching materials. Alternately it could also mean creating an atmosphere of mutual support
and feedback among a group of academics as part of an ongoing process of reflection on
teaching practice or course content. Summative feedback on the other hand, is used when
academics wish to collect evidence of their teaching for probation, or to support applications
for promotion or a teaching award. However, the collection of this evidence could equally be
used to support a teacher in formatively reflecting on and improving their teaching or
teaching materials.
The online peer review system does not distinguish among the various terms used in relation
to peer review, nor are any assumptions made about the purpose for the collection of
evidence or feedback. The purpose for using evidence produced via this system is up to you.
However, it is important that you:
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
3



identify the intended use of the resulting evidence/feedback;
reflect on the different types of evidence you may want to collect;
develop a plan for using the online peer review tool.
The main functions of the online peer review system are to:
 provide the banks of criteria that act as prompts for what could be peer reviewed –
as an individual academic you still have the key role of identifying the items you
would like to have peer reviewed. You can then use your selected criteria as a
mechanism for discussing with your chosen peer reviewer/s how the peer review
process will take effect;
 store the collected evidence/feedback in a secure database so that it may be used
for the purpose you intend.
Constructing a Review
As a new author of a peer review, you need to consider how you will use the peer review
system. For example you might want to use the system:
 for self-review and reflection as you design or redevelop a course;
 in a formative way to obtain constructive feedback from your peers to help you make
changes to a course;
 to reflect on peer feedback prior to using the system for summative purposes; for
example to support applications for awards or promotion
 to gather feedback from students about your teaching for formative or summative
purposes;
 as an assessment tool that students can use to evaluate their own or their peers’
performance in an assignment against assessment criteria as either formative
feedback or for summative assessment;
 for peer review of journal or conference papers providing a means for reviewers to
review manuscripts against a set of specified criteria.
When creating your peer review you have the option to select different response types for
the criteria you choose (if using the wizard, only one response type can be chosen for all
criteria initially, but you can manually change the response types at any time during the
wizard process). These response types include Boolean, combo boxes, free form text,
frequency scales, multiple choice and multiple option, phrase completion and comments. In
deciding which of the response types is most appropriate for your review, you need to
consider whether you want to receive feedback/evidence in the form of:
 qualitative feedback such as constructive suggestions via free form text
fields/comments boxes, which will support the development of teaching, the
improvement of teaching materials or the development of a culture supportive of
improving teaching;
 quantitative feedback such as agreement ratings, frequency scales and so on, which
will support applications for promotion or teaching awards;
 Both qualitative and quantitative feedback which can be used for either formative or
summative peer review purposes.
The online peer review system can assist you in each of these approaches and the following
sections provide you with the information you need to construct a review that meets your
objectives.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
4
3.0 Overview
An Introduction to the Peer Review System
The online peer review system allows academic staff to have their courses and curricula
critiqued through peer review processes. Either complete courses or specified elements of
the course may be selected for review. The online peer review system can be used for selfevaluation, reflection, professional development or peer review. The review may be
customised to focus on teaching and content provided in any setting. Authors of the review
are easily able to create custom reviews that are structured with a wide range of commonly
used academic criteria, as well as creating and specifying their own criteria. Peers can then
be invited to participate in a review by addressing these criteria with their own responses and
comments, through the use of easy web-based forms. When the data is collected, it can be
exported into a spreadsheet for ease of analysis and presentation. This data can then be
used to evaluate the course and identify areas for possible improvement.
In summary, the online peer review system and the supporting website provides highly
focused, just in time information to enhance the teaching and learning knowledge and
expertise of academic staff. It promotes a scholarly approach to learning and teaching
because it sustains reflective practice, and provides a structured and informed approach to
peer review. Further, the online peer review system provides a means by which staff can
have their work publicly affirmed and use this as evidence to support their applications for
promotion and awards
Hierarchy Explanation
A ‘Peer Review’ is made up of several ‘Banks’. These banks in turn contain ‘Criteria’, which
are specific items to be included in the review.
An example is as follows. You may want to review a course for instructional content as well
as the availability of support. You may also want to evaluate the use of media or evaluate the
assessment processes. Banks are available for each of these areas. The banks are
populated with specific criteria that you may select to use. For example, in the assessment
bank you would find criteria such as: ‘constructive feedback was provided on all work’ or
‘assignments are returned within the specified timeframe’. You may also add your own
criteria if the issue you wish to address is not already included in the available criteria.
The common course evaluation criteria you may wish to use are supplied with the system.
These criteria are drawn from the literature and from discussion with academic staff from
partner higher education institutions.
The system also gives you the ability to create and structure your own criteria specific to your
courses. Each criterion has several different response type options. You can use traditional
Likert scales, yes/no answers, short text answers as well as enabling comments on each in
order to get the most helpful responses from reviewers.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
5
Categories of Users
The online peer review system uses a flexible and multi-tiered security system that provides
access to specific modules based on the logged in user's access level. There are several
categories of user accounts that can be used within the online peer review system they are
as follows:
Author: Someone who creates and structures reviews, edits and managing them. An
‘Author’ also takes the responsibility of inviting candidates to participate in the peer review
and exporting the results.
User: Someone who can receive invitations to partake in review of course systems and
websites, by filling out the online peer review forms. They can also be invited to become
authors on a project by people in the Author category.
Administrator and Super Administrator: Reserved for those who set up the system or who
have been invited to be administrators; these accounts have full control over the online peer
review system and data.
Glossary of Commonly Used Terms
Review: An online evaluation of a course, created through web forms and buttons.
Bank: A structured set of criteria addressing an area to be reviewed.
Criteria/Criterion: Specific criteria relating to teaching or course materials that are contained
within banks for review.
User/Author/Administrators: User categories of the system (see categories of users above).
Publish/Unpublish: Publishing a review allows it to be accessible to all your assigned users
to complete the review.
Exporting Results: Exporting results refers to exporting the end results of a review into an
Excel spreadsheet.
Response type/Scale: The type of response a user will make to a question in a peer review.
4.0 Getting Started: Setting Up a Peer Reviewer Account
Obtaining an account
In order to use the peer review system you will first need access to the project webpage at
http://www.unisanet.unisa.edu.au/peerreview/default.asp
Once you have requested access to the system the administrator will be notified and the
account manually reviewed. Anyone requesting access from an approved educational
institution will be authorised and their account made active.
Logging in
Once you have a login, you will be greeted by a login screen. Enter your username and
password as detailed to you in the email and click ‘Login’ to proceed.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
6
You will now have access to all the different features of the peer review system. Be sure to
remember to change your password and user settings in the User settings section.
If you have forgotten your password, click the link that says “Forgot my password’ on the
login page and enter your email address. Your password will be reset and a new password
sent to that email address.
5.0 Filling Out a Peer Review
Introduction
This section outlines how to contribute your own personal responses to a pre-existing review.
You may have been invited to complete the review via email or URL. In order to respond,
you will need a ‘User Account’, which is detailed in the ‘Getting Started’ section of these
instructions.
Once you have an account and you are logged in, click the navigation link 'My Reviews'
section. You should see a list of reviews that you are permitted to complete (Figure 1). If you
cannot see any reviews, or cannot see the specific review you have been asked to fill out,
the author or co-author of the review (usually the person who invited you) needs to invite you
to the review with the ‘Invite Users’ function (for more information see the ‘Selected Review
Page’ section of the guide.
Take note of the review availability status and the dates specified, because you will have to
fill out the review within these dates. Click the 'Load Review' button to open the review page.
Figure 1: ‘My Reviews’ interface with example review
Responding to Questions
Once on the review page, you can see a table that provides details of the review question
banks, including the total number of criteria in each bank, how many criteria you have
answered, how many you have remaining and the total percentage completed (Figure 2).
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
7
Figure 2: ‘Selected Review’ interface with example question banks
To fill out a criteria bank, click the
green arrow icon next to the question bank. This will
open the review criteria (answer) page at the point where specified ((Figure 3). It is usually
easiest to start at the beginning and work your way through each question. However, the
system will automatically track your progress so you can start at any point that suits you.
Figure 3: Review criteria (answer) page with example criteria
Each bank will have different criteria each with different response types. These can be
completed with simple checkboxes, forms and tick boxes. For more detailed explanation of
each response type, see the ‘Explanation of Response Types (Scale Type)’ section in this
guide.
To reset all your entered values on the form, click the 'Revert' button at the bottom of the
form. Once you are happy with your responses, click 'Save’ to proceed to the next question
bank. This will continue to the end of the review. You can save an incomplete review and
return later to complete the remaining review providing you do so prior to the date the review
period is scheduled to end. At the end of this process you will be returned to the review
table, which provides a summary of your responses and a confirmation message that
indicates you have successfully completed the review.
If at any time before the review is closed you wish to change your response, you can do so
by clicking the
green arrow icon next to the question bank, and changing your input in the
same fashion as you added them.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
8
Toolbar Icons and Functions
As you are using the online peer review system you will notice several icons located on the
top of each page, along with a summary of the selected review. The purpose and
functionality of these icons are detailed below:
Review Summary: Clicking this image at any time within the review will take you back
to the table of contents.
Back and Forward (reversed and red): These can be used to navigate back and
forward through a review. Useful if you wish to skip questions to return to later.
Unload/Deselect Review: If you wish to contribute or modify another review, you can
use this icon to 'deselect' this review and take you back to the reviews page to choose
another.
Information: Some criteria may have additional information, containing useful data or
links that may provide further clarification to their purpose or function.
6.0 Creating a Review
Introduction
To create a review, first you will require ‘Author’ privileges. Only ‘Author’, ‘Administration’ and
‘Super Administration’ accounts have this level of permission. If you do not already have an
‘Author’ account, you can request one as detailed in the ‘Getting Started’ section. See an
explanation of user categories in the ‘Overview’ section.
Figure 4: ‘Review Management’ interface
As an ‘Author’, you should have access to the ‘Review Management’ link on the left hand
navigation bar. Click the link to travel to the ‘Review Management’ page (Figure 4).
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
9
Review Management
The following options are available to ‘Author’ and ‘Administrator’ users only (see the
‘Overview’ section for more information on user categories).



Create New Review - Allows you to create a new peer review.
Manage My Reviews - Provides a list of reviews you have created or have been
assigned to you by another author. Allows you to swap reviews and modify content.
Use Selected Review - If you already have a review selected, this will take you to the
management page of that review.
The following options are available to ‘Administrator’ users.


Create New Review for User - Allows an administrator to create a review and assign
it to another users account.
Supervise Reviews - Allows an administrator to modify and update other user review
formats.
Create New Review
The ‘Create New Review’ function will lead you to an online form (Figure 5). When
completing a review for personal or limited use, you may leave these fields blank. However,
if you are creating a review that will be accessible to other users, please try to make it as
descriptive as possible.
A description of each field is outlined below:

Review Name: The name of the subject or course to be reviewed.

Review Description: A short description of the course and review, what the review
intends to achieve.

Category: Which category the review falls under.

Access URL: If the review subject is a web based artifact, or can be accessed via
the internet, include the URL here.

Open From and Until Dates: Allows a specified time period in which the reviews are
available for gathering data.

Is the Review available?: If you would like the review to be accessible by all users of
the peer review system, select 'yes' from the drop down menu.

Assign to User (Admin only): This feature allows you to create a review for a specific
user, so it will appear in their name as their creation. (See administrator controls)

Results Format: Allows you to specify whether results are reported anonymously or
in an 'identifiable' mode where individual Reviewers' results can be viewed

Use the Wizard: The Wizard is a simple way to create a review built. Once you have
created the review using the Wizard, you can go into your review and make more
advanced adjustments as required.

The ‘Revert’ button will remove all the changes you have made to the form and set
them to default.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
10

The ‘Create Review’ button will save your changes and return you to the ‘Selected
Review’ page.
Figure 5: ‘Create New Review’ form
Manage My Reviews
The ‘Manage My Reviews’ interface shows each review you have created or have been
assigned to as an author (Figure 6). By clicking the A-Z icons, you are able to sort the data
results in an ascending or descending order.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
11
Figure 6: 'Manage My Reviews' interface
You will notice each review has three icons next to its title:
Spanner - Allows you to change the name, opening dates and descriptions of the
review.
Tick - 'Selects' this review as your currently selected review, taking you to its Selected
Review page.
Chart - Allows the summary results of a published review to be downloaded to your
hard drive (Microsoft Excel) format. If this is greyed out, it means the review is unpublished.
In addition to these, at the top of the page you will notice the Selected Review Toolbar,
which gives you greater options in modifying your reviews as detailed in the next section.
Selected Review Page
The ‘Selected Review’ page is accessed via the ‘Review Management’ page and shows an
outline of the currently selected review and any question banks that have been assigned to
it, along with their individual criteria. From here you can add, delete, reorganise and edit
question banks and criteria for use in your review. If the page looks empty, this may be
because there are no question banks added to the review. Use the
or
‘Create New Bank’
‘Add Existing Bank’ functions to start populating the peer review.
The ‘Selected Review’ toolbar sits above the review table and is used to create new review
components as well as assign rights to different users and publish completed reviews. Note
that if an icon is greyed out, this is because you cannot use that function because the review
is published or unpublished.
Selected Review Toolbar Icons:
View Selected Review - Will take you to the Selected Review page for the current
review, if you are not there already.
Wizard - Launches the Wizard. The Wizard is a simple method for creating a review.
Create new Bank - Creates a new Question Bank.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
12
Add existing bank - Allows you to pick and choose criteria banks from the pre-defined
banks created by the peer review project team as well as other users of the system.
Manage Selected Review - Go to the Manage Review page for the selected review.
Set Authors - Allows you to assign other peer review users as authors of your review,
by adding their email. For more information on the user roles, see the Overview section.
Show all - This toggle will turn the visibility of non-usable reviews/banks/criteria on and
off. Default is off.
Preview Review - Will allow you to step through your review as it would be seen by
users, without recording results.
Publish Review - Once a review is published, users can be invited to complete the
review. The ‘Publish Review’ option should only be selected once the review is complete and
you are happy with it.
Unpublish Review - Unpublish if you wish to make changes and updates to your
review.
Invite Users - This will allow you to invite people by email address to participate as
users in your peer review.
View Results - Will show you a summary of the responses and data collected from
users in your review.
Export Results - Will download a saved copy of the review results in XLS format (Excel
Spreadsheet). The author can export the entire review or the responses within an individual
bank to an Excel spreadsheet. Once the review data is exported the author and any other
authorised users can manipulate, analyse and graph the results to suit their specific
requirements.
View Results by Reviewer - Will show you a summary of the responses and data
collected from individual reviewers.
Export Results by Reviewer - Will download a saved copy of the review results for
individual reviewers in XLS format (Excel Spreadsheet).
Deselect Review - Will take you back to the review select screen.
Once the review is populated you will notice that there are additional icons inside the
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
13
question banks and criteria listed. These allow you to create and edit individual banks and
criteria, customising their use to suit your own review.
Edit Criterion or Bank - Allows you to modify the contents of a question bank or criteria.
Create New Criterion - This will allow you to create a new criterion in the current
question bank it is located in. See the ‘Creating New Criteria’ section for more details.
Move Up - Will move the selected criteria up one position in the list of banks. Useful for
structuring questions in the order they should be answered.
Move Down - As above, but moving the criteria down.
Options - Some criteria have editable options, such as several possible answer choices
that need defining. This will allow you to edit and modify them.
Properties - Some criteria have configurable properties, such as labels in the ‘Phrase
Completion’ response type. This will allow you to modify and edit them.
Populating your review with content
Now that you have created a review, you can fill it with new or existing content. This comes
in the forms of ‘Banks’ and their criteria.
See the ‘Content Creation, Criteria and Banks’ for information on how to start adding content
to your review.
7.0 Content Creation, Criteria and Banks
Introduction
‘Banks’ are used to organise criteria into sets and subsets, so you can neatly organise your
criteria under different headings. For example, if you had a review that reviewed a course on
two different aspects, you could split your criteria into separate banks in order to make
navigation easier for you and user(s) using your reviews.
You also have the option of accessing a set of pre-defined question banks that contain
common course review criteria that have been drawn from the literature and other ALTC
funded peer review projects (see Appendix A for list of all included banks/criteria and the
associated references). Approved users who have administrator access will be continuing to
populate the system with new criteria so that other authors will be able to utilise the added
criteria as the system evolves.
The ability to create custom banks of criteria enables authors to focus on particular areas of
concern within a course review. Authors can also create their own banks and criteria, which
are then flagged as their own user created banks (Figure 7). The author can choose to share
a bank with other authors, or recommend that their bank be placed in the default,
administrator created bank group. This collaborative feature of banks and criteria provides
the online peer review system with a flexible means to extend and refine criteria in the
system.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
14
Populating a Review with Existing Banks
Once you have created a review, you may add one of the existing question banks via the
‘Add Existing Bank’ icon on the ‘Selected Review’ page.
Figure 7: 'Add Existing Bank to Review' form
You will have two options, choosing the ‘Source Bank’, and a ‘Parent Bank’. Most of the time
you will only choose the ‘Source Bank’, but in special instances you may want your bank
category to have multiple sub-categories. This can be accomplished with the ‘Parent Bank’
functionality. An example where this would be appropriate would be if you created a bank
called ‘Digital Media’ and then assigned to that bank child banks such as ‘Images’, ‘Audio’
and ‘Video’. Each of the child banks would contain unique criteria, but still be classified under
the ‘Digital Media’ parent bank.
Once a source bank is chosen, click ‘Add Bank’ to add it to your review.
If you would like to further customise these preset banks, you can edit their content by
clicking the spanner icon in the ‘Selected Review’ page, provided the review is unpublished.
Note, you can add to, delete from and edit these criteria in any way without affecting the
original criterion. Each time you use a copy of the shared criteria it will create a unique copy
for you, so that you don't overwrite the originals.
Creating New Banks
To create a new question bank, click the
page.
‘Add New Bank’ icon in the ‘Selected Review’
Figure 8: 'Create New Bank for Review'
Give your question bank a name (this is commonly defined by the category of criteria it will
contain) and then specify whether the bank will be ‘active’. Modifying the availability of a
bank will determine whether a bank is available to other users and the administration system
(Figure 8).
You will have two options, choosing the ‘Source Bank’, and a ‘Parent Bank’. Most of the time
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
15
you will only choose the ‘Source Bank’, but in special instances you may want your bank
category to have multiple sub-categories. This can be accomplished with the ‘Parent Bank’
functionality.
Click the ‘Create Bank’ button to add the bank to your review and take you to the ‘Selected
Review’ page. You may return to this page at anytime to edit and update the bank details,
(provided the review is unpublished) by clicking the spanner icon.
Creating New Criteria
To create a new criterion, use the
‘Create New Criterion’ icon located in the ‘Selected
Review’ toolbar in the ‘Review Management’ page. If you cannot see this icon or any
question banks or criteria listed, this is probably because your review doesn't yet have any
‘Banks’. See the ‘Populating a Review with Existing Banks’ or ‘Creating New Banks’ sections
for more information.
Alternatively if you have ‘Administrator’ access, you can manually define criteria through the
‘Administration’ section.
Figure 9: 'Create Criterion' form
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
16
The ‘Create Criterion’ form shown in Figure 9 above allows you to create a new criterion and
specify the ‘Bank’ with which it will be associated. If a suitable bank does not exist in your
review, you will need to create one through the ‘Create Banks for review’ function. After you
give the criterion a name, you need to provide a description of that criterion in the ‘Full
Criterion Text’ field. The ‘More Information Text’ field is optional, but valuable for the
reviewer, as this field provides you with the option to provide an extended description,
justification for the criterion drawing on the literature as well as links to exemplars.
Some criteria have already been populated with ‘More Information Text’. This should still be
customised to suit your specific review. In the following example, the author has customised
the ‘More Information Text’ field by highlighting the ways in which course under review has
been designed to achieve the criterion. Figure 10 shows the criterion ‘The student
assessment workload is reasonable’ as well as the associated ‘More Information Text’
revealed by clicking on the ‘I’ button. The additional information contained in this field
provides the rationale behind the criterion drawn from resources available via the Centre for
the Study of Higher Education at the University of Melbourne. The academic staff member
has customised this field to explain how the modules in the course under review are staged
to provide adequate time for learners to synthesise the module material and practice the
related applied skills as they prepare for each assignment. This additional customised
information helps the reviewer to consider the academic staff member’s perspectives and to
provide more focused feedback. The process of completing the ‘More Information Text’ field
similarly helps the academic staff member whose course is being reviewed to reflect on their
own teaching and the design of their course materials.
Figure 10: Customised ‘More Information Text’ field designed to facilitate reflection and to
provide contextual information relevant to the teaching or course under review
Reviewers completing your peer review will be able to respond to the questions in different
ways according to the response type you have set. For example, you can have Yes/No style
checkboxes, or multiple choice answers and traditional Likert scales. For a full list of the
available response types, see the ‘Explanation of Response Types’ section below. By ticking
the enable comments tickbox, users will have the option of leaving feedback on the criteria
as short text answers.
Click ‘Create Criterion’ to add the criterion to your currently selected bank. You will then be
returned to the ‘Selected Review’ page which will show your new criteria inside a question
bank. To further modify and arrange this data, see the ‘Editing and Modifying Existing
Criteria’ section.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
17
Explanation of Response Types (Scale Type)
There are many response types available when creating criteria. The preset response types
function as follows:
Boolean: Allows possible answers ‘Yes’ and ‘No’.
Boolean Neutral: Allows answers ‘Yes’, ‘No’ and ‘Neutral’.
Combo: Allows you to define a multiple answer drop down combo box. (See the ‘Editing and
Modifying Existing Criteria’ section for details on how to configure).
Freeform Text: There is no scale, only offering the user to write a response. This also can be
achieved with other scales by using the 'Comments' function.
Frequency Scale A: Allows answers ‘Never’, ‘Sometimes’ and ‘Always’.
Frequency Scale B: Allows answers ‘Never’, ‘Sometimes’ and ‘Frequently’.
Likert Scale: A single selection scale of ‘Strongly Disagree’, ‘Disagree’, ‘Neutral’, ‘Agree’,
‘Strongly Agree’.
Multiple Choice: Allows the user to select a single option from a list of criteria.
Multiple Option: Allows the user to select multiple options from a list of criteria.
Phrase Completion: Allows the user to define a custom scale, from 1 to 10, naming the
minimum to maximum.
Yes / No: User can select 'YES' or 'NO' as possible answers.
Some of these response types require further customisation of names and options. See the
‘Editing and Modifying Existing Criteria’ section for more details.
Editing and Modifying Existing Criteria
You are able to edit and update criteria in your review through the ‘Selected Review’ page as
long as the review is unpublished.
Note, you can add to, delete from and edit these criteria in any way without affecting the
original criterion (Figure 11). Each time you use a copy of the shared criteria it will create a
unique copy for you, so that you don't overwrite the originals.
Figure 10: Criteria with modification icons
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
18
On the ‘Selected Review’ page, you will notice that next to each criterion, there should be
several icons. Depending on whether a review is published, some of these options may not
be available. They function as follows:
Edit Criterion or Bank - Allows you to modify the contents of a Question Bank or
Criteria.
Create New Criterion - This will allow you to create a new criterion in the current
question bank it is located in. (See the ‘Creating New Criteria’ section for more details).
Move Up - Will move the selected criteria up one position in the list of banks. Useful for
structuring questions in the order they should be answered.
Move Down - As above, but moving the criteria down.
Options - Some criteria have editable options, such as several possible answer choices
that need defining. This will allow you to edit and modify them.
Properties - Some criteria have configurable properties, such as labels in the Phrase
Completion response type. This will allow you to modify and edit them.
Author-Created Data
The ‘Author-Created Data’ option allows authors to create their own banks, criteria and
criteria option templates, which can be re-used across multiple reviews (Figure 12). This is in
contrast to banks and criteria created in a 'Selected Review', which can only be used in that
review. Use ‘Author-Created Data’ to create your own banks and criteria if you expect to use
the same criteria in future reviews. ‘Author-Created Data’, banks and criteria are created and
modified in the way previously described in this section of the guide.
Figure 11: ‘Author-Created Data’ interface
You can also create and manage ‘Criteria Option Templates’. These are reusable sets of
response options that can be assigned to ‘Multiple Option’ and ‘Multiple Choice’ criteria.
Click ‘Create Criteria Option Template’ and give the criteria option template a name (Figure
13). Then click ‘Create Author-Created Criteria Option Template’ to create the template.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
19
Figure 123: 'Author-Created Criteria Option Template'
Now enter the options you require to be included in the template (Figure 14).
Figure 13: ‘Author-Created Option Templates' - Managing
Once you have created a ‘Criteria Option Template’ you can assign it to any ‘Multiple Option’
or ‘Multiple Choice’ criteria by clicking the
‘Options’ icon as shown in Figure 14 below.
Figure 14: Managing 'Author-Created Criteria' interface
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
20
Now choose the ‘Criteria Options Template’ that you would like to assign to the criteria
Figure 16).
Figure 156: Assigning criteria to a 'Criteria Options Template'
Click ‘Manage Criteria Option Templates’ to view and edit your ‘Author-Created Criteria
Option Template’.
8.0 Administrator Guides and Controls
Introduction
As an ‘Administrator’ you will have access to a wider variety of features than regular users.
Administrator access is indicated by the presence of a new section, ‘Administration’ on the
side navigation toolbar (Figure 17). There are a number of management tools which are
explained in the following sections.
Figure 16: 'Administration'
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
21
Banks and Criteria
One feature ‘Administrator’ users have in addition to the features available to regular users is
the ability to add new banks and criteria without attaching them to any specific review. This
may be useful if you are debugging or wishing to create generic criteria sets for use in
several courses.
In addition, administrators also have the ability to manage all criteria, banks, response types
and categories within the system. Modifying and updating these criteria will see their
changes reflected across all reviews.
Managing Response Types and Categories
‘Administrators’ have the ability to rename, reword and restructure response types to suit
their liking. These changes will be reflected across all reviews. For example, if you would
rather Likert scale be called an ‘Agreement Scale’, this can be modified here.
‘Administrators’ also have the ability to add and create new ‘Categories’ for use in defining
types of reviews.
Create New Review for User and Supervise Reviews
In addition to the regular ‘Review Management’ page that authors see, ‘Administrators’ can
see two extra options on this page, ‘Create New Review for User’ and ‘Supervise Reviews’
(Figure 18).
‘Create New Review’ is identical to the ‘Create New Review’ section, with the added option
to 'Assign to User.' This combo box will allow you to add someone else to act as the ‘Author’
of the review.
‘Supervise Reviews’ is also identical to the ‘Manage My Reviews’ section, with the added
functionality of allowing ‘Administrators’ to view and manage all reviews in the system,
regardless of who is set as the ‘Author’.
Figure 178: 'Review Management' page showing 'Administrator' options
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
22
9.0 References
Gosling, D. (1996). What do UK educational units do? International Journal for Academic
Development, 1(1), 75- 83.
Ingvarson and Rowe (2007). Conceptualising and evaluating teacher quality: Substantive
and methodological issues Australian Council for Educational Research. Paper
presented at the Economics of Teacher Quality Conference, Australian National
University, 5 February 2007.
Kohut, G. F., Burnap, C., and Yon, M. G. (2007). Peer observation of teaching.
College Teaching, 55, 19-25.
Murphy,T., McLaren I. and Flynn, S. (2009) Toward a summative system for the assessment
of teaching quality in higher education International Journal of teaching and Learning
in Higher Education, 20, 226-236.
Schön, D. A. (1991). The reflective practitioner: How professionals think in action (Paperback
ed.). Cambridge: Ashgate.
Schön, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass Inc.
Wood, D. (2009). A scaffolded approach to developing students’ skills and confidence to
participate in self and peer assessment. Proceedings of the ATN Assessment
Conference, Melbourne.
Wood, D. and Friedel, M. (2009). Peer review of learning and teaching: Harnessing collective
intelligence to address emerging challenges. Australian Journal of Educational
Technology, 25(1). Retrieved 21 February 2009 from
http://www.ascilite.org.au/ajet/ajet25/wood.pdf
Wood, D. and Friedel, M. (2008). Peer review of online learning and teaching: new
technologies—new challenges, Proceedings of ASCILITE 2008: Hello! Where are you
in the landscape of educational technology? Deakin University, Melbourne.
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
23
Appendix A: Built-in banks and criteria incorporated into
the online peer review system
CORE PRINCIPLES
ACTIVE STUDENT ENGAGEMENT IN LEARNING

A supportive, non-threatening teaching/learning environment is fostered

Students are encouraged to express views and ask questions- time and
opportunity is allocated for this to occur

Student engagement is encouraged through the use of suitable questions

Immediate and constructive feedback is provided where appropriate

Enthusiasm is demonstrated for teaching and learning
BUILDING ON STUDENTS’ PRIOR KNOWLEDGE AND EXPERIENCE

Measures are taken to determine students’ prior knowledge and
understanding

Students’ current knowledge and understanding is built upon

Where appropriate, student contributions are used and built upon
CATERING FOR STUDENT DIVERSITY

Appreciation is shown for the different levels of knowledge and understanding
in a group

Different learning needs and styles within the group are addressed

A focus is placed on building confidence, enthusiasm and intrinsic motivation

Students are encouraged to take responsibility for their own learning

A balance of discursive interactive and didactic strategies are employed to
accommodate different student needs

Teacher-directed strategies are implemented effectively where required

A balance is achieved between challenging and supporting students

Activities/tasks are designed to allow students of differing abilities to
participate/engage and demonstrate/enhance their learning

Examples or opportunities for discussion that cater for cultural diversity are
provided
ENCOURAGING STUDENTS TO DEVELOP/EXPAND THEIR CONCEPTUAL
UNDERSTANDING

Students are assisted to bridge the gap between their current conceptual
understanding and the next ‘level’

Students are assisted to become aware of what the next levels are

Individual student learning is stimulated, encouraging students to become selfdirected learners

Students are challenged intellectually

Students are encouraged to internalize or ‘construct’ their individual
conceptual understanding

Deep (intrinsic) rather than surface (extrinsic) approaches to learning are
encouraged

The teacher works cooperatively with students to help them enhance
understanding

The teacher demonstrates a thorough command of the subject matter
STUDENT AWARENESS OF KEY LEARNING OUTCOMES

Students are progressively made aware of key learning outcomes

A focus is placed on learning outcomes at key points in the presentation
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
24



Synthesis of key learning outcomes is emphasized towards the conclusion of
the session so that individual student follow-up work is well focused
Students are encouraged to accept responsibility for learning issues that
require follow-up and consolidation
Students are aware of the link between key learning outcomes and
assessment (formative and summative)
ACTIVE USE OF LINKS BETWEEN RESEARCH AND TEACHING

Links between research outcomes and learning are emphasised where
appropriate

Research links are used appropriately, given the level of student conceptual
development

Students’ awareness of what constitutes research is developed
APPROPRIATE USE OF EDUCATIONAL RESOURCES AND TECHNIQUES

Information Technology (IT) techniques are used effectively

An appropriate balance of Information Technology (IT) and other strategies
are employed

Available classroom resources are utilised effectively to support student
learning

Resources, materials and literature are utilised to support student learning

Specific educational strategies and techniques are employed in the design and
delivery of teaching sessions, to achieve key objectives
LOGICAL PRESENTATION OF MATERIAL

An early, brief structural overview of the session is provided

Material is presented in a coherent manner which allows students to
understand the development of the session

Adequate time is provided for review at key stages, including closure

Closure is established to assist students to draw together and understand
major issues and identify individual learning needs and short-comings
EVALUATION

Feedback is sought progressively during the session

Feedback is acted upon
References
Crisp, G. Sadler, R., Krause, K., Buckridge, M., Wills, S., Brown, C., McLean, J.,
Dalton, H., Lievre, K. & Brougham B. (2009) Peer review of teaching for
promotion applications, from the ALTC funded project Peer review of Teaching
for Promotion Purposes: a project to develop and implement a pilot program of
external peer review of teaching in four Australian universities. Retrieved 21
February 2010 from
http://www.adelaide.edu.au/clpd/peerreview/Internal_pr_protocol_Mar09.doc
FACE TO FACE TEACHING
PLANNING

Appropriate time is allocated to the content of the presentation

The mode of presentation is suited to course content

An appropriate venue is utilised

Use of technology is well planned and implemented

Key points are presented in appropriate sequence

Transitions between parts of the presentation are smooth
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
25


An appropriate amount of time is set aside for questions
Different teaching models are incorporated in lectures
CONTENT

An opening summary is utilised to present major points and conclusions in
order to assist students to organise their listening

Aims and objectives are stated in the opening summary

Key areas to be covered are outlined in the opening summary

Explanation of how content links with previous lectures and courses is
provided in the opening summary

Key terms are used to act as verbal subheadings and memory aids

Real-life examples are provided to illustrate key ideas

Analogies are utilised to make comparisons between the content of the
presentation and knowledge the students already have

A variety of visual media is utilised to enable students to see as well as hear
what is being said

A conclusion is utilised to reinforce the presentation

The conclusion of the presentation summarises key areas/points

The conclusion of the presentation links with the opening summary

The conclusion of the presentation refers to the next stages of topic/course

The conclusion of the presentation suggests ways for students to follow up on
the presentation

Lecture notes are provided and include key points from the lecture opening
summary, body and conclusion
PRESENTATION

The presentation starts and finishes on time

The presentation is paced appropriately

Eye contact is maintained with students

Communication occurs on a personal level, allowing the teacher to relate to
students

Tone of voice is varied appropriately

Enthusiasm for the topic is exhibited

Physical activity such as moving around the room and gesturing with hands is
utilised to hold the attention of students

The voice is projected or amplified so that those in the back of the room can
hear clearly

The use of slang or repetitive words, phrases or gestures is avoided

Humour is used to positive effect

Techniques to facilitate oral comprehension are utilised
GENERAL INTERACTION

Questions are asked of the entire group

Students’ names are used when asking and answering questions

All students are encouraged to participate equally in discussions (students are
prevented from dominating discussion and quiet students are encouraged to
engage with the group)

Disruptive students are dealt with appropriately

Positive reinforcement is provided when students ask questions, answer
questions or make comments

Questions and answers from students are repeated to ensure that all students
hear the discussion (particularly for large groups)

Questions are answered using a variety of methods
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
26

Students are aware when the teacher is available for consultation
EVALUATING LECTURES

Immediate feedback is actively sought from students

Staff actively reflects on lecture presentation
SMALL GROUP INTERACTION

Group members are introduced to one another

A welcoming and engaging learning environment is provided

Students are involved in setting ground rules for issues such as constructive
criticism, minimum participation etc

Student and tutor roles and expectations are discussed

Possible discussion topics are explored with group members

Discussion is monitored and kept ‘on-track’
References
Ideas for effective large-group learning and teaching, Learning & Teaching @
UNSW, University of New South Wales. Retrieved 21 February 2010 from
http://learningandteaching.unsw.edu.au/content/userDocs/large_group_ideas.
pdf
O’Neill (2003) Small group (including tutorials) & large group teaching, Centre for
Teaching and Learning, Good Practice in Teaching and Learning, University
College Dublin. Retrieved 21 February 2010 from
http://www.ucd.ie/teaching/printableDocs/Good%20Practices%20in%20T&L/s
mallGroup&LargeGroup.pdf
Race, P. et al (2007) Smarter Lectures, A compendium from ‘Making Teaching
Work’ (Phil Race and Ruth Pickford, 2007), ‘Making Learning Happen’ (Phil
Race, 2005) and ‘Lecturing: A Practical Guide (Sally Brown and Phil Race,
2002). Retrieved 21 February 2010 from http://phil-race.co.uk/wpcontent/plugins/download-monitor/download.php?id=43
Resources, Teaching large classes AUTC project. Retrieved 21 February 2010 from
http://www.tedi.uq.edu.au/largeclasses/popups/resource1.html
Sullivan, R. & McIntosh, N. (1996) Delivering effective lectures, JHPIEGO Strategy
Paper. Retrieved 21 February 2010 from
http://www.reproline.jhu.edu/english/6read/6training/lecture/delivering_lecture.
htm
ONLINE LEARNING AND TEACHING
CLARITY OF EXPLANATIONS

The approximate effort required (hours of study) in undertaking each module is
stated

The course content is appropriate to the learning outcomes

The overall purpose of the course is clearly stated

The approximate effort required (hours of study) in undertaking various
aspects of the course reflects the objectives

The learning outcomes include reference to Graduate attributes

Objectives or learning outcomes are clearly stated and achievable for each
section or module

It is clear at the beginning of each module what learning resources the student
will need to complete the module (e.g. library books, web documents,
materials for experiments)

Learning modules include an overview of the content to be covered and the
processes by which it will be achieved
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
27


Reference is made throughout the modules to the learning outcomes
The course specifies any assumed knowledge
BUILDING STUDENT KNOWLEDGE

The course provides ways for students to review/gain assumed knowledge

Course content is appropriate to the student group

Course content is informed by current scholarship

Course content is gender inclusive

Course content is presented in a culturally sensitive way

New ideas are applied in real contexts

The materials include appropriate examples or case studies

Learners are given the opportunity to practice new ideas through activities

The course provides ways for students to follow up ideas and scholarship

The materials use summaries to consolidate what has been learnt

The course provides ways for students to reinforce their learning
LEARNING ACTIVITIES

Learning activities are appropriate for the targeted learning

The purpose of each learning activity is clearly stated

Learning activities are varied

Learning activities reflect the increased complexity of the ideas

Students use their personal knowledge to engage with activities

Learning activities are used for a variety of purposes as appropriate, introduce
ideas, consolidation, recall, critical thinking

Learning activities are gender inclusive

Learning activities are culturally sensitive

Learning activities promote self assessment

The time expected on each activity is clearly stated
ASSESSMENT

Each learning module includes an opportunity to demonstrate expected
performance

Summative assessment requirements are clear

The summative assessment promotes engagement with the ideas rather than
just memorisation

Summative assessment requirements are directly related to the stated
learning outcomes of the course

The types of summative assessment tasks are congruent with the Graduate
Qualities profile of the course

Assessment criteria are provided for each summative assessment task

Models or examples of summative assessment items are provided

Students are provided with information about the expectations of the form of
assessment (eg essay, report, oral presentation)

Students are provided with information about the presentation of the
assessment (eg word limits, use of graphics, layout)

The assessment criteria are reflected in the allocation of marks in the
summative assessment

The weighting of summative assessment is appropriate to the proportion of the
course being assessed

The summative assessment is directly aligned with the sequence and
presentation of the learning materials

Opportunities for formative assessment are provided throughout the course

Formative assessment is directly related to the learning processes and
outcomes
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
28









Formative assessment activities are varied
Online quizzes provide meaningful feedback about the correctness of student
responses
Students receive positive feedback for correct responses in online quizzes
Online quizzes provide students with opportunity to correct their errors and try
again
Feedback in online quizzes encourages students to explore concepts further
The student assessment workload is reasonable
The number of assessment tasks is appropriate
Summative and formative assessment draws on students experiences
Summative and formative assessments do not make unreasonable
assumptions about a student’s social or cultural background or life
experiences
EVALUATION

Link to evaluation is provided

Feedback from previous student evaluations is provided
SUPPORT

Contact information for support services is clear

Availability of resources from Library is clear

Useful resources available through Learning Connection are specified and
information provided on accessing them

Comprehensive information about and links to administrative information such
as relevant University policies and key dates is provided
HUMAN INTERACTION

Name of teacher and contact details are clear and correct

The kinds of support students can receive from the teacher and how this can
be accessed are clearly stated

Information on how to access other learning opportunities such as audio or
videoconferences or online chat sessions is clear

Availability of face-to-face contact opportunities is specified

Activities for building an online community are provided (eg participants page,
on line social discussions, defined activities)

Clear guidelines are given for group interactions in online discussions,
including aims and objectives and assessment requirements

Innovative use is made of synchronous (i.e. chat) and asynchronous (i.e.
discussion forums, email lists) communications to enhance learning (for
example online role plays, debates, student presentations)

Level of participation/facilitation by the teacher is appropriate for the activity

Feedback from the teacher is timely and designed to encourage learners to
engage in further discussion
INTERFACE DESIGN

Page layout is well designed and visually appealing

There is consistency in interface design and the same look and feel in all
pages

Navigation buttons and icons are easy to use and understand

All navigation buttons and links can be accessed using keyboard as well as
mouse

Home, Backward and forward links are provided so that students can navigate
easily

Hyperlinks use words that clearly identify where they lead
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
29








It is clear when hyperlinks take the learner out of the course materials
Text is broken up with appropriate use of graphical elements
Graphics do not dominate the screen
Use of screen space is appropriate
The use of headings and sub-headings, highlighted text and bullets enhance
the readability of the text
Font sizes are legible (12 to 14pt for body text and defined using styles)
Fonts are restricted to two families per page. Extended text generally uses san
serif typeface
There is sufficient contrast between the type and images and the background
WRITING STYLE AND ACCURACY OF TEXT

The style of writing encourages students to engage

Explanations are clear

The text is gender neutral

The text is culturally inclusive

New terms are introduced and explained

Terminology is consistent

Verbs are in active voice

The text is accurate in the facts and interpretations it provides

References are accurate and consistent

References to online articles and sites include author, title, URL provided as
an active link and date last accessed

Spelling and grammar are correct

Materials from external sources are used within the boundaries of the
copyright law
MULTIMEDIA

Particular media (e.g. images, animations, video, audio, print materials) are
included

Media are designed to achieve specific learning outcomes

Media are integrated into the teaching and learning processes

Use of the media is appropriate for target student group

Learning outcomes can be achieved with the media used

Images aid visualisation and facilitate understanding of concepts

Images have text equivalents (alt text) and captions

Images use more than colour to convey information

Diagrams and graphics are appropriate in terms of their informational content

Tables and worksheets include column and row headers and captions

Animations / video clips are used to convey information that cannot be
achieved through still graphics eg transition over time, 3D perspectives

Animations / video clips attract attention and stimulate interest

Animations / video clips facilitate interactivity

Animations / video clips can be halted

Animations include text equivalents and descriptions for sequences requiring
detailed explanations

Video clips include synchronised captions for audio tracks

Audio clips enrich the learning material

Audio clips include transcriptions for spoken dialogue

Audio can be turned off

Information about the file size is provided alongside links to video and audio
clips

Interactive components can be accessed using keyboard as well as mouse
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
30





Printed or online readings are at an appropriate level of difficulty for the
student group
The readings are appropriate in content
Readings in PDF format are available in text mode
Essential readings are indicated
The total number of readings is appropriate
WEB 2.0

The web 2 technology provides an appropriate feature set

Appropriate training and support materials are provided

Appropriate security features are provided

Group size is appropriate to the learning activity

Audience is appropriate to the learning activity

The teacher adopts an appropriate role

Assessment is appropriate to the learning activity and the technology

Consideration has been given to the ‘afterlife’ of content generated during the
learning activity

Staff and students have received appropriate training to use the technology

Student are aware of what to expect from web 2 technologies and the benefits
to be gained

Guidelines for ethical behaviour/communication are clearly defined

Students are aware of issues surrounding ownership of work

Appropriate activities are provided to build a sense of community

Appropriate scaffolding and support is provided

Students are actively encouraged to engage with content and each other

Assessment expectations are made clear to students

Assessment considers both quantity and quality of content and participation

Peer and self assessment are used where appropriate

Appropriate backed-up procedures are implemented to prevent loss of
assessable student work
WEB 3D

The genre is appropriate to the learning activity

The ‘mission’ is made clear to students

Elements such as story, fantasy, whimsy, competition and beauty are utilised
appropriately to engage students in the virtual environment

Opportunities are provided for students to develop and experiment with online
identity

Students are provided with an opportunity to explore the virtual environment at
their own pace

An electronic overview map is provided to assist students to navigate the
virtual environment

Guiding objects or automatic guided tours are provided to familiarise students
with the virtual environment

Navigation is easy and allows students to move freely as in the real world

Objects in the virtual environment are easy to identify/distinguish

The purposes of objects in the virtual environment are clear

Objects in the virtual environment are easy to access

Objects in the virtual environment look and behave in a manner appropriate for
the learning activity

Students are provided with a demonstration of how to carry out actions

The purpose of actions is clear

Skills required to execute actions are achievable

All available actions are apparent
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
31





T he effect of completed actions is apparent
Students are aware when a control has begun or ended
Students are aware of why a control has taken place
Input devices are appropriate to the learning activity and convenient to use
Students are provided with necessary training to use input devices
TECHNICAL ASPECTS

The system requirements are specified

The online resource is robust eg it does not "crash" the user's system and it
does not cause the browser to freeze

Operates with standard plug-ins provided in recent browsers and operating
systems

Functional even when features such as Javascript are not supported

Hyperlinks are active

Links to external sites open in a new window and user is informed

Page download times within the courses site do not exceed 10 seconds

User is informed if an external site will take longer than 10 seconds to down
load

Pages are printable

Printed pages are accurate
References
Aldrich, C. On simulations and serious games. Retrieved 21 February 2010 from
http://clarkaldrich.blogspot.com/.
Baille-de Byl, Penny and Taylor, Janet A. (2007) A web 2.0/web3D hyprid platform
for engaging students in e-learning environments. Turkish Online Journal of
Distance Education, 8 (3). pp. 108-127. ISSN 1302-6488. Retrieved 21
February 2010 from http://tojde.anadolu.edu.tr/tojde27/articles/article_7.htm
Burigat S., Chittaro L., Navigation in 3D virtual environments: Effects of user
experience and location-pointing navigation aids, International Journal of
Human-Computer Studies, November 2007, Vol. 65, No. 11, pp. 945-958.
Retrieved 21 February 2010 from http://hcilab.uniud.it/publications/200713/NavigationSupportVEs_IJHCS07.pdf
Hinckley, K. Tullio, J., Pausch, R., Proffitt, D. & Kassell, N. (1997) Usability analysis
of 3D rotation techniques, ACM/SIGGRAPH UIST’97 Symposium on User
Interface Software & Technology. Retrieved 21 February 2010 from
http://citeseer.ist.psu.edu/cache/papers/cs/9689/http:zSzzSzresearch.microsof
t.comzSzuizSz..zSzuizSzPaperszSzUISTrotationfinal.pdf/hinckley97usability.pdf
Lamb, B. (2004) Wide open spaces: Wikis, ready or not, Educause Review 39
Maier, P. & Warren, A. (2000) Integrating Technology in Learning and Teaching,
Kogan Page.
McConnell, D. (2000) Implementing Computer Supported Cooperative Learning,
Kogan Page.
Mallan, K. & Giardina, N. (2009) Wikiidentities: Young people collaborating on virtual
identities in social network sites, First Monday, 14(6).
Mindl, J. L. & Verma, S. (2006) Wikis for teaching and learning, Communications of
AIS, 18(1).
Owen, M., Grant, L., Sayers, S. & Facer, K. (2006) Opening education: Social
software and learning, futurelab.
Palloff, R. & Pratt, K. (2007) Building virtual communities: Techniques that work! 23rd
Annual Conference on Distance Teaching and Learning.
Salmon, G. (2002) E-tivities – the key to active online learning, Kogan Page
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
32
Salmon, G. (2004) E-Moderating – the key to teaching and learning Online, 2nd Ed.,
Kogan Page.
Jaques/Salmon (2007) Learning in groups: A handbook for face-to-face and online
environments, Routledge Edition: 4, Paperback, 360 pages.
Sulaiman, S. & Ahmad R. (2003) Enhancing design criteria for novice virtual
environment designers through the identification of usability problems,
Platform 3(1). Retrieved 21 February 2010 from
http://www.utp.edu.my/publications/platform/Platform%20v3n1.pdf#page=38
World Wide Web Consortium, 2008. ‘Web Content Accessibility Guidelines 2.0
(WCAG 2.0)’. Accessed 14 January 2010. Retrieved 21 February 2010 from
http://www.w3.org/TR/WCAG20
West, J. A. & West, M. L. (2008) Scaffolding Wiki projects For different levels of
learning, 24th Annual Conference on Distance Teaching and Learning.
FIRST YEAR EXPERIENCE
FAMILIARISING STUDENTS WITH THE UNIVERSITY’S PHYSICAL
ENVIRONMENT, ACADEMIC CULTURE, AND SUPPORT SERVICES
(SENSE OF BELONGING) BY PROVIDING:

Campus maps highlighting course relevant facilities are provided to students

A course ‘induction pack’ containing relevant information regarding
course/degree, study support and FAQs is provided to students

Relevant information regarding course/degree, study support and FAQs is
provided on the Course or School website

Relevant information regarding course/degree, study support and FAQs is
provided at the School office

Students are aware of where they can access course/degree, study support

Course coordinators/lectures are familiar with the support available for
students

Students are aware of the resources they will require for the course

Students are aware of University operating systems and where they can
access support if required

Relevant university policies are introduced and explained to students

Orientation activities are provided to familiarize students with key facilities on
campus

Orientation activities are provided throughout the first year of study
GIVING STUDENTS AN UNDERSTANDING OF WHAT THEIR LEARNING WILL
INVOLVE AND WHERE THEIR COURSE WILL LEAD THEM

Students are prepared s for their new learning environment

Students are introduced to the culture of the discipline and mad aware of how
their unit of study relates to future study

Course, assignment, assessment expectations are made explicit to students

Students are aware of minimum attendance/participation guidelines

Students are provided with clear guidelines on plagiarism

The course outline is easy to understand and outlines what students can
expect to achieve as well as what they are expected to do

Essential course information is provided online (e.g. on a course website) to
students
PROMOTING AND SUPPORTING STUDENTS’ ENGAGEMENT WITH THE
UNIVERSITY AND THEIR PEERS

Students are provided with social activities to meet each other, and students
from senior years
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
33







Appropriate time is allocated at the beginning of the course to discuss any
orientation problems students may be experiencing
Course coordinators/lectures are available to meet with students during
Orientation and at other allocated times
Orientation Information is part of a Faculty wide initiative and included in all
unit of study outlines
Students who are at risk are identified early and provided with relevant support
Induction activities are evaluated by students
Student feedback regarding induction activities are discussed in an open
forum
Good staff/student relationships are fostered
DEVELOPING THE KNOWLEDGE AND SKILLS OF STUDENTS FROM DIVERSE
BACKGROUNDS

An appropriate amount of time is allocated at the beginning of the course to
discuss the activities students will be asked to participate in (group work, class
discussion, presentations, lab work)

Students are made aware of graduate attributes and how they will be
developed in their study

Students are encouraged to examine their own values and explore the moral
and ethical values of a diverse society

Students are aware of different learning styles

Practice exercises and quizzes are utilised to provide students with feedback

Feedback is provided early in the course

Feedback is timely and appropriate

Study skills are taught as part of the course
References
Asmar, C., Brew, A., McCulloch, M., Peseta, T. & Barrie, S. (2000) The first year
experience project, Institute for Teaching and Learning, University of Sydney.
Bovill, C., Morss, K. & Bulley C. (2008) Quality enhancement themes; The first year
experience curriculum design for the first year, The Quality Assurance Agency
for Higher Education. Retrieved 21 February 2010 from
http://www.enhancementthemes.ac.uk/documents/firstyear/Transition%20%20Final.pdf
First Year Experience, Enhancing students’ academic orientation & transition
website, University of Sydney.
Orientation to University life, Institute for Teaching and learning, University of
Sydney, Retrieved 21 February 2010 from
http://www.itl.usyd.edu.au/pastProjects/FYE/initiatives/pdfs/tips_orientation.pdf
Communicating Expectations, Institute for Teaching and learning, University of
Sydney. Retrieved 21 February 2010 from
http://www.itl.usyd.edu.au/pastProjects/FYE/initiatives/pdfs/Tips_expectations.
pdf
Giving Feedback, Institute for Teaching and learning, University of Sydney
http://www.itl.usyd.edu.au/pastProjects/FYE/initiatives/pdfs/tips_feedback.pdf
Kift, S. (2008, 30 June - 2 July) The next, great first year challenge: Sustaining,
coordinating and embedding coherent institution–wide approaches to enact
the FYE as “everybody’s business”. Paper presented at the 11th Pacific Rim
First Year in Higher Education Conference, Hobart, Tasmania.
Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005) The first year experience
in Australian Universities: Findings from a decade of national studies.
Retrieved 21 February 2010 from
http://www.cshe.unimelb.edu.au/pdfs/FYEReport05KLK.pdf
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
34
Kuh, G. D. (2007) What matters to student success in the first year of university?
Paper presented at the 10th Pacific Rim First Year in Higher Education
Conference 2007, Brisbane, Australia.
Learning & Teaching Centre website, University of Glasgow good practice
resources, first year experience. Retrieved 21 February 2010 from
http://www.gla.ac.uk/services/learningteaching/goodpracticeresources/firstyearexper
ience/#d.en.115859
Student experience project, Charles Darwin University. Retrieved 21 February 2010
from http://www.cdu.edu.au/tlqg/sep/firstyear.html
Tinto, V. (2006). Research and practice of student retention: What next? Journal of
College Student Retention: Research, Theory and Practice, 8(1), 1-19.
TEACHING RESEARCH NEXUS
KNOWLEDGE OF CURRENT RESEARCH

Course and subject materials reflect current discipline knowledge

Students are encouraged to engage with research findings and literature

Students are provided with opportunities to discover and discuss new research
topics and questions
THEORY AND PRACTICE OF RESEARCH

Discipline knowledge is explored in the course

Ethical issues in research, including data collection and acknowledgment
practices are explored in the course

Students study the ways in which researchers work and examine the nature
and use of specific methods

Students are provided with research activities to develop their research skills
CRITICAL THINKING AND ENQUIRY

Activities such as problem solving, enquiry, analysis, reflection and critical
thinking are included in the course

Students are encouraged to question knowledge in the disciplines and discuss
competing perspectives in the field of study
ENGAGEMENT IN A SCHOLARLY COMMUNITY

Students are aware of the work of faculty/discipline researchers

The teacher shares their interest and enthusiasm for research as part of
teaching

Faculty seminars and workshops are provided for students to attend

Students collaborate with other students and with academics on research
projects

The teacher encourages students to undertake further research
REFLECTIVE PRACTICE

Students are provided with activities which assist them to reflect on what and
how they learn

The teacher applies current theories of student learning

The teacher undertakes and disseminates research about learning and
teaching (scholarship of teaching and learning)

The teacher reflects on their teaching and research practices and
interconnections
References
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
35
Research-teaching nexus resources, University of New South Wales. Retrieved 21
February 2010 from
http://learningandteaching.unsw.edu.au/content/RandI/research_nexus/rtn_res
ources.cfm?ss=5
The teaching research nexus project website. Retrieved 21 February 2010 from
http://www.trnexus.edu.au/
Nexus articulation, University of Wollongong. Retrieved 21 February 2010 from
http://www.uow.edu.au/content/idcplg?IdcService=GET_FILE&dDocName=U
OW065199&RevisionSelectionMethod=latestReleased
INTERNATIONALISATION
BROADEN SUBJECT AREAS THROUGH INTERNATIONAL/INTERCULTURAL
APPROACHES

International content is included in the curriculum where educationally
possible/desirable

Specific references to contemporary International and Australian content is
provided

Ethical issues in globalisation (such as social justice, equity, human rights and
related social, economic and environmental issues) are included in the course

International and national case studies are compared and contrasted

Historical accounts of the development of current international
discourses/practices are included in the course

Studies of professional practices in other nations/cultures are included in the
course

Studies of how knowledge is constructed differently across cultures in the
subject area are compared and contrasted

Student/staff diversity in utilised to facilitate discussion and exchange of ideas

Dialogue and collaborative learning activities are included in the course
ENCOURAGE INTERACTIVE CROSS-CULTURAL EXCHANGES

International student exchanges are incorporated or encouraged within the
program

Students are actively encouraged to engage with others from different
backgrounds in their study activities

Students are encouraged to join international associations that are affiliated
with their discipline
ALIGN COURSE CONTENT, LEARNING ACTIVITIES AND ASSESSMENT

Assessment is aligned with concepts, issues and events from the coursework
that are relevant to the development of international and intercultural
perspective

Learning activities and related assessment items draw on cultural contexts as
well as disciplinary knowledge

Internationally comparative activities are included in assessment exercises
ENCOURAGE AND MODEL INCLUSIVE STRATEGIES AND OPENNESS TO
DIVERSITY

Mutual respect is demonstrated between staff and students:

Students are encouraged to gain a deeper knowledge and understanding of at
least one other culture’s customs, history, language, literature, philosophy,
economics, and politics
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
36


Students and staff are encouraged to learn a second language as a basis for
appreciating the challenges of self-expression in a language other than their
mother-tongue
Students are encouraged to complete a course in cross-cultural
communication or international studies
DEMONSTRATE SENSITIVE AND INCLUSIVE TEACHING PRACTICES

An awareness of language or behaviour that is ethnocentric or racist is
demonstrated

An awareness of cultural stereotyping is demonstrated

An awareness of the complex nature of other nationalities and cultures is
demonstrated

Materials from international and intergovernmental organisations (including
international research) are utilised to broaden the learning experience and
knowledge base of students and academic staff

The diversity of students and staff in courses is utilised to exchange
knowledge and ideas
References
Internationalising the curriculum, GIHE good practice resource booklet –
internationalising the curriculum tip sheet – course level strategies, Griffith
University. Retrieved 21 February 2010 from
http://www.griffith.edu.au/__data/assets/pdf_file/0011/186905/IoC_Courselevel.pdf
Centre for the Enhancement of Learning and Teaching resources on
internationalisation. Retrieved 21 February 2010 from
http://www.lancs.ac.uk/celt/celtweb/internationalisation
UniSA Graduate Quality 7: Internationalisation. Retrieved 21 February 2010 from
http://www.unisa.edu.au/ltu/staff/practice/internationalisation/documents/gq7.p
df
EMBEDDING PEER REVIEW IN BLENDED AND ONLINE LEARNING
ENVIRONMENTS
CLEAR GOALS
For students’ learning and for design of the learning environment:

Clear goals for students’ learning and an understanding of how those goals
are meaningful and appropriate for the students, the course and the context

Clear rationale for the design of the learning environment, including the
chosen blend of options
Pre-review questions for the reviewee:

What are your intentions for student learning in this aspect of the
subject/unit/teaching?

Why have you designed this subject/aspect of the subject in the way that you
have?
CURRENT AND RELEVANT PREPARATION

Currency and relevance of the content

Teaching and learning practices that are informed by current scholarship and
awareness of relevant innovations

Taking into account students’ expected previous knowledge and experience,
including experience of similar learning environments
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
37

Timely updating of teaching materials and resources for students, including
currency of online learning sites
Pre-review questions for the reviewee:

How did you prepare for this aspect of the subject/teaching this time?

What did you take into account in your preparation?

How did you adapt the subject for this particular group of students?

How did you ensure currency?
APPROPRIATE METHODS FOR LEARNING

Learning and teaching and assessment methods that are appropriate for the
learning objectives, students, context and available resources

Opportunities for student independence and choice, for example flexibility of
learning modes and/or choices of content or focus

Opportunities for students to develop graduate attributes relevant to the
subject/learning activity

Fostering of students’ active engagement in learning, for example through
opportunities for inquiry and exploration of ideas

Fostering of student interaction and collaboration

An appropriate level of intellectual challenge

Opportunities for students to relate what they are learning to broader contexts
eg work, life experience, the broader discipline

Flexibility to respond to students' ideas and understandings, feedback and
changing situations
Pre-review questions for the reviewee:

Are there any particular teaching and learning methods or activities on which
you would like the reviewer to give feedback?

Is there anything that you do not want feedback about (for example because
you already intend to make changes)?
EFFECTIVE COMMUNICATION AND INTERACTION

Clear communication with students about expectations, including guidance on
requirements and options in blended learning environments

Clear pathways and navigation in online and blended environments

Clear explanations

Motivating student interest and perceived relevance

Responsiveness to students’ understandings, ideas and progress in learning

Responsiveness to students' communications and questions

Effectiveness of co-ordination/communication with any other staff teaching in
the subject.
Pre-review questions for the reviewee:

Are there any particular aspects of your communication that you would like
feedback about?
IMPORTANT QUESTIONS
Outcomes for students:

Evidence of student engagement

Evidence of student learning: desired outcomes and unexpected learning
outcomes
Other outcomes if applicable:

Evidence of outcomes related to any other intentions of the learning activity
(for example evidence of the effectiveness of a learning innovation in
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
38
achieving particular goals, evidence of effective collaboration with
colleagues/tutors)

Evidence of broader significance – eg potential for the adaptation and scalingup of an innovation

Presentation of scholarly reports of practice to colleagues and others.
Pre-review questions for the reviewee:

Are there forms of engagement or intended outcomes that you would like the
reviewer to give feedback about?
IMPORTANT QUESTIONS

Learning from students and adapting teaching in response, during teaching
and afterwards

Seeking a variety of forms of evidence about teaching

Acting on the evidence – showing evidence of how previous feedback has
been built in to improve

Scholarly reflective practice informed by self, literature, students, peers and
other sources
Pre-review questions for the reviewee:

How has previous reflection and feedback informed this aspect of your
teaching/subject?
References
ALTC Funded project (2007) Embedding peer review of learning and teaching in elearning and blended learning environments. Retrieved 21 February 2010 from
http://www.altc.edu.au/project-embedding-peer-review-learning-uts-2007
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
39
The Australian Learning and Teaching Council has provided funding for this project.
User Guide: Online Peer Review System
2