ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
Building a community-informed framework for assuring quality in distance
learning programmes
Richard Walker & Wayne Britcliffe
E-Learning Development Team, University of York
Abstract
This short paper reports on the work of the University of York’s Distance Learning Forum in
developing an interactive, evidence-based framework for distance learning programmes, addressing
the specific dimensions of quality for the delivery of teaching, learning and assessment activities to
distance learning students. The framework was designed to help inform planning and scoping tasks
for prospective distance learning programmes, whilst assisting newly established programmes in
keeping on track in the period after validation but before the 3-year review. It was also intended to
foster a community-focused approach to quality management, unlocking the secrets of existing
programmes by drawing on examples from their practice, to help make quality standards more
intelligible to prospective programme teams. The paper reports on how the framework has been
applied by two new postgraduate distance learning programmes, shaping their development
through the programme scoping and approval stages in 2012.
1. Background
Blended course delivery at the University of York has grown rapidly since the establishment of a
centrally-supported virtual learning environment in 2004 and the introduction of dedicated support
provision, with the aim of enhancing the learning experience for campus-based students (Beastall &
Walker, 2007). Fully online provision has followed a different pathway, with an institutional
commitment to support the ‘careful growth of distance learning, particularly to enable flexible
professional development at postgraduate level’ (University of York, 2010: 5). This has resulted in a
gradual increase in distance learning provision through online delivery over the last decade; when
considering new programme proposals the emphasis remains on niche developments where the
University has sector-leading teaching and research expertise, rather than on volume in upscaling
online programme delivery.
Currently there are fifteen accredited distance learning programmes delivered by seven
departments at York, with the majority offered at postgraduate level (certificate / diploma/masters).
[1] The portfolio is small in number but individual programmes are innovative and based around
flexible study designs. In contrast to HE institutions such as the Open University which have
dedicated central support units to promote the development of distance and open learning,
responsibility for programme delivery at York is devolved to individual departments: programme
teams manage their affairs independently from the centre, subject to their own departments’
quality assurance (QA) procedures, after the validation of new programmes by the University’s
Teaching Committee. However, over the years programme leaders have taken advantage of personal
networks to collaborate and share practice, recognising the benefits of experience sharing,
consulting with practitioners beyond disciplinary boundaries. This collaboration was indeed
1
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
formalised in 2007 with the creation of a Distance Learning Forum, established to facilitate the
exchange of information and discussion of issues of common interest and concern in the running of
distance learning programmes, whilst also serving as a source of advice and expertise to the
University’s Teaching Committee on distance learning matters (e.g. programme validation and
programme review). [2] The Forum meets on a termly cycle and is chaired by the Pro Vice
Chancellor for Teaching and Learning and its membership consists of representatives from all key
central services (e.g. marketing, library, e-learning) and representatives from all programme teams.
It has proven effective in raising the profile of distance learning within the institution and delivered
enhancements to institutional marketing and registration processes for new students , and it has
also fostered the sharing of pedagogic practice, which have been realised through a series of
collaborative initiatives, such as the Forum’s peer observation programme for distance learning
tutors and managers (Walker & Hall, 2011), as an alternative to the University’s peer support for
teaching scheme.
2. Drivers for the development of a QA framework for distance learning
Consideration of the need for the development of a dedicated quality assurance framework for
distance learning programmes arose from a review of QA models by the Forum over academic year
2010-11.The review explored how programme teams addressed stakeholder groups and managed
the feedback cycle to students and conducted programme review and evaluation activities. The
review findings, which were reported to the Forum in February 2011, confirmed a convergence of
practice across the programme teams in the way that they consulted with stakeholders during the
design and approval phases for new programmes and in their management of the feedback cycle
once a programme had been launched. However, variations in practice emerged in terms of the
evaluation methods the programme teams used, as well as in their training and support provision notably the levels of training and support afforded to new tutors. These findings prompted a
recommendation for a QA working group to be set up by the Forum to provide illustrative guidance
to distance learning programme teams on appropriate standards and methods for assuring the
quality of their programmes. The intention was to capture the good practice on evidence in the
review, as well as to address areas for attention, particularly where variation in practice had been
observed.
The working group’s membership consisted of distance learning tutors, programme managers, QA
and e-learning specialists, and was tasked with the development of a framework to help inform
planning and scoping tasks for prospective distance learning programmes, as well as to assist newly
established programme teams in keeping on track in the period after validation but before the 3year review. The objectives for the framework are summarised in Table 1 below.
Table 1: Objectives for the QA framework for distance learning
(i)
To provide clear guidance (accessible, engaging and useful) to prospective programme
teams on how to tackle the planning and validation phases of programme development
(often a lengthy process over a two-year cycle), with the intention that this might lead
to better formed proposals to University Teaching Committee, complementing existing
guidance on the university website.
2
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
(ii)
(iii)
To offer an on-going resource for approved programmes which can be used to help
assist self-regulation, keeping on track in the period post-validation and before the
formal 3-year review when the programme is scrutinised in detail by a visiting panel, and
for on-going management after the validation process has been completed and the
programme is absorbed within departmental QA procedures (annual programme
reviews).
To serve as a reference tool for validation teams (non-distance learning experts).
It was hoped that the process of developing the framework would help to foster a communityfocused approach to quality management, unlocking the secrets of existing programmes by drawing
on examples from their practice, to help make quality standards more intelligible to prospective
programme teams.
3. Design of the QA framework
The development of the framework was conducted over academic year 2011-12 and was informed
by best practice guidance on QA standards for distance learning (QAQE SIG: 2011), including insights
from an influential Quality Enhancement in E-learning pathfinder workshop hosted at Teesside
University in June 2008. [3] The workshop highlighted a series of valuable resources on institutional
procedures and approaches to quality assurance and quality enhancement in e-learning activities,
including UCL’s quality framework for its MSc in International Primary Health Care (UCL, 2008). The
working group members were able to draw on these resources during the initial stages of their
thinking, notably adapting and developing further the quality standards in the UCL framework to suit
the University of York’s context.
The resulting framework was organised into three key components, offering guidance on the
programme review cycle, quality standards (covering the six dimensions of quality) and the key
documentation that programme teams need to manage, as depicted in Figure 1 below.
3
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
Figure 1: Design of the QA framework
Specifically the resource offers guidance to distance learning programme teams in the following
areas:


Programme review cycle: addressing routine procedures for monitoring the performance of
a distance learning programme, with attention to timescales (key milestones) and
suggested levels of scrutiny;
Dimensions of quality: six themes focusing on course materials, the interactive learning
environment, tutor performance and development, assessment and feedback, academic and
pastoral support provision and administrative and pastoral support provision. For each
theme the resource addresses standards for quality management, with suggested evaluation
methods which a programme team can adopt and indicators of quality failures to alert staff
to potential problems when monitoring programme delivery and management;
4
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes

Key documentation: a summary checklist of policies, plans and guidance resources which a
programme team should develop and maintain, and guidance documentation that team
members should refer to in developing their own processes.
The framework was formally adopted by the Distance Learning Forum in summer 2012 and made
available as a static PDF resource to current and prospective distance learning programme teams.
Subsequently a web-based version of the resource was created in May 2013, incorporating links to
illustrative programme resources (excerpts from programme handbooks and online tools) from a
range of distance learning programmes, with much of the evidence drawn from a repository
prepared for a recent QAA institutional audit (2012). The interactive format was intended to help
make the resource dynamic, enabling the community of distance learning practitioners across the
university to share and update exemplar policies, plans and guidance resources, as well as to
contribute to the on-going review and definition of quality standards for online programmes. The
interactive format also has the added value of enabling the cross-linking and referencing of
standards to evidence and key documentation, supporting ‘views’ by either of these dimensions.
4. Application of the QA framework: contribution to programme development and delivery
The framework has been used by the University’s E-Learning Development Team to help inform staff
who are thinking about developing their own distance learning programmes, setting out the key
issues for consideration when preparing for the programme approval and validation phases. The
framework has also been employed to guide two new programme proposals through a development
lifecycle and most recently has also been referenced in recommendations made by a University
Teaching Committee panel during a three-year review for an established distance learning
programme.
This paper focuses on two new postgraduate programmes, which were developed by teams working
closely with the E-Learning Development Team, referencing the hard copy (static) framework at
various stages of their development in 2012.
The Postgraduate Certificate in Defending Human Rights (Centre for Applied Human Rights) was
developed over the summer and autumn terms in 2012, with the first module delivered in the spring
term of academic year 2012-13. The programme follows a problem based learning design, with a
weekly release of new content resources to students. The programme leader, by her own admission,
was starting from scratch with no prior knowledge of distance learning delivery.
The MSc in International Humanitarian Affairs (Post-war Reconstruction and Development Unit,
Department of Politics) was launched in January 2013, following a development period of two years.
The programme is designed around group-based discussion activities and draws upon externally
(guest) authored content materials. The programme leader was not new to distance learning, having
served as a tutor on the University’s Public Policy and Management distance learning programme,
although this was the first time that he had overseen the development of a new distance learning
programme.
5
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
5. Evaluation methods
A series of structured interviews were conducted over the summer of 2013 with the PG Cert. and
MSc programme leaders, which had been guided through the approval and development stages
using the hard copy (static) quality framework as a supporting resource.
The interviews focused on the stages that programme leaders negotiated in developing their
programme from a proposal to a fully approved programme, and they were invited to reflect on
their experiences of the planning and validation phases and the quality of support that they had
received along the way. The aim of the research was to gain insights into their experiences in
developing a new distance learning programme, and in so doing, elicit feedback on the support
resources that they had made use of and the levels of support that were required to get their
programmes up and running. The rationale for this research was to help improve support provision
for future distance learning programme development. We therefore opted for an interpretive
research methodology, focusing on perceptions of their experiences, establishing ‘meaning’ from the
standpoint of the actors (Orlikowski & Baroudi, 1991), rather than through objective measurements.
6. Reflections on programme development
Drawing together findings from the interviews, we observed six distinct phases in the programme
development cycle which leaders referred to when narrating the development of their programmes,
ranging from definition of programme proposals through to delivery and on-going programme
management. The phases of development are listed below, with observations from the programme
leaders on the resources and support that they drew on in developing their programmes:
Phase (i): Getting Started
(Definition of programme proposal)
Evidence:
‘Consultations with the E-Learning Development Team were useful in setting out the
possibilities and providing delivery models to explore further. This involved enrolment on
archived versions of comparable distance learning programmes, as well as shadowing a ‘live’
programme from a competitor as an external auditor. The QA framework and Distance
Learning checklist were most valuable in representing the stages for the development of the
programme and the issues to think about – very useful but scary – good scary. Good in
reinforcing the message that the majority of effort should be devoted to the pedagogy and
content development, rather than technical issues.’ (PG Cert. programme leader)
6
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
Phase (ii): Negotiating programme approval
(Approval by departmental Board of Studies & validation by University Teaching Committee)
Evidence:
‘The framework also helped with the University Teaching Committee validation processes –
which were very painful to prepare document for, but forced you to think about accessibility
issues, formatting etc., and forced you to have a clear conception of what you want.’ (PG
Cert. programme leader)
Phase (iii): Programme development
(Steps towards the development of the online course environment; definition of programme
management processes and confirmation of programme delivery team)
Evidence:
‘The framework has also been helpful for recruitment – identifying themes which we need to
address in the job specifications for the administrator and teaching fellow posts – the types of
things that they should be doing – we made sure that they were included in the job
specification from the beginning.’ (MSc programme leader)
Phase (iv): Pre-delivery checks
(Upfront planning for launch of the programme)
Evidence:
‘The Module 1 evaluation process will draw on feedback from tutors and administrators, and
students and also VLE statistics reporting on student activity within the module site. The
framework has been helpful in identifying themes and headings to address -putting together
processes – timings and things to look for. It has also helped with the division of labour
amongst the programme team – who looks out for what when reviewing progress in the
delivery of the programme. This has contributed to the development of an ongoing evaluation
plan, with evidence being collected on the way.’ (MSc programme leader)
Phase (v): Delivery issues
(Procedural delivery issues to address with students to ensure that they can fully engage with
the programme)
Evidence:
‘The framework also helped to prompt actions, such as the development of course evaluation
forms and prompted consultation with students about accessibility issues and their access to
technology. Some things you can never anticipate though – such as YouTube being banned
in Pakistan.’ (PG Cert. programme leader)
7
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
Phase (vi): Programme management
(On-going responsibilities for programme team in managing delivery and assuring quality)
Evidence:
‘The framework is also effective – telling me to pay attention to issues in the future – such as
tutor training, which are not relevant now but will be part of our practice as the programme
grows and we develop a tutorial team. It is good to think about these things in terms of
planning and handover to other people. I am keen to employ external reviewers of the course
materials and to engage with other York programme leaders through peer observation
process to learn more about art of possible. (PG Cert. programme leader)
‘(The evaluation plan) will feed in to an end-of- year programme review to be conducted in
December 2013, a light-touch version of the three-year review, instigated and managed by
the programme team to ensure that the programme remains on track.’ (MSc programme
leader)
6. Conclusions and next steps with development of the QA framework
The combined evidence from the interviews suggests that the hard copy version of the framework
served as a useful guide to both programme leaders at various milestones in their programme
development, although its value depended on the prior experience and familiarity of the respective
programme teams with distance learning delivery, as well as practical issues related to timing and
the pace of programme development, which determined how they made use of it . Notably the MSc
programme team made far less use of the framework at programme definition and approval stages
than the PG. Cert leader did, which may be attributed both to the MSc programme leader’s prior
familiarity with distance learning development, and the progress that his team been made through
longer lead-time preparation, prior to the framework becoming available.
The evidence suggests though that as a reference resource, the framework was effective in drawing
attention to specific planning and delivery issues (e.g. job specifications for programme team roles
and evaluation plans for unit and end-of-year reviews), which could then be addressed as part of
both programme teams’ planning activities. The E-Learning Development Team referenced sections
of the framework in highlighting these recommended actions in the preparation and delivery of the
programmes. In this sense the framework complemented existing institutional guidance and
resources on programme development (e.g. distance learning delivery checklists and templates
available on the University’s e-learning support website). It is unclear from our research whether
the resource would have had any value as a stand-alone tool, although the development of the
interactive version of the framework may facilitate future self-guided reviews of QA issues. Notably
the web-based version, through its links to supporting evidence and artifacts, offers some
advantages over the hard copy in interpreting the quality standards under review, which may in turn
support self-guided review.
‘The framework is now online and much easier to use. It provides guidance for programme
teams which is comprehensive and easy to use – wonderful for anyone starting out. New
programmes teams should make sure they use it. We use it quite frequently (now our
8
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
programme is up and running) and it has been used to help with the draft of the end of
Module 1 report.’ MSc programme leader)
In terms of next steps, the Distance Learning Forum remains committed to the on-going refinement
of the standards of quality and supporting evidence, drawing on artifacts and materials developed
by existing and new programmes. To this end the online design will enable new supporting evidence
to be easily incorporated within the framework. The Forum also wishes to solicit feedback on the
usability and value of the resource from external institutions, and in the spirit of the pathfinder
QAQE projects has released both the hard-copy and interactive versions of the framework to the
community under a creative commons licence for their use and adaptation. [4]
Notes
[1] The portfolio of the University of York’s open and distance learning courses may be viewed at:
http://www.york.ac.uk/study/postgraduate/courses/distance/#tab-2
[2] The terms of reference for the University of York’s Distance Learning Forum are available here:
https://www.york.ac.uk/staff/teaching/groups/distance/
[3] Further information on the Quality Enhancement in E-learning workshop is available here:
https://www.scm.tees.ac.uk/pathfinder/qeworkshop.html
[4] The hard copy and web-based versions of the QA framework were released over the summer
(2013) under a creative commons licence. They may be accessed at:
https://vle.york.ac.uk/bbcswebdav/institution/Distance%20Learning%20Forum/Quality%20Framew
ork%20Website/DLF_Home.html
References
Beastall, L. & Walker R. (2006). Effecting institutional change through e-learning: An implementation
model for VLE deployment at the University of York. Journal of Organisational Transformation and
Social Change, Vol. 3:3, pp 285-299.
Orlikowski, W. & Baroudi, J. (1991). Studying Information Technology in Organizations: Research
Approaches and Assumptions. Information Systems Research (2), pp. 1-28.
Quality Assurance - Quality Enhancement in e-Learning Special Interest Group (2011). A Toolkit for
Harnessing Quality Assurance Processes for Technology Enhanced Learning. Retrieved from:
http://qaqe-sig.net/wp-content/uploads/2011/03/Toolkit_version_2011_9_3.pdf
UCL (2008). Web based MSc Programme in International Primary Health Care: Quality Framework. In
(Eds.) M. Jara, M. Fitri. & S. Cranmer (2008). Evaluation of E-Learning Courses. London Knowledge
Lab. Institute of Education, November 2008, Appendix 7. ISSN 1753-0385. Retrieved from:
http://www.wlecentre.ac.uk/cms/files/occasionalpapers/evaluation_of_online_courses_25th.pdf
9
ALT-C short paper #344: Building a community-informed framework for assuring quality in distance
learning programmes
University of York (2010). Learning and Teaching Strategy 2010-2015: A Consistent Culture of
Quality. Retrieved from: https://www.york.ac.uk/staff/teaching/strategy/
Walker, R. & Hall, K. (2011). Peer observation for distance learning e-tutors: Evaluation of a crossprogramme trial scheme. ALT-C 2011 - thriving in a colder and more challenging climate. 6-8
September 2011. University of Leeds, UK. Retrieved from:
https://vle.york.ac.uk/bbcswebdav/xid-1035353_4
Address for correspondence
Richard Walker, Wayne Britcliffe
E-Learning Development Team, J.B. Morrell Library, University of York, Heslington, York, N.Yorks.
YO10 5DD.
richard.walker@york.ac.uk
wayne.britcliffe@york.ac.uk
10