Surveying the student experience …
1
Surveying the student experience: institutional collaboration and benchmarking
Mantz Yorke, Lancaster University and Bernard Longden, Liverpool Hope University
Abstract
This paper discusses methodological issues relating to a survey of the experience of
part-time students in higher education, and indicates that the methodology used is
adaptable to a variety of situations. Whilst the focus of the paper is on methodology and
its transferability, some initial findings from the survey are presented.
Introduction
In the UK, the experiences of part-time students have received little research attention –
save for intra-institutional studies that are rarely communicated beyond the boundaries
of the institution – since the pioneering study carried out by Bourner et al (1991), despite
the increasing numbers of students who choose to participate in higher education via
this mode of study. In 2005-06, there were more than 850,000 students enrolled parttime in higher education programmes at sub-degree, first degree and taught master's
level across the whole of the UK.
The lack of attention to the experiences of part-time students is all the more surprising
because of national policy imperatives regarding both participation and workforce
development. Changes being mooted in the funding methodology for England in order
to support flexibility in provision and completion (HEFCE, 2007, para 59ff) add an extra
dimension.
Bourner et al (1991) attracted 2876 usable responses which constituted roughly a 70%
response to their wide-ranging survey of students on part-time degree programmes.
There was a widespread satisfaction with the academic courses, but the responses
relating to institutional facilities were more mixed. A survey of 556 part-time students in
Scotland (Schuller et al 1999) came to broadly similar conclusions.
More recently, as the third strand in a three-strand study of part-time students for
Universities UK, Callender et al (2006) conducted an on-line survey of 2654 part-time
students that focused to a considerable extent on to economic aspects of engagement
and barely touched on students’ course experiences – indeed, they took the view that
this could largely be left out of consideration because of the National Student Survey.
Data from the NSS in 2005 and 2006 (Table 1) show that, at the time when their studies
were approaching their conclusion, students had strongly positive perceptions regarding
their experiences in higher education. It should be noted, however, that part-time
students more often responded to the NSS by phone, which tends to be associated with
more positive ratings on the NSS (Surridge, 2007).
1 July 2008
Surveying the student experience …
2
2005
Scale
mean
2005
Number of
responses
2006
Scale
mean
2006
Number of
responses
Scale
The teaching and learning on
my course
Assessment and feedback
Academic support
Organisation and management
Learning resources
Personal Development
4.13
3.94
3.89
3.89
3.89
4.03
12710
13017
12412
12402
8901
12767
4.13
3.94
3.89
3.84
3.88
4.00
11732
11960
11318
11399
8192
11744
Overall
4.20
13431
4.18
12283
Source: Surridge (2007, Annex D)
The authors, who conducted a pioneering survey of the experiences of full-time first-year
students in the UK (Yorke and Longden, 2008), saw the need for a similar study of the
experiences of part-time students, at all stages of engagement with higher education
(i.e. not relying on the retrospection inevitable with the NSS), which would provide
‘baseline’ data emphasising their experiences of their programmes that would be useful
sectorally for a number of purposes. With a substantial proportion of part-time study at
undergraduate level being concentrated in ‘post-1992 universities’, it seemed sensible to
focus attention on these institutions in the first instance. Approaches were made to
various potential funders for support, but without success despite the policy significance
of the study and the potentially high level of return on any funder’s investment that could
accrue from the proposed methodology (see below).
First thoughts
The original plan had been for a postal hard-copy survey which would be optically
readable in order to produce, from ‘check box’ responses, a data matrix for statistical
analysis. There would in addition be opportunities for respondents to comment, at
whatever length they chose, on matters that they felt needed to be expressed.
The costs of such a survey (design, production, administration and postages) are
roughly £2 per person surveyed. In the post-1992 institutions part-time student
enrolments run up to approximately 16,000 1, though the median enrolment was
considerably lower. Involving around a dozen institutions of any size would represent
roughly 10 per cent of the total sectoral enrolment, and would push the total costing
towards £200,000. It was appreciated that raising such a level of funding would, in the
In contrast, the Open University’s enrolment of similar students reached nearly 170,000 in 200506.
1
1 July 2008
Surveying the student experience …
3
relatively short time available for seeking it, be likely to be challenging in the extreme.
An alternative approach offered a better prospect of achieving results.
A number of post-1992 institutions were approached regarding their possible
involvement on a partial self-help basis. If they would be responsible for distributing
surveys to their part-time students, the costs of central administration and management
could be reduced to a figure of around £40,000. In return, they would have access to
data – in a suitably anonymised form – from the other (broadly cognate) institutions
which were participating in the study. The majority of the approached institutions agreed
to participate on this basis, and it was a major disappointment that the search for funding
proved unavailing. If the above approach could be termed a Plan B, a Plan C would be
needed if anything at all were to be achieved during the academic year 2007-08.
Cutting according to the cloth
The only feasible way of making progress was to cut out the relatively expensive printbased survey and instead to seek to exploit the potential of the internet. The codirectors of the project had available a small amount of money which could be used to
cover the costs of convening meetings of participating institutions and to set up an
internet-based survey. The institutions that had expressed their willingness to engage in
the print-based survey indicated that they were prepared to contribute appropriately to
the electronic survey. In practice, this meant that their major commitment would need to
be to construct lists of relevant students’ electronic addresses and to e-mail to the
students an invitation to participate (and probably a reminder).
Design issues
The response rate to electronic surveys is often very low (a survey of part-time students
for Universities UK attracted an overall rate of 4.7%: Callender et al, 2006), and the
authors’ experience had shown that lengthy surveys (in the UK, if not in Australia and
the United States) were susceptible to respondents’ attenuation of engagement. This
meant that the survey of in-institution experience would have to be shorter than originally
envisaged for the print-based approach, particularly bearing in mind the wide variation in
demographic background that exists in the part-time student body which would
necessitate a considerable number of demographic questions if analyses were to be
meaningful.
The identification of institutions
A majority of the ten post-1992 universities (from across the UK) that were originally
approached had indicated their willingness to participate, though this majority included
only institutions from England and Scotland. For an initial survey that would inevitably
be limited in size, it was felt that representativeness of the sample in all respects was
less important than having a group of institutions that were committed to the work:
subsequent studies would need to take representativeness more fully into account.
Other post-1992 universities were approached on an ad hoc basis, in the main where
they had expressed an interest in part-time provision and/or institutional research
activity. Eleven institutions made up the final sample.
The development of the survey instrument
1 July 2008
Surveying the student experience …
4
A meeting was convened at Liverpool Hope University in early December 2007, whose
purpose was to develop amongst the participants a shared understanding of the survey
and what it could reasonably be expected to achieve. It would also lead to agreement
on the nature of ‘the deal’ between the participants, including how data would be shared
and handled, and what principles would influence the reporting of the work to a wider
audience.
Although the original idea had been to focus attention on undergraduate-level study,
there was sufficient desire amongst the participating institutions to include postgraduate
taught programmes in the survey. These were therefore included.
The starting-point was the survey instrument prepared for the first-year experience [FYE]
survey (Yorke and Longden, 2007). This was deemed to be too long for the electronic
survey, and in any case its item-pool was not entirely appropriate. A subset of the FYE
items, to which was added some new items (mainly demographic), was considered by
project team members and iterated between them electronically until all ‘could live with’
the revision even if it was not exactly what they would have preferred.
‘The deal’ between the institutions and the project
With collaborative work, it is important to be clear from the outset as to the nature of ‘the
deal’ between the various parties involved in the work (see earlier comment). With this
in mind, documentation prepared for the initial meeting of the project team included a
paper dealing with such matters. There is always the possibility of further issues arising
as the project progresses, and it is important to be alert to them, and attend to them
without delay since mutual trust can be compromised if matters are not opened to
discussion or are allowed to drift.
Ethical issues
The survey was planned to align with the ethical guidelines published by the British
Educational Research Association (BERA, 2004). Participating institutions took differing
stances regarding the need to send the proposal for the survey through their ethics
committees. The majority took the view that the survey was part of the institution’s
normal quality assurance and enhancement operations, and that this was sufficient as
regards approval. Two, however, took the view that the involvement of human subjects
required them to seek formal approval from their ethics committees. Whilst this added
an extra complication to the organisation of the survey, it did throw up some useful
points that were incorporated into methodological practice (for example, making the
survey accessible in some form to all possible respondents). It was made clear that the
mode of reporting would be such that no individual or institution would be identifiable,
and that the only people to deal with the individual responses would be the project’s codirectors. A report of the survey would be made available to the students in the
institutions concerned.
Constraints
It was originally envisaged that the survey would be run during March 2008. This would
allow students in their first year to comment on feedback on work – an issue that has
consistently been shown to be of concern in responses to the NSS. However,
institutions would be at that time involved in encouraging final-year students (some of
1 July 2008
Surveying the student experience …
5
whom would be part-time) to complete the NSS, and it was decided to defer the
administration of the survey until the following month, after the (unusually early) Easter
break. It had to be left to institutions to decide exactly when invitations to participate
would be sent out, since local considerations (such as term dates and internal surveys)
had to be taken into account. It was hoped that the vast bulk of responses would be
received by 9 May, but ‘late entries’ would be accepted for the final analyses. In
practice, some variation in the dates of e-mailing invitations and in receiving responses
would be unlikely to have a significant impact on the results.
Some initial results
The dataset that had accumulated by 9 May 2008 was cleaned of duplicated and other
questionable responses. The results reported below are drawn from this dataset, and
are interim. At the time of writing the dataset is continuing to grow, and is expected to
near 3,000 by the time that the survey is closed. There are currently 815 responses
from taught postgraduates, 854 from first-degree students, and 357 from students on
foundation degrees, HNC/D programmes and other sub-degree courses. In Table 2, the
interim data from first-degree and other programmes at undergraduate level are
compared with those from Phase 1 of the study of the first-year experience: the
postgraduate experience is arguably rather different.
1 July 2008
Surveying the student experience …
6
First Deg
Sub-Deg
FYE
N = 854max
N = 357max
N=7109max
Staff made it clear from the start what they expected
The teaching on my programme has shown me what I need to do to be successful in my studies
I am satisfied with the quality of the teaching on my programme
I am finding my programme intellectually stimulating
3.78
3.72
3.59
3.97
3.73
3.57
3.48
3.78
3.89
I have been able to contact academic staff when I needed to
I am satisfied with the level of tutorial support on my programme
3.74
3.48
3.72
3.42
3.69
The criteria used in assessing work have been made clear to me
I have received detailed comments on my work
Feedback on my work has been prompt
Feedback on my work has helped me in my learning
The way in which programme assessments (...) are scheduled causes problems for me
3.83
3.30
3.02
3.43
3.20
3.66
3.33
3.08
3.35
3.20
3.53
My programme is well organised administratively
3.10
3.08
3.62
I haven’t done the background reading expected of me
I find it difficult to balance academic and other commitments
I am not able to attend some formally timetabled sessions because of other demands on my time
Academic work at this level is harder than I expected it to be
I am coping with my academic workload
3.56
2.55
3.16
3.28
3.67
3.50
2.49
3.25
3.27
3.54
3.09
I have made at least one close friend at this institution
I discuss academic work with fellow students outside timetabled sessions (by one means or another)
I feel that I belong to an academic grouping (e.g. department; school) within this institution ##
3.66
3.62
3.19
3.67
3.78
3.16
4.27
My institution’s student support services have been sufficient for my needs
The library provision is good enough for my needs
I have been able to access general computing resources at the institution when I needed to
The institutional catering facilities are adequate for my needs
3.20
3.55
3.76
3.19
3.15
3.60
3.69
3.44
3.31
I worry about financing my way through higher education
I find it difficult to study in my home
2.97
3.26
3.09
3.15
2.44
I am getting what I expected from my programme
I would recommend my part-time programme to a friend
3.57
3.69
3.43
3.52
n/a
Item
3.74
3.67
3.69
3.52
3.18
3.09
3.47
3.28
2.92
n/a
2.94
3.33
n/a
2.47
3.94
4.01
n/a
n/a
n/a
Table 2.
Comparisons of interim findings from the survey of part-time students with
those from the study of the first-year experience.
Notes.
All items are scored such that higher ratings indicate a more positive perception to the
matter at issue.
Items in bold are identical in the two surveys.
Items in normal font are closely similar. The item on academic grouping (marked ##) was
stated negatively in the FYE study.
Items in italic are particular to the survey of part-time students. n/a = not asked.
Bearing in mind the provisionality of the data in Table 2, the following points seem to be
emerging.
 Part-time provision is perceived as less well organised than first-year
programmes.
 First degree PT students in particular seem to find their programmes stimulating.
 Feedback issues seem to be common to both studies.
 PT students appear to have been more diligent regarding background reading.
1 July 2008
Surveying the student experience …





7
PT students have greater difficulty in balancing academic and other
commitments.
PT students appear to be better at coping with their workload.
Whilst friendship formation is less strong for PT students, they seem to have a
stronger sense of ‘belonging’ to an academic grouping (but particular caution is
needed on the latter point because of the reversal of the questionnaire item).
Library and computing resources are less highly rated by PT students
PT students are more positive about financing their way through higher
education.
From the demographic data, around three quarters of undergraduate respondents
indicated that they had opted for part-time study because of the need to study alongside
other commitments. Fee issues and flexibility were, for a minority, co-reasons. The
source of tuition fees was roughly evenly divided between the students and their
employers. However, the provision of funding for ancillary expenses fell overwhelmingly
to the students themselves, with only around 1 in 8 indicating a contribution from an
employer.
The questionnaire asked respondents to indicate the best and worst aspects of their
experiences to date as a part-time student. As is evident from Table 2, the bulk of
student experiences were positive – sometimes very strongly so (space considerations
preclude their inclusion in this paper). However, and arguably of greater importance for
the enhancement of the student experience, there were some strongly-expressed
negative comments:
1) A large proportion of the students on this course are part time and hold
down full time jobs, but a lot of tutors/admin support do not take this into
account and consequently have unrealistic expectations of the amount of
time that we can devote to the course outside of the scheduled lectures.
2) Disorganisation - timetables being released on a Friday when the next
semester starts the following Monday does not allow time for work
commitments to be rearranged to accommodate the course.
Female, 26-30, Architecture, Building & Planning, preparing for promotion
Often I have felt excluded from the mainstream full-time course, particularly
in the beginning when very little guidance or information was available on
part-time study. […] I have also experienced issues such as being refused
library books because I was only studying 'part-time'.
Female, 36-40, Social Studies, improving capability in current employment
Assignment feedback: This was a common problem throughout 2nd and 3rd
years with no written or personal feedback on the majority of the
assignments set. One module had the assignment topic the same as that for
the final exam. The results were so late in that if you did badly in the
assignment there was no time for damage control before exam
Male, Over 60, Business & Administrative Studies, studying for personal satisfaction
Commentary
Interesting as the emerging data may be, the focus of this paper is on methodology
rather than on empirical data.
1 July 2008
Surveying the student experience …
8
Inter-institutional collaboration on studies relating to their provision is probably underexploited as a method. The advantages include:
 The enhancement of understanding that accrues from simply discussing relevant
issues
 The benchmarking potential of comparing one’s own institution’s results with
those of others (even if, for reasons of sensitivity, these are presented in
aggregated form)
 More directly, the identification of aspects of provision that could be improved,
and the enhancement activity that would follow
 The potential capacity-building as regards the handling of statistical data and,
more generally, institutional research
 The ‘value for money’ aspect, in which a relatively small outlay on the part of an
individual institution has the potential to produce information, and to develop
capacity, to the benefit of the institution.
In a self-funded project of this kind, there is no requirement (explicit or implicit) to bear in
mind the particular interest that might be espoused by a funding organisation – but the
downside of this is that constraint on the funding actually available imposes limits on
what might be achieved.
The support of participating institutions, largely on an ‘in kind’ basis, has made possible
a survey that otherwise could not have been conducted because of a lack of funding.
The general methodology outlined in this paper is potentially adaptable to a variety of
topics which might be addressed through institutional research, whether on an intrainstitutional basis or on the basis of collaboration between consenting (and in practice
probably cognate) institutions.
References
Bourner T with Reynolds A, Hamed M and Barnett R (1991) Part-time students and their
experience of higher education. Buckingham: SRHE and Open University Press.
BERA (2004) Revised Guidelines for Educational Research, 2004. At
www.bera.ac.uk/publications/pdfs/ETHICA1.PDF .
Callender C, Wilkinson D and Mackinnon K (2006) Part-time students and part-time
study in higher education in the UK. Strand 3: a survey of students’ attitudes and
experiences of part-time study and its costs 2005/06. London: Universities UK.
HEFCE (2007) Review of the teaching funding method: second consultation on changes
to the method. Bristol: HEFCE.
Schuller T, Raffe D, Morgan-Klein B and Clark I (1999) Part-time higher education:
policy, practice and experience. London: Jessica Kingsley.
Surridge P (2007) The National Student Survey 2006. At
www.hefce.ac.uk/pubs/rdreports/2007/rd14_07/ .
1 July 2008
Surveying the student experience …
9
Yorke M and Longden B (2007) The first-year experience in higher education in the UK:
Report on Phase 1 of a project funded by the Higher Education Academy. Available at
www.heacademy.ac.uk/assets/York/documents/ourwork/research/FYE/FirstYearExperie
nceRevised.pdf .
Yorke M and Longden B (2008) The first year of experience of higher education in the
UK. York: The Higher Education Academy. Available at
www.heacademy.ac.uk/assets/York/documents/resources/publications/FYEFinalReport.
pdf .
1 July 2008