Strategic Use of Study Environment Surveys to Promote Change
K. J. Herrmann, Centre for Teaching and Learning, Aarhus University, Denmark, kh@clu.au.dk
A. J. Bager, Centre for Teaching and Learning, Aarhus University, Denmark, abager@clu.au.dk
Background
Using survey instruments to obtain student feedback about courses and programmes is common in North
American, British and Australian universities; however, evaluation often becomes ritual without promoting
any change (Richardson, 2005). While keeping students happy is certainly important, satisfaction in itself is
an ambiguous indicator of quality. The challenge is to identify salient indicators of important factors in the
study environment and to report results in a way that is likely to promote change.
Triennial surveys of the study environment are mandatory in Denmark; however, the survey
implementation is the prerogative of the individual educational institution. The Aarhus University Study
Environment Survey (AUSES) was developed in 2007 and revised in 2011 with the aim of measuring
students’ perception of the study environment. In the latter survey 11,401 students completed the
questionnaire. In this paper we report the conceptual basis of the questionnaire, main results, as well as
strategic considerations when reporting results.
Conceptual framework
The overall framework of the AUSES was inspired by Tinto (1975, 1999); his conception of the higher
education system as a disciplinary community, which the student has to adapt to; and the analytical
distinction between social and academic integration. It was furthermore inspired by the Course Experience
Questionnaire (Ramsden, 1991) linking perception of good teaching, clear goals and standards, and
appropriate workload and assessment methods with both student satisfaction and qualitative differences
in students’ approaches to learning. Finally, it was inspired by Biggs (2002) concept of constructive
alignment, that is, the extent to which learning goals, teaching/learning activities, and assessment criteria
are aligned.
1
Figure 1 – Conceptual framework
Background
Student characteristics
- Gender
- Age
- High School GPA
- Undergraduate/
master student
Perception of study
environment
Social environment
- Contact with teachers
- Social infrastructure
- Collaboration
Feeling of integration
Social integration
Well-being
Academic integration
Well-being and
satisfaction
Academic environment
- Alignment
- Feedback
- Physical surroundings
- Perceived workload
Figure 1 presents the overall conceptual framework in the AUSES. It shows that both the student’s personal
characteristics and the student’s perception of contextual factors such as contact with teachers, alignment,
and feedback are expected to affect the feeling of being integrated in the study, socially and academically.
The feeling of integration is expected to constitute an important part of students’ well-being and study
satisfaction.
The questionnaire and main results
Survey items were developed in 2007, revised in 2011, and will be revisited again for the 2014 AUSES. The
dimensionality of the questionnaire was examined using explorative factor analysis and the questionnaire
has been successful in discriminating between departments within each faculty. Table 1 gives an overview
over key dimensions along with sample items.
2
This section will shortly describe the content of the dimensions. By contact with teachers we mean the
students’ perception of the possibility of getting in contact with lecturers and that the lecturers seem
interested in students. The collaboration dimension covers whether the students are part of study groups
on a daily basis or during exam. By social infrastructure we mean the students’ perception of whether there
are an adequate number of social and academic activities to participate in along with the student being
content with the opportunities for engaging socially fellow students.
Table 1. Key dimensions and sample items, AUSES 2011
Perception of the social
study environment
Dimension
Sample item
Contact with teachers
The teachers that I have been in contact with generally seem
interested in the students
Collaboration
Do you form part of a study group or do you have a study buddy on a
daily basis?
How do you assess the possibilities for social contact with fellow
students?
Requirements for tests and exams are clearly defined and
communicated
Social infrastructure
Perception of the academic
study environment
Alignment
Feedback
Workload
Feeling of integration
Satisfaction
Physical surroundings
Academic integration
Social integration
Satisfaction and wellbeing
There are good possibilities for getting feedback in relation to my
academic performance
How often do you fail to complete study-related work that you should
have done?
When I need a study carrel, I am mostly able to find one
My studies have made me feel part of an academic community
My fellow students are generally kind and cooperative
I generally feel content with my studies
The alignment dimension covers the perception that courses are appropriately organised with a logical
connection between learning objectives, exam requirements and teaching activities. Feedback is the
students’ perception of the possibility of getting guidance and feedback on academic performances. The
workload dimension contains the students’ perception of workload and satisfaction with their own effort.
In the questions about physical surroundings students answer whether it is possible to find a suitable place
to study and work in groups when needed. All the above mentioned factors are perceptions of the study
environment that contribute to the feeling of being integrated (see fig. 1). The integration dimensions are
described further below.
By being academically integrated we mean that the interest in the subject areas covered by the study
programme has increased in the course of the studies, and that the student feels part of an academic
community. By socially integration we mean that the students perceive peers to be open, friendly and
cooperative; that they feel being part of the social student community and do not continuously feel lonely
in their studies.
Using statistical regression analysis to explain variance in student well-being and satisfaction, we found
significant and positive correlations between student overall satisfaction and a) a positive experience with
3
teaching staff, b) the students evaluation of the possibilities to meet and socialise with peers at university,
c) the amount and quality of feedback on assignments, d) a perception of alignment between learning
requirements, assessment criteria, and teaching/learning activities. We also found, that the factors most
strongly affecting well-being – irrespective of study programme, personal background and study situation
(undergraduate or master student) – were 1) the feeling of being academically integrated, and 2) the
feeling of being part of a greater social community.
Strategic use of the survey
For the AUSES to promote change it had to 1) be perceived by readers as a valid instrument within an
academic discourse, 2) identify salient issues that needed attending to, 3) report results at the level
responsible to and capable of changing the study environment, and 4) establish a reference against which
results could be interpreted.
Law
86%
86%
81%
Political
Science
Economics
84%
Psychology
BS 2011
I generally feel very comfortable with
my study programme
BS 2007
With readers of the results being fellow researchers, it was important that results would be considered
legitimate within the university context. Several steps were taken in this respect: a separate chapter
described the theoretical basis of the questionnaire; a technical report was published including descriptions
of the full questionnaire, the data collection procedure, and response rates; overall confidence intervals
were calculated to show the
Table 2. Example of how to report results
precision of results; nonresponders were compared
to responders to detect any
sampling bias; questionnaire
items were analysed with
Satisfaction and well-being
exploratory factor analysis.
82%
86%
To identify salient issues,
items were pooled into
How satisfied are you with your study
84%
83%
84%
86%
80%
90%
programme on the whole?
summative scales each
describing a theoretical
Learning objectives and alignment
dimension such as social
The learning objectives for the
integration. We then ran a
individual courses are clearly
42%
64%
65%
64%
57%
65%
formulated and clearly communicated
statistical regression analysis
with student satisfaction as
The teaching methods chosen support
my learning regarding meeting learning
50%
56%
53%
54%
37%
53%
the dependent variable to
objectives and exam requirements
explore the relative
Contact with lecturers
importance of social
integration, feedback etc.
It is easy to establish personal contact
61%
69%
64%
62%
39%
69%
with most lecturers
Analysis showed very strong
correlations between student
...
...
...
...
...
...
...
satisfaction with their studies
and social and academic integration respectively. Correlations also documented the importance of working
in collaboration with peers; perceiving teachers and instructors to be friendly and easily approachable; the
4
provision of opportunities for students to socially interact; students being given feedback on their work.
From the perspective of an academic development unit it was especially interesting to see student
satisfaction being dependent on their perception of constructive alignment between course goals,
teaching/learning activities, and assessment criteria. This particular results suggest, that there might be a
spill-over from the effort to strengthen constructive alignment to student overall satisfaction.
Being aware that the full reports of 70-80 pages each would not be read by many, we had to come up with
a way of summarising results and Table 2 is an excerpt of such a summary for the Faculty of Business and
Social Sciences. The table gives a quick overview of results, which are described in greater detail later in
the report. Listing the departments in the columns makes it possible to benchmark ‘like units’, that is,
departments within the same faculty, sharing basic structural and disciplinary tenets. The colours were
attributed so that the better the results, the lighter the colour and vice versa. Thus in Table 2 it quickly
becomes apparent that the Department of Psychology should be especially interested in exploring further,
why their students perceive less alignment than students from other departments. In subsequent
discussion of the results these tables were often referred to and heads of departments would be concerned
with too many ‘dark colours’.
Initiatives as a result of the AUSES
By the end of the first AUSES in 2007 the survey went from being ‘just’ a legally required survey to a highprofile project with the university management’s attention, and following the 2011 AUSES a £1 million
pound fund was created to support projects aiming at promoting the study environment. Comparing the
data from 2007 and 2011 made it possible to evaluate both initiatives at the university level and the local
level; for example, the Department of Political Science did in fact manage increase the amount of feedback
on student assignments, and the AUSES data from 2007 compared to 2011 documented this. The
questionnaire has in part been adopted by other universities and the findings from the AUSES have also
been used as an onset to initiate small-scale research projects that have been published in the Danish
academic development journal.
Challenges and work to be done in the future
Multi-programme departments pose an increasing challenge when reporting results. For example, the
Department of Economics offers three rather different programmes. In this case it would be most precise,
valid, and useful to report results for each programme separately. On the other hand, the precision won
disaggregating results may come at the prize of the general overview. At the present, we are holding onto
the departments as unit of analysis but offer the service of providing departments and study boards with
analyses of individual programmes or levels of study (e.g., undergraduates vs. masters).
More work has to be done improve the construct validity of some of the scales. For example, the feedbackitems are restricted to feedback on assessed assignments. A more complete feedback-scale would include
feed-forward as well as the feedback students are receiving in tutorials and in their everyday study. Also, at
the present we are in the phase of interviewing students to develop items describing academic integration
5
from the students’ perspective. Academic integration is a complex construct because it has different
connotations and meanings depending on the disciplinary context.
Links and references
Link to Centre for Teaching and Learning, Aarhus University: www.cul.au.dk
Link to a full report translated into English: http://cul.au.dk/uploads/media/AUSES_Report_BSS.pdf
Link questionnaire items (in English) and factor analysis: http://cul.au.dk/uploads/media/AUSES_items.pdf
Link to survey website (in Danish): http://www.au.dk/studiemiljo2011
Biggs, J. (2003). Teaching for quality learning at university: what the student does. (2 ed.) Open
University Press.
Richardson, J. (2005). Instruments for obtaining student feedback. Assesment & Evaluation in
Higher Education, 30, 387-415.
Tinto, V. (1975). Dropout From Higher Education: A Theoretical Synthesis of Recent Research.
Review of Educational Research.
Tinto, V. (1997). Classrooms as Communities: Exploring the Educational Character of Student
Persistence. Journal of Higher Education, 68, 599-623.
Tinto, V. (2007). Research and Practice of Student Retention: What Next? Journal of College Student
Retention: Research, Theory & Practice, 8, 1-19.
Ramsden, P. (1991). A Performance Indicator of Teaching Quality in Higher Education: the Course
Experience Questionnaire. Studies in Higher Education, 16, 129-150
6