INSTITUTIONAL REPORTING
LEONE NURBASARI, ANU
PAMELA SARLY, ACU
HIEDI WILKINSON, USC
Institutional Reporting from the
Australian Graduate Survey
Leone Nurbasari
Planning and Performance Measurement
July 2014
Presentation Overview
•
•
•
•
•
•
Key Performance Indicators
Program (Course) Review reports
Discipline reporting on CEQ/PREQ
Graduate destination reporting
Comments analyses
Other reporting
3
Key Performance Indicators
•
•
ANU Strategic Plan
College Operational Plans and KPIs
4
KPI generation
•
•
Raw unit record data at 6 digit FOE, assigned to College scores based on
taught disciplines (rolling 5 year average, minimum 10% of load taught by
that College)
Benchmarking against Go8 data that is weighted according to ANU
discipline weightings
5
Example of some underlying 6 digit FOE data and calculations for KPI generation
6
Program (Course) Review Reports
•
•
•
•
Program reviews
Data warehouse
reports – CEQ,
Employment &
Further study
Supplemented by
additional
detailed analyses
(including item
level results)
Supplemented by
anonymised
open-ended
comments
7
Discipline reporting – CEQ/PREQ
•
•
•
•
Pseudo colleges (custom groupings of 2, 4 and 6 digit FOEs)
Pseudo departments (also custom groupings of FOE)
Benchmarking – Go8 and national, based on ANU taught FOE (excludes
non ANU disciplines)
Percentage of individual student-item response in
disagreement/neutral/agreement categories, aggregated to scales
8
9
Graduate destination reporting
•
•
•
Data warehouse
reports –
employment
rates, sector,
industry, salary,
employer, job
search
Cohort-specific
analysis
Ad-hoc data
requests
10
Comment analyses
•
•
•
CEQuery with charting via Excel
Various student cohorts and org. units
Adopting Geoff Scott’s method of proxies for
importance and quality
11
Other reporting
•
•
Topic Papers – comprising data
from various surveys and
student feedback sources, eg.
HDR Satisfaction Report
Ad-hoc reporting from internal
pivot tables – GDS, CEQ,
PREQ
12
Questions?
• Email Leone.Nurbasari@anu.edu.au
• Web http://unistats.anu.edu.au/surveys
13
AGS Data and Reporting at ACU
Pamela Sarly | Acting Manager, Statistical Analysis & Surveys
2014 Survey Manager Information Forum
17 July 2014
A bit about ACU…
ONE OF THE
FIRST
EDUCATION
PROVIDERS IN
AUSTRALIA
1857
Good
Samaritan
Sisters
Teacher
Training
in NSW and
VIC
1900
1963
Teacher
Training in
NSW, VIC,
QLD and
ACT
Merger of
education
colleges,
forming
ACU
1991
2014
One of the
fastest
growing
universities in
Australia
Australia's leading Catholic
university which is supported by
more than 2,000 years of Catholic
intellectual tradition.
12 Schools across 4 Faculties
490
6,263
21,934
826
Higher Degree Research Students
Postgraduate Students
Undergraduate Students
Non-Award Students
12,952 commencing students
16,561 continuing students
29,513
Enrolments
22,245
EFTSL
2,839 international students
Staff FTE
• 1,051 Academic
• 982 Professional
Data as at 11 July 2014
AUSTRALIAN GRADUATE SURVEY
Data Governance at ACU
• A central contact for surveys:
o Office of Planning and Strategic Management
• Publish aggregated data through internal sites:
o Staff Site (www.acu.edu.au/opsm)
o SharePoint site
• Data updates communicated to relevant staff via email
AUSTRALIAN GRADUATE SURVEY
Institutional Reporting
• Strategic Plan - Traffic Light Report
• Quarterly Report
• TEQSA Risk Indicators
• New Course Application
• Course Review and Renewal
TRAFFIC LIGHT REPORT
• Published in July and December
• Progress against the University’s Strategic Goals and Key Result
Areas
STRATEGIC SCORECARD
QUARTERLY REPORT
Strategic Plan Goal 1: Student Experience
•
Published in January, April, July and October
•
High level quantitative data that is linked to Strategic Goal 1 (Student
Experience)
AUSTRALIAN GRADUATE SURVEY
For the future…
•
Use of SPSS TAS for qualitative data analysis and reporting
•
Better linkage to Student Evaluation on Unit and Teaching surveys
•
Future of AGS – to inform our new Strategic Plan 2015+
Thank you
Questions?
Australian Graduate Survey
COMMUNICATING RESULTS FOR
EFFECTIVE DECISION MAKING
Note that the figures contained within this presentation
are fictitious.
Why do we need to communicate results?
AGS results are used at USC to:
 Inform strategic decision making
 Assess program and institution performance
 Influence improvements to programs
 Review learning and teaching
 Develop internal policies
Why use visualisations?
“Numbers have an important story to
tell. They rely on you to give them a
clear and convincing voice.”
Stephen Few
Choosing the
right medium
Textual
Visual
Oral
Tapping into
human nature
Over 50% of the
cerebral cortex is
involved with the
processing of visual
inputs.
-Lu & Dosher
There are over 130
million sight receiving
cells in the retina.
- Vries 2011
Source: Atranik 2011
Tableau
Knowing your
audience
USC’s Organisational
Chart
Know your
goal
Enable insight
"The purpose of
visualization is insight,
not pictures"
Shneiderman 1999
GDS Dashboards Flowchart
CEQ Dashboards Flowchart
Levels of Analysis
 Graduate Destination Survey Benchmark Group




Dashboard
Course Experience Questionnaire Longitudinal
Dashboard
Graduate Destination Survey Faculty Dashboard
Graduate Destination Survey Program (Course)
Dashboard
Course Experience Questionnaire Program (Course)
Dashboard
How
dashboards
are used
Are they being used?
Who is using them?
What are they using
them for?
 Director
 - Quick reference point
 Program leaders
 - Program reviews
 Executive
 - Marketing and recruitment
 - Identifying strengths and weaknesses
 - Information sharing
What next?
 Reports by Field of Education
 Beyond Graduation Survey
 University Experience Survey
 Internal Surveys
 Interactive Dashboards
 Qualitative Data Visualisations
 Atranik.org 2011, http://antranik.org/functional
Bibliography





areas-of-the-cerebral-cortex.
Azzam, T & Evergreen, S 2013, Data Visualization,
Part 2: New Directions for Evaluation, Number
140, Google eBook.
Card, S, Mackinlay, J & Shneiderman, B 1999,
“Readings in Information Visualization - Using
Vision to Think”, Morgan Kaufmann,
Massachusetts.
De Vries, J 2011, The Five Senses, Random House,
Victoria.
Few, S 2006, Information Dashboard Design: The
effective visual communication of data, O’Reilly,
California.
Lu, Z and Dosher, B 2013, Visual Psychophysics:
From Laboratory to Theory, MIT Press, ISBN:
9780262019453.
University of Leicester (Learning Development)
2012, Presenting numerical data,
<http://www2.le.ac.uk/offices/ld/resources>
If you have any
questions after
today:
hwilkins@usc.edu.au
Questions?