HESA for Planners Objectives • Identify best practice around quality assurance and use of data • Improve our understanding of check documentation and how it can be utilised • Introduce the downloadable files and how they can be used • Outline the future information landscape • Better understand the IRIS outputs • Learn from each other “You never finish HESA, you abandon it” Utilisation of time – ‘opportunity cost’ Efficient and cost-effective procedures Best practice Collaborative approach to data Resource Systems that work for the organisation How to be good… • Data ownership: - Systems (storage issues) - People • Translation: - From HEIs internal data language to an external data language - The extent to which these match - The variety of external languages that an HEI has to work with • Documentation - How, who, when • Education - Value of data and transparency Evidence (or anecdote) from the KIS • New requirement – high profile • Data spread across institutions – – – – – No documentation Little/no control No standardisation/comparability Variable quality Variable approaches to storage • …being assembled and managed in spreadsheets Spreadsheets • Often created by people who don’t understand principles of sound data management • Conflate data and algorithms • Almost impossible to QA • Spread and mutate like a virus Search “Ray Panko spreadsheets” The institutional perspective HESA for planners Wednesday, 15 May 2013 Alison Hartrey Head of Planning SOAS, University of London HESPA – what we do • Higher Education Strategic Planners Association provide a forum & network to discuss shared areas of interest provide a focus for planning input to statutory bodies etc. promote awareness and understanding of planning issues contribute to the professional development of planners link with planning & associated communities 11 HESPA - governance • Executive drawn from across the sector • includes representation from England, Scotland & Wales • post-1992 and pre-1992 institutions • various mission groups and independent HEIs • working towards a full transition under the PHES umbrella • flat rate membership fee per institution (2013/14) • NATIONAL-PLANNERS-GROUP@jiscmail.ac.uk 12 HESA – planning role at SOAS • Direct responsibility for: • Student Record • Aggregate Offshore Record • Institutional Profile Record • Key Information Set • Oversight of other HESA returns and ability to download check documentation and data supply • Produce NSS data for Ipsos MORI and supply DLHE data to the University of London Careers Service 13 HIN target list and UCAS files • Used when we are adapting our “as delivered” HESA extracts to ensure the correct data is being extracted • UCAS files (*J and level 3 qualifications expansion file) used as a double-check against the qualifications records held in the student records system 14 Check documentation • Used when answering Minerva queries • Used for basic data checks (cost centres, major source of tuition fees) • Used across years for credibility checks • Used for FOI responses 15 Data supply • Used in a variety of institutional reporting to ensure consistency of student population across years – in particular • annual review & financial statements • OFFA/widening participation reporting • equality & diversity reporting • annual programme review updates • FOI requests …. but we could use it for a lot more 16 IRIS • Another check on cost centres and student load (HESES recreation) • Used for a wide variety of HEFCE checks and HEFCE funding streams - RDP, SNC, TRAC(T), NSP • Tendency to concentrate on just the reports needed at the time due to concentration on the HESA Student Record and the time lines 17 HESA performance indicators in HE in the UK • Underlying data from the HESA performance indicators is written back into the core student record system: LPN, state school, previous HE, entry qualifications, SEC, region of domicile, young/mature, mode, level • Used by Planning and the Widening Participation Team for onward analysis and reporting • Allows us to more easily compare whole population and special interest groups as well as assess the impact of initiatives across a time line 18 Embedding HESA into institutional thinking – a rough guide HESA for Planners, May 2013 John Busby, Deputy Director of Corporate Planning The contention Institutions that embed HESA and HESA requirements in their thinking are more likely to: submit higher quality data; extract more value from HESA data to inform their development (benchmarking, business intelligence, …). be better place to respond where policy impact/funding is derived from HESA data (HEFCE SNC, NSP …) Thinking = people / resources, skills, policy, standards, processes (and attitude?) HESA = in the broadest sense (not just Student, and a provider of information as well as a collector) Signs of embedding Senior Management awareness and interest (depth of involvement in institutional sign off, reporting) Close working between compilers of HESA Student, HESA Staff and HESA Finance-HEBCIS, other exercises (e.g. KIS, TRAC(T), OFFA, REF), Finance, Planning, HEFCE liaison Widespread use of HEIDI, purchases of bespoke data by institution or amongst groups of institutions to develop BI HESA collections seen as a requirement but also an opportunity (institutional reflection, comparative data) Understanding permeates academic departments / schools / faculties A self-assessment Last report to most senior governance body on trends from HESA data? Extent of involvement in sign-off? (Senior Management awareness) Last time HESA Student and Staff compiler met, or met with the TRAC(T) compiler, or REF team? (Close working) Last time HESA data from HEIDI was warehoused, or a bespoke data set specified and purchased? (Use of data) Last time collection of a HESA field prompted institutional reflection? (HESA as opportunity) Last time a member of academic staff was helped to understand HESA data? (Widespread knowledge) Potential obstacles Pace of change and team workloads Institutional boundaries between key functional areas (Registry, Planning, Finance, HR, QA function ...) Disconnect between the business processes on which data quality depend and those that compile returns and interpret derived data HESA compiler hidden in the ‘broom cupboard’? Lack of senior management buy-in What may help? HESA integration or HESA co-ordination group (formal, informal, self-help) Consider physical location of key systems teams and HESA compilers around Planning / Finance Promote senior management awareness through: Development of insightful business intelligence and tools from HESA data (internal reflection and comparator benchmarking) Emphasise league table impact Talk to your academic community about your data (social science researchers, ...) Examples of using HESA data (Student) Examples of using HESA data (Student) Examples of using HESA data (benchmarking) Some closing observations Warehouse the HESA Student data supply tables from the data collection system (helps with interpretation of DDS, derived data exercises from HEFCE, FOIs) Try the HEIDI API Look forward to the new Cost Centre structure Audits and auditors can be extremely useful Ensure that there is joined up discussion in your institution of the expanded Institutional Profile return Please support HESPA! Any questions? John Busby, john.busby@york.ac.uk Discussion • Discuss the following as a table: • How good are you at HESA? - Consider factors such as data ownership, documentation, staffing, knowledge, resilience, training, systems, data quality process – how extensive and sophisticated it is. How often you use HESA data and what for and how is the process and data managed/structured internally. What are the barriers and how do you overcome them? • Now consider and rate your own institution: 1st class 2:1 2:2 3rd Unclassified Using check documentation What is check documentation? An Excel workbook which displays the data in a series of tables Used by analysts at HESA for quality assurance Available after any successful test or full commit Why should I use it? Check documentation gives an overview of the submitted data which can help identify potential issues Provides context to the queries raised by HESA The institution will be able to spot anomalies that HESA would not Comparison feature also useful for later commits/test commits to monitor changes Check doc is one of many reports and is best used in conjunction with other reports Task 1. In your groups, or individually, complete check documentation tasks 1-4 How can check doc be used? • Use the check documentation guide produced by HESA as a starting point • Many of the items provide year on year comparisons: Using check documentation Different populations and groupings are used for each item in the check documentation, including derived fields For 2012/13 the definitions sheet has moved to the coding manual Who are those 5 students?? • To get the most out of check documentation and work out whether something is an error, you need to identify the records behind the table • To do this you can use Data Supply which contains much of the raw data submitted alongside the derived fields used by HESA • Pivot tables can be used to recreate items and identify particular cells Identifying students: • The HESA for Planners manual contains instructions on recreating the populations and conditions used in check documentation • As an example we will recreate item 6a ‘Student cohort analysis’…. Check doc changes for 2012/13 • Revised tolerances • Items 1, 2 & 3 will now highlight year on year changes of +/- 10%/50 students • Item 11 will look at sector averages rather than just the previous year • Move to JACS3 and new cost centre coding frame • New Fees tab • More detailed breakdowns, summations and percentage changes added to enable checking Item 2a - Qualifications awarded What is the difference between 2 and 2a? Item 2 Item 2a Shows the qualifications awarded to students in the format that will be published in the student volume Displays the year on year differences using the qualifiers field of XQLEV501 (including the split out of PGCE and Post grad cert in Education. Used to check that the qualification awarded are, in the main, those which they were aiming for More consistent with the SFR variance figures Item 2a - Qualifications obtained by students on HE course by level of qualification obtained and mode of study (2012/13 and 2011/12) Item 7 – Highest qualifications on entry • Now split into 7a & 7b ‘proportion of highest qualification on entry for first years’ • Subtotals also added to item 7a Item 12 – average instance FTE • This item has been broken down further to provide a three way split of starters, leavers and ‘others’. • The different groups may have very different FTE values that impact the average Other reports Minerva …is the data query database operated by HESA • During data collection HESA (and HEFCE) raise queries through Minerva and institutions answer them • These responses are then reviewed and stored for future use by HESA and the institution Data submitted Quality assurance Queries raised Queries resolved Sign-off Using Minerva for quality assurance • Responses from previous years are retained in the Issue Report • Review targets set for the current year • Queries raised by HESA are prioritised: Contextual Intelligence • At the request of the National Planners Group HESA have formulated a ‘public’ version of Minerva • Designed to give users of the data additional context • HESA has published a query to Minerva to which HEIs can add notes about their institution e.g. ‘we recently opened a new department’ • HESA will not interact with what is added • Will remain open throughout the year • HESA will extract and send the information to accompany data requests Using downloadable files Downloadable files • Data Supply (Core, subject, cost centre, module and qualifications on entry tables) • NSS inclusion (person and subject) and exclusion files • POPDLHE • TQI/UNISTATS • All available after every successful full and test commit Using downloadable files • - The files should be utilised to: Carryout additional DQ checks Benchmarking Planning/forecasting Improve efficiency (recreating data from scratch unnecessary) League tables • Student staff ratios by institution and cost centre • First degree (full-time for Guardian) qualifiers by institution and league table subject group • Average total tariff scores on entry for first year, first degree students by institution and league table subject group. • Data is restricted to tariffable qualifications on entry (QUALENT3 = P41, P42, P46, P47, P50, P51, P53, P62, P63, P64, P65, P68, P80, P91) (Times applies ‘under 21’ restriction, Guardian applies ‘full-time’ restriction) • Full-time, first degree, UK domiciled leavers by Institution, League table subject, Activity • Full-time, first degree, UK domiciled leavers entering employment • Graduate employment/Non graduate employment/Unknown • Positive destinations /Negative destinations • Expenditure on academic departments (Guardian) • Expenditure on academic services (Guardian, Times, CUG) • Expenditure on staff and student facilities (Times, CUG) Who cares? Why bother? • …because you can’t afford not to care • In what space does student recruitment take place? • Extended coverage… • Subject based… • Supply and demand are linked to measures of quality • Internationalisation of Higher Education But be aware of the tail wagging the dog… • Collectively we can become obsessed about specific measures… • …and dangerously on the wrong type of measures… • …and instead of good (or accurate) ranking being born out of doing your day to day business well, it becomes fuelled by quick fixes • Measurement culture tends to trade long-term value for short-term gains… • …this holds true in ‘data world’ But there are gains to be sought – both in terms of quality and benchmarking “Firstly, you need a team with the skills and motivation to succeed. Secondly, you need to understand what you want to achieve. Thirdly, you need to understand where you are now. Then, understand ‘aggregation of marginal gains’. Put simply….how small improvements in a number of different aspects of what we do can have a huge impact to the overall performance of the team.” Dave Brailsford, Performance Director of British Cycling But there are gains to be sought – both in terms of quality and benchmarking “Firstly, you need a team with the skills and motivation to succeed. Secondly, you need to understand what you want to achieve. Thirdly, you need to understand where you are now. Then, understand ‘aggregation of marginal gains’. Put simply….how small improvements in a number of different aspects of what we do can have a huge impact to the overall performance of the team.” Dave Brailsford, Performance Director of British Cycling Is there a correlation between this spread and league table positioning? Before you begin… • Remember the different populations • Use derived fields (those beginning with an X!) • The INSTCAMP field can be used to better analyse and understand your data Demonstration… Performance indicators • The PI tables (available from the HESA website) give sector wide data on: - Non-continuation rates - Widening participation of under-represented groups and those in receipt of DSA - Research output - Employment of leavers • Included are benchmarks and definitions DDS • http://www.hesa.ac.uk/content/view/2664 Scenario planning The impact of fee increases on applications Total applications 05/06 – 11/12 heidi 60,000 50,000 40,000 The Nottingham Trent University 30,000 Sheffield Hallam University 20,000 10,000 0 04/05 05/06 06/07 07/08 08/09 09/10 10/11 11/12 % change of total applications heidi 20.0 15.0 10.0 5.0 0.0 05/06 06/07 07/08 08/09 09/10 10/11 -5.0 -10.0 -15.0 -20.0 -25.0 The Nottingham Trent University Sheffield Hallam University 11/12 Total applications by regions heidi 300,000 250,000 200,000 Total Yorkshire & the Humber 150,000 Total East Midlands 100,000 50,000 0 04/05 05/06 06/07 07/08 08/09 09/10 10/11 11/12 NSS 05/6 results versus 06/7 applications heidi Student:Staff ratios 2005/06 % change in applications by subject for sector 6 4 Nursing 2 Education 0 -2 -4 Creative arts -6 -8 Law -10 Mass communications -12 Total applications % change 2006/07 2006/07 Subject profile Nottingham Trent University Check documentation Sheffield Hallam University Performance indicators 2006/07 2005/06 Building condition Total Non-residential - condition A & B 90 80 70 60 50 40 30 20 10 0 The Nottingham Trent University Sheffield Hallam University 2005/06 Building condition Total Non-residential - condition A & B What we ‘know’… - Some subject areas are more price elastic than others? - Applicants take note of NSS? - Condition of the estate matters to applicants? - Some socio-economic groups are more affected by fee increases than others • Each of these variables might have a value of x number of applicants What we don’t know… • - …but could scenario plan for… Future government policy on HE funding Social/cultural/economic impact The power of perception NSS 2011/12 Overall satisfaction 90 80 70 60 50 40 30 20 10 0 Nottingham Trent Unviersity Sheffield Hallam University Student:Staff ratios 2011/12 Subject profile Nottingham Trent University Check documentation Sheffield Hallam University Performance indicators 2011/12 • Was 2.3% difference, now 0.2% 100 90 80 70 60 50 40 30 20 10 0 2005/06 Building condition Total Non-Residential - Condition A & B The Nottingham Trent University 2010/11 Building condition Total Non-Residential - Condition A Sheffield Hallam University heidi - % change of total applications 20.0 15.0 10.0 5.0 0.0 05/06 06/07 07/08 08/09 09/10 10/11 -5.0 -10.0 -15.0 -20.0 -25.0 The Nottingham Trent University Sheffield Hallam University 11/12 heidi - % change of total applications 20.0 15.0 10.0 5.0 0.0 05/06 06/07 07/08 08/09 09/10 10/11 -5.0 -10.0 -15.0 -20.0 -25.0 The Nottingham Trent University Sheffield Hallam University 11/12 12/13 Aggregation of marginal gains The power of perception • …can be influenced by the power of data http://www.youtube.com/watch?v=ZWTJ_TPraLQ • If you don’t like what they’re saying, change the conversation… • …what data are you using on the website and is it the right data? • …repositioning - find what you are good at and sell it (both internally and externally)…. • …but never neglect what you need to improve • http://www.ucl.ac.uk/about-ucl ` Higher Education Information Database for Institutions Estates Destinations Finance HE-BCI Student Staff Applications Equality National student survey Derived statistics Available to all HEIs heidi.hesa.ac.uk Capabilities Collate, crossreference and interpret information View, create and export reports, charts and custom tabulations Generate aggregations, ratios and percentages Benchmark the performance of your institution against others Use heidi data within your own business intelligence software Embed heidi reports and charts into your own website(s) Adjust the year or the group from within the report Notes and definitions provided Sharing • New to heidi • Allows reports and charts to be shared with others – Including non-heidi users • Share by email using the ‘send to’ link • Use the HTML code to embed reports or charts into websites Download to any version of PowerPoint Adjust the year or the groups from within the chart Use groups and sub-groups to compliment analysis of charts Charts New Radar Chart Application Programming Interface • Does your HEI have it’s own data warehouse or BI system? • Use API to specify wanted heidi data and retrieve in a usable format • API is aimed at users familiar with writing and understand programming code. HESA can provide support to colleagues involved with API at your institution The future Re-designed DLHE data set Federated user accounts Benchmarking functionality We are always keen to hear feedback, especially ideas and suggestions for future release of heidi. Please send any comments to heidi@hesa.ac.uk Sign up heidi account • Contact your Local Administrator for access to heidi – Expert account allows access to all features – Standard account allows access on a view mode heidi@hesa.ac.uk JISC • Mailing service which allows heidi users to make new contacts, ask questions and share knowledge and best practice www.jiscmail.ac.uk Welcome slide HESA for Planners (Student Record) Seminar Anthea Beresford – Data Assurance Consultant, HEFCE May 2013 HEFCE’s Data Assurance Team activity April 2013 to March 2014 The aim of this session is to advise you of: • areas of HEFCE’s Data Assurance Team’s coverage from April 2013 to March 2014. Data assurance activity of the Data Assurance Team • team of 3, supplemented by a consultant; • we are part of the overall data assurance framework; • annual audit plan agreed by our Funding Round Process Board (internal Executive oversight) and our Audit Committee to which we report regularly and provide an annual report on audit outcomes; • ever changing activity, for example, as funding rules change and new areas become important; • interested in both funding and non-funding issues with data. Core roles of the Data Assurance Team 3 distinct types of activity: • data audit; • data verification; • data reconciliation. Areas identified for attention from April 2013 to March 2014 • Student funding data work: • HESA student data verification work pre-sign-off; • HESES verification work pre-sign-off; • 2011-12 outturn review of FT UG HEFCE-funded student non-completion rates. • Research Funding data work: • Research income from Charities; • Research income from Business; • Research HESA student data exploratory work. • Key Information Set (KIS) 2013/14 • Destination of Leavers from Higher Education (DLHE) 2011/12 Areas identified for attention from April 2013 to March 2014 (cont.) • National Scholarship Programme (NSP) • 2011-12 Funding and Monitoring Data (FAMD) (reconciliation) exercise • BIS Service Level Agreement work: • Access to Learning Fund (ALF). • Higher Education Business and Community Interaction Survey (HEBCIS) • Student Number Control (SNC) • Equivalent and Lower Qualifications (ELQ) Data audit – established work • 2011-12 outturn review of FT UG HEFCE-funded student noncompletion rates: • desk based review of a 5% random sample of FT UG HEFCE-funded students. • National Scholarship Programme (NSP): • desk based request for explanations of differences between HESES11, HESA 2011-12 and HESES12 new entrant student numbers. • Research income from Charities and Business; Key Information Set (KIS) 2013/14; Destination of Leavers from Higher Education (DLHE) 2011/12; Access to Learning Fund (ALF): • Pre-audit visit review of data; on-site visit; post visit follow-up; issue of audit report with recommendations; completed action plan for approval; closure of audit; implementation of funding adjustments following the Appeals process, where relevant. Data audit – established work • Current audit programmes can be found at: http://www.hefce.ac.uk/whatwedo/invest/institns/funddataaudit/dataaudit/ • Current audit reports can also be at that link. Note the new KIS 2012/13 report. Data audit – developmental work • Research HESA student data exploratory work; Higher Education Business and Community Interaction Survey (HEBCIS); Student Number Control (SNC) and Equivalent and Lower Qualifications (ELQ): • Pre-audit visit review of data; on-site visit; post visit follow-up; issue of audit report with recommendations; completed action plan for approval; closure of audit; implementation of funding adjustments following the Appeals process, where relevant. Data verification • HESA student data verification work pre-sign-off: • Desk based; • Working in conjunction with HESA; • Querying institutions on their data during the student data collection period to assist institutions in identifying potential data issues for correction before sign-off; • We will publish guidance nearer the time at: http://www.hefce.ac.uk/whatwedo/invest/institns/funddataaudit/dataverifi cation/ • HESES verification work pre-sign-off: • Desk based; • Working with institutions between initial submission and sign-off, obtaining explanations for data differences or changes to data. Data reconciliation • Reconciliation between HESES11 and HESES11 re-creation based on HESA 2011-12 data: • Desk based; • Thresholds for selection; • 2 stage process this year due to students with undetermined completion status (FUNDCOMP=3). The ‘Completion Status Survey’ is currently underway where we are asking institutions to update their completion status information for those who were returned as FUNDCOMP=3 in their 2011-12 return. The deadline for sign-off for this is 19 July 2013. We have therefore selected institutions who currently break the selection criteria thresholds. All institutions will be looked at again following submission of the Completion Status Survey; • Gain explanations of differences; • Action plan and amendment of data as necessary; • Formal sign-off by institution; • Implementation of any funding adjustment following an Appeals process. Data reconciliation (cont.) • The link to guidance on our website concerning this area of activity can be found at: http://www.hefce.ac.uk/whatwedo/reg/assurance/datareconciliations/ Finally…. Any questions? Thank you for listening a.beresford@hefce.ac.uk The HE information landscape Update Recommendations to RPG • Governance for data and information exchange across the sector • Development of a common data language – Data model, lexicon, thesaurus • Inventory of data collections • Specific data standards work – JACS – Unique Learner Number 2. Data model, lexicon and thesaurus • Review of existing collections/definitions – the as is • Better understanding of differences/similarities – In definitions – In terminology • Coming from both angles: – What are collectors asking for? – What are institutions supplying? • To inform future standardisation and data sharing discussion • Deliverables: – Data model, lexicon and thesaurus – Maintenance plan 3. Inventory of data collections • • • • HEBRG survey identified 550 lines of reporting Very little detail (width) Is it complete? (length) We need a solid understanding of the current burden – To help HEIs become more joined up in their reporting – To challenge data collectors to reduce duplication • Deliverables: – Database of collections – Maintenance plan 4a JACS development • Problems: – JACS could have far broader use – Current structure has run out of space • Analysis of requirements • Exploration of coding options • Deliverable: – Road map for future development 4b ULN implementation • ULN widely accepted as a Good Thing – Reducing burden by replacing existing IDs – Adding value through better data linking/sharing • What are the real barriers to adoption? • What would it take to resolve these issues? • Deliverable – An assessment of where we currently are with ULN – Commitment to a roadmap? Governance? • What will it do? • What authority does it have to progress actions? • How will the work be delivered and coordinated? Proposal • • • • A programme of work… …made up of specific projects… …overseen by a Programme Board… …and reporting to a Sponsoring Group • Utilising best practice from Managing Successful Programmes Sponsoring group Chair of the Programme board Programme board Programme Management Office SRO/Programme Director Project A Project B Project C RPG Chair of the Programme board Programme board Programme Management Office SRO/Programme Director Project A Project B Project C RPG Chair of the Programme board Programme board Programme Management Office SRO/Programme Director Project A Project B Project C www.hediip.ac.uk @hediip HEDIIP • Enhance the arrangements for the collection, sharing and dissemination of data and information • Programme management office based at HESA • Publishing and maintaining the inventory of data collections • Carry forward the established projects – Common data language – Replacement for JACS – Implementation of the ULN • Other strategic developments Keep in touch If you require additional training help, including bespoke visits to your institution, get in touch with the training department… w: www.hesa.ac.uk/training e: training@hesa.ac.uk t: 01242 211472 Follow us on Twitter: @HESATraining