Abstract
Business intelligence software is ubiquitous in private industry to measure, monitor,
and communicate progress toward strategic goals; this process is being adopted in
higher education. This report explores the degree to which institutional research
offices are involved in institutions’ definitions and use of key performance indicators
and the selection and use of business intelligence software.
T a b l e o
o f C
C o n t e n t s Key Findings ................................................................................................................................................. 1 Section 1: Key Performance Indicators ........................................................................................................ 3 Institutional KPIs ...................................................................................................................................... 3 KPI Process ............................................................................................................................................... 4 KPI Categories .......................................................................................................................................... 4 Reporting KPIs .......................................................................................................................................... 5 Section 2: Business Intelligence Software ................................................................................................... 6 Choosing Reporting and Analytics Software Tools .................................................................................. 6 Evaluating Software Tools for IR Work .................................................................................................... 7 Section 3: Changing IR Role ......................................................................................................................... 9 Section 4: IR Budget ................................................................................................................................... 11 Appendix .................................................................................................................................................... 13 Survey Methodology and Respondent Characteristics .......................................................................... 14 Testing of Differences in Means ............................................................................................................ 15 Survey Instrument ................................................................................................................................. 16 I n t r o d u c t i o n As higher education institutions focus on performance improvement, business intelligence garners increased attention, and the professionals who manage it are more visible. We now know that vast stores of data amount to a cacophony without the tools and expertise needed to strip away the noise and find real meaning that informs decisions. So how are institutions finding that meaning? What investments are they making in business intelligence and performance management? Who are the people most engaged in the processes of turning data into information? The Association for Institutional Research (AIR), with assistance from Ellucian, developed a survey to answer these questions. Ellucian’s expertise in working with postsecondary education institutions provided useful insights that were incorporated into the survey of AIR members. Thanks go to the 542 respondents who gave us a grassroots view of these important issues. We were delighted to hear from a wide range of institutions including public four-­‐year, private four-­‐year, and public two-­‐year colleges and universities. We now have a better sense of the tools, methodologies, and budgets committed to measuring and reporting performance. Thanks also go to Drs. Darlena Jones, Amelia Parnell, and Leah Ewing Ross of AIR who served as the lead researchers for this project. They developed the survey, compiled the data, conducted the analyses, and wrote the report. Mr. Christopher Coogan served as AIR’s liaison with Ellucian in order to achieve this robust examination of a topic of great importance to higher education. We are pleased to share our findings and look forward to continued discussions on this topic. Key findings are reported below and are followed by the full research report. I hope these findings and the detailed information contained in this report encourage productive campus discussions that focus on improving the use of data to prove and improve higher education. Randy L. Swing, Ph.D. Executive Director Association for Institutional Research July 30, 2014 Tallahassee, Florida K e y FF i n d i n g s Demand for information is exploding in higher education, and institutional researchers are the center of their institutions’ data and information hubs. Because of this need for data, business intelligence (the skills, applications, and technologies used to leverage an institution’s data to support data-­‐informed decision-­‐making) is growing in popularity among senior leaders. The Association for Institutional Research (AIR) recently conducted a survey to determine institutions’ use of business intelligence (BI) and members’ roles in BI. Using KPIs: Most higher education institutions are adopting key performance indicators (KPIs), and institutional researchers are leaders in this arena. Survey evidence: Copyright AIR, 2015 and used with permission by Ellucian PAGE 1 •
90% of respondents said KPIs are important to their institutions. •
79% of respondents said their IR offices will be very involved in data collection to support KPI reporting within the next three years. •
80% of respondents said their institutions track enrollment KPIs. •
71% of respondents said their IR offices are the primary authorities on campus to collect KPI data. • 41% of respondents said KPIs originated in the IR office. Selecting Analytical Software: In selecting institutional technology solutions, IR offices are part of decision making teams. Survey evidence: •
42% of respondents said IR offices either participate in decisions or are the decision makers in choosing their institutions’ enterprise resource planning (ERP) systems. •
68% of respondents said IR offices either participate in decisions or are the decision makers in choosing their institutions’ BI solutions. Changing IR Role: The IR role will change for some IR professionals in the near future. Survey evidence: •
53% of respondents said they anticipate a role change in IR within the next three years. •
38% of respondents predicted their roles will evolve toward being more strategic within the next three years. •
27% of respondents noted that changes in analytical technologies will provide opportunities to conduct more advanced analytics. Budget: IR professionals are somewhat optimistic for stable or expanded office budgets in the near future. Survey evidence: •
32% of respondents expect expansion of their IR office budgets over the next three years, 46% anticipate no changes in their office budgets, and 22% expect their budgets to be reduced. Copyright AIR, 2015 and used with permission by Ellucian PAGE 2 S e c t i o n 1
1 : K
K e y P
P e r f o r m a n c e II n d i c a t o r s Key Performance Indicators (KPIs) are metrics that provide leaders information to evaluate an institution’s success or its progress toward a strategic goal. Examples of KPIs for higher education are enrollment, retention, and graduation rates. I n s t i t u t i o n a l K
K P I s Nearly every respondent (90%) reported that KPIs are somewhat to very important to their institutions (Chart 1). When asked if KPIs are a major responsibility for IR offices, 64% said yes. Over half of respondents (57%) said it is very important that KPIs have consistent definitions across institutions to support benchmarking efforts. We asked respondents to identify the areas from which their KPIs originate. More than half of respondents reported that KPIs originate from the president/cabinet, 47% reported academic leadership, and 41% reported IR offices. External groups, such as state or federal agencies, play smaller roles (Chart 2). NOTE: This question was multi-­‐response; responses total more than 100%. Chart 1: Importance of KPIs KPIs are somewhat or very important to insatuaon 90% KPIs are a major responsibility for IR office 64% It is very important for KPIs to have consistent definiaons for benchmarking 57% Importance of KPIs Chart 2: Originaaon of KPIs President/Cabinet Academic leadership Insatuaonal Research State Internal commibees Federal Department commibees Other funding organizaaons Don’t know Other 51% 47% 41% 37% 32% 29% 16% 9% 8% 7% Copyright AIR, 2015 and used with permission by Ellucian PAGE 3 K P I P
P r o c e s s 3 years from now: Very involved Change Table 1: How involved was the IR office with the following processes at your institution? How involved do you expect to be three years from now? Creation of Key Performance Indicators (KPIs) Data collection to support KPI reporting Reporting on KPIs Strategic planning institution-­‐wide Other institutional performance improvement initiatives Current: Very involved The KPI process includes identification of performance metrics, collection of data to measure current performance, reporting of data, strategic planning based on the outcomes of those metrics, and other institutional improvement initiatives. AIR asked survey respondents the degree to which their IR offices are involved in the KPI process. The percentages of respondents who noted that they are “very involved” in different stages of the KPI process are presented in Table 1. We also asked respondents what they anticipate their IR offices’ participation with KPIs will be in three years; they predict higher levels of future involvement for every stage of the KPI process. Creation of KPIs and work on other institutional performance improvement initiatives are the areas in which survey respondents expect the greatest increases in levels of involvement. 53% 73% 69% 53% 58% 69% 79% 78% 65% 75% 16% 6% 9% 12% 17% K P I C
C a t e g o r i e s Increased involvement Table 2: Which of the following KPI categories does your institution currently track? How will the IR office’s involvement change over the next three years? Enrollment Academics Recruitment/Admissions Finance and Administration Student Services/Student Life Alumni Relations/Advancement Workforce Management Institution Tracks KPI Performance metrics can be used in many areas of an institution. Respondents were asked to indicate in which areas their institutions track KPIs (Table 2). We found that most institutions track enrollment, while few track workforce management. We also asked respondents if their IR offices would have increased involvement over the next three years in these KPI categories. 80% 73% 65% 53% 35% 35% 22% 57% 66% 58% 39% 48% 34% 35% Copyright AIR, 2015 and used with permission by Ellucian PAGE 4 R e p o r t i n g K
K P I s Overwhelmingly, respondents said that their IR offices are the primary authorities to collect data on their institutions’ KPIs (Chart 3). Chart 3: Primary Authority to Collect KPI Data Insatuaonal research Don’t know 6% Department leaders 6% Academic/Provost Office 9% Department Leaders 7% Finance/CFO 6% Other 5% Operaaons/COO 3% VP Student Services 3% Informaaon Technology 2% 2% Operaaons/COO 1% Finance/CFO 1% 58% 8% 4% Informaaon Technology Insatuaonal Research President 10% Other Chart 4: Primary Reporters of KPIs Academic/Provost Office 71% When asked who reports information on KPIs at their institutions, 58% of respondents cited their IR offices (Chart 4.) Other KPI reporters include provosts, presidents, and department leaders. We asked respondents to identify all of the areas to which KPIs are reported (Chart 5.) Two-­‐thirds of respondents said their institutions’ KPIs are reported to leadership and 40% said their accrediting bodies received KPI data. Nearly one-­‐third (29%) of respondents said KPI data are reported on their websites and to the public. Chart 5: Reporang KPIs Leadership 67% Accrediang bodies 40% State/federal agencies 39% Faculty 34% Staff focused on measurement NOTE: This question was multi-­‐response; responses total more than 100%. 31% Website 29% The public 29% 16% Funding agencies Other 8% Copyright AIR, 2015 and used with permission by Ellucian PAGE 5 S e c t i o n 2
2 : B
B u s i n e s s II n t e l l i g e n c e SS o f t w a r e C h o o s i n g R
R e p o r t i n g aa n d A
A n a l y t i c s SS o f t w a r e TT o o l s We asked respondents if their IR offices are part of decision-­‐making processes to select their institutions’ enterprise resource planning (ERP) and business intelligence (BI) reporting and analytics software tools (Chart 6). Chart 6: Choosing ERP and BI Solware We found that 42% of respondents’ IR offices either participate in 1% decisions or are the decision makers 18% in selecting their institutions’ ERP 41% systems. 50% We also found that 68% of respondents said their IR offices 58% either participate in decisions or are 32% the decision makers in selecting their institutions’ BI solutions. ERP system BI soluaon Decision maker 1% 18% Paracipated 41% 50% No involvement 58% 32% Regarding institutions’ reporting and analytics tools/software, 76% of Chart 7: Choosing Reporang and Analyacs respondents’ IR offices either Solware Tools participate in decisions or are primary decision makers (Chart 7). 21% 26% When respondents were asked what they anticipate in terms of their IR 50% 75% offices’ influence on future purchases of reporting and analytics software 24% tools, 96% believe their IR offices will 4% Current insatuaon participate in the decisions or will be Future IR tools tools decision makers. Decision maker 26% 21% Paracipated 50% 75% No involvement 24% 4% Copyright AIR, 2015 and used with permission by Ellucian PAGE 6 E v a l u a t i n g SS o f t w a r e TT o o l s ff o r II R W
W o r k We asked respondents to identify the ideal features for software tools (Chart 8.) Nearly three-­‐fourths noted that they desire a tool that provides access to all required data, and 69% would like the ability to capture historical data and decisions. NOTE: This question was multi-­‐response; responses total more than 100%. Chart 8: Features of an Ideal Solware Tool 74% Access to all required data 69% Ability to capture historical data and decisions Ability to engage all major departments electronically Out of the box integraaon with ERP Built-­‐in KPIs and metrics for tracking performance 53% 50% 48% Respondents described the features their ideal software tools would contain. In their words… Be user friendly to someone who doesn't know how to code or do SQL. I should be able to build queries and reports as simply as it is to use Office or surf the web. Publish directly to and provide interactive use on websites without specialized web programming. Ease of use and ease of learning. Organization and easy search for reports etc. We don't have a high-­‐powered predictive analytics tool, and that's what we need. Be able to integrate data from multiple sources and be interactive in filtering fields (users get to pick fields and criteria) and also be able to do a drill down (e.g. start with university overall enrollment and then add layers like UG/Grad, In state-­‐Out-­‐of-­‐state, Male/Female, etc.) and be available via web interfaces It should readily interface with other databases. It should allow for real-­‐time (or at least near-­‐real time) visualizations of data out of the box. It should provide a user interface designed to be easy for a beginner to learn, with options available for more advanced users. Copyright AIR, 2015 and used with permission by Ellucian PAGE 7 We presented survey respondents with a list of KPI tools currently on the market and asked them to indicate their satisfaction with the tools they use (Table 3). % Respondents whose Table 3: Software Tools institutions use tool for KPIs Microsoft Excel 60% Satisfaction of respondent using tool in IR work Dissatisfied (Answer = 1, 2) Neutral (Answer = 3) Satisfied (Answer = 4, 5) 2% 7% 90% Business Objects (SAP) 6% 10% 25% 65% Entrinsik 5% 25% 13% 63% IBM Cognos 14% 25% 14% 61% Crystal Reports 16% 7% 36% 57% SharePoint 16% 12% 35% 53% Blackboard Analytics 8% 15% 35% 50% Nuventive TracDat 2% 0% 50% 50% Evisions Argos 11% 23% 46% 31% We asked respondents to list other software tools used in their IR work, which resulted in a list of 34 additional tools; SPSS, SAS, Access, and Tableau were the most common. We asked respondents to identify all barriers they experience when using their current software tools for IR work (Chart 9). The need for more training and the cost of the tools were noted as the most common barriers. Chart 9: Barriers to Using Tools 39% Need more training 38% Cost 33% The right data aren't available 30% Inability to engage with all insatuaonal departments 29% Solware is not user-­‐friendly 28% Lack of historical data and decision capture No enterprise resource planning integraaon 24% Copyright AIR, 2015 and used with permission by Ellucian PAGE 8 S e c t i o n 3
3 : C
C h a n g i n g II R R
R o l e We asked respondents what changes they expect in their IR roles in the next three years. The predominant theme in respondents’ predictions was that IR will have a more strategic role (Table 4). Table 4: Changing IR Role in Next Three Years More strategic/planning More data/analytics focused More assessment focused Use of more BI tools More reporting More technical More involvement in educating users More of a support role Leaving the job In a new position Don't know No change % of Responses 38% 17% 8% 7% 6% 1% 1% 1% 1% 1% 5% 5% In their words… The role of IR within the university setting is changing particularly in the aspect of strategic planning, data support for decision making. Involved in more strategic planning and assessment activities More automated required reporting to different agencies. More time spent answering and asking questions related to student success and college goals. My role will evolve into more of a support capacity. We have developed automated program review report application and we have implemented a more user-­‐friendly Academic Assessment tool. I believe that we will be more in the area of outcomes assessment. IR will take over a more direct role in the long-­‐term planning of BI with also hands-­‐on development of BI (ODS/EDW build-­‐out, meta-­‐data, report /dashboard development, data governance). IR will go from strictly BI consumer to a key player in expanding and maintaining BI and NEIU. Copyright AIR, 2015 and used with permission by Ellucian PAGE 9 We asked respondents to reflect on how advances in analytical technologies will change the responsibilities of their jobs. The predominant theme in respondents’ predictions was that IR professionals will have the abilities to conduct more analyses (Table 5). Table 5. Change in Responsibilities Due to % of Advances in Analytical Technologies Responses Allow for more analyses 27% Improve efficiencies 10% More reporting 10% Don't know/No answer 8% General statement 8% Other users accessing data 8% Need to adapt to new technologies 6% No changes to job 6% New hires with different skills 4% Automated reporting 2% Data story teller 2% Managing data 2% More programming design 2% Needs training 2% Pushing beyond current tasks 2% Support role 2% In their words… I will do less simple ad hoc and more in-­‐
depth analysis of data. I'm looking forward to the upcoming changes— they can't come soon enough. The hope is that analytics allows our office to spend less time on standard and ad hoc enrollment reporting (because it is much less manual and more seamless) and allows our office to spend more time on analysis, evaluation, and assessment. It will make data readily available to end users. It will ascertain the accuracy of data so that we can always have consistent data report no matter the time of the year the data is pulled from the student administration system. If done properly, it will make us more productive (better and faster analysis without the issues of data gathering and cleansing, which now take up more than half of the time). New hires must be more proficient and experienced in using such technologies. Will need to walk in the door with the basic training already completed. Copyright AIR, 2015 and used with permission by Ellucian PAGE 10 S e c t i o n 4
4 : II R B
B u d g e t We asked respondents to anticipate how their IR offices’ budgets will change over the next three years; the responses were very different by sector (Chart 10). •
All Respondents: We found 46% of respondents expect no change in their budgets and 32% expect their budget to expand over the next three years. •
Public 4-­‐Year Institutions: Disaggregating by U.S. sector, we find more than half of respondents from public 4-­‐year institutions believe their budgets will stay the same and 28% anticipate expansion of their budgets over the next three years. •
Public 2-­‐Year Institutions: Over 90% of respondents from public 2-­‐year institutions in the U.S. anticipate their budgets will stay the same or expand over the next three years. •
Private 4-­‐Year, Not-­‐for-­‐Profit Institutions: Respondents from private, 4-­‐year, not-­‐for-­‐profit institutions in the U.S. were less positive. We found 50% of respondents expect their budgets to shrink over the next three years. NOTE: We only report responses from the three sectors listed above because of lack of responses in other sectors. Respondents from public 4-­‐year institutions rated this item statistically higher than respondents from private, 4-­‐year, not-­‐for-­‐profit institutions (p < .05). Chart 10: Anacipated Change to IR Budget Over Next Three Years 3% 29% 2% 6% 26% 43% 46% 10% 40% 52% 43% 35% 17% 16% 5% 4% 6% 2% All Respondents Public, 4-­‐Year Public, 2-­‐Year Private 4-­‐Year, Non-­‐
profit Expand by 25% or more 3% 2% 6% 0% Expand up to 25% 29% 26% 43% 10% Stay the same 46% 52% 43% 40% Reduce up to 25% 17% 16% 6% 35% Reduce by 25% or more 5% 4% 2% 15% Copyright AIR, 2015 and used with permission by Ellucian 15% PAGE 11 We asked respondents to approximate the amount spent annually by their IR offices on analytical software tools. The largest segment, 42%, noted that less than $1,000 is spent annually, and only 25% of respondents reported that their Chart 11: Approximate Annual Amount Spent by offices spend $5,000 or more IR Office on Analyacal Solware annually (Chart 11). In a separate, but related question, we asked respondents to identify the amount of the single largest purchase made by their IR offices in the past year. One-­‐quarter noted that they spent less than $1,000 and 46% said $5,000 or more. Up to $1,000 42% $1,001 to $5,000 $5,001 to $9,999 $10,000 or more 32% 11% 14% Chart 12: Analyacal Solware Expenditures from IR Office Budget Yes, all, 15% No, 31% We asked respondents whether analytical software expenditures come out of IR offices’ budgets. Nearly one-­‐third (31%) said “no” and 15% reported that all expenditures come from IR office budgets (Chart 12). Yes, some, 54% Copyright AIR, 2015 and used with permission by Ellucian PAGE 12 C l o s i n g There’s no doubt about it: institutional research professionals have an increasingly vital role in helping institutions identify, understand, and act upon opportunities for performance improvement. More than half of survey respondents said they foresaw their roles changing in the future, and nearly 40% said they expected to adopt a more strategic role in their institutions’ decision-­‐making processes. That all makes sense given the overwhelming emphasis on key performance indicators today. A full 90% of respondents said KPIs are important to their institution. That importance is here to stay. And that’s exciting. With the right tools, training, and expertise, institutions can get the insights they need to truly elevate and increase overall performance to better serve students. Ellucian Perform is for campus leaders who have multiple sources of data and want to drive a culture of data-­‐based decision making throughout the entire institution. To learn more about the industry’s first-­‐to-­‐
market platform for establishing and maintaining a data-­‐driven culture of performance and how this solution can significantly support the changing role of IR, visit Ellucian.com. Wayne Bovier, Vice President of Product Management, Ellucian Association for Institutional Research 1435 E. Piedmont Drive, Suite 211, Tallahassee, FL 32308 850.385.4155 (phone), 850.385.5180 (fax) Ellucian Headquarters: 4375 Fair Lakes Ct, Fairfax, VA 22033, USA Phone: +1 800.223.7036 To download copies of this research, visit www.ellucian.com Copyright AIR, 2015 and used with permission by Ellucian PAGE 13 A p p e n d i x S u r v e y M
M e t h o d o l o g y aa n d R
R e s p o n d e n t C
C h a r a c t e r i s t i c s Survey metrics were written by assessment experts at AIR in collaboration with staff from Ellucian, a company that specializes in business intelligence (BI) software. Sixty-­‐five questions on the survey explored the types of key performance indicators (KPIs) used on respondents’ campuses, use of BI software, and respondents’ personal experiences with BI software. AIR randomly selected 1,546 members of the Association to participate in the study. The survey participants were divided into three groups, and the survey was divided into three sections; each survey group received one section of the survey in order to reduce respondents’ survey burden. An online surveying system was used to collect data April 22—May 6, 2014. One invitation email and two reminder emails were sent to survey participants. We collected 542 responses for a 35% response rate. Of the 542 responses, 536 individuals self-­‐
identified their IR roles and the locations of their institutions (Table 6). Table 6: IR Role by Institution Location College or university, in an IR office College or university, not in an IR office System office Non-­‐profit organization Retired Other TOTAL Location of Institution Outside the U.S. Canada and Canada 9 7 0 3 1 0 0 2 0 0 0 0 10 12 U.S./U.S. territories 415 64 16 4 2 13 514 Total 431 67 17 6 2 13 536 For the purpose of this report, we focused on the 424 respondents who work in college or university IR offices in the U.S./U.S. territories or Canada. (In the report, reference to institutions in the U.S. includes U.S. territories.) The distribution of the 424 responses by survey is shown in Table 7. Table 7: Survey Survey 1 Survey 2 Survey 3 TOTAL # Responded Response Rate 158 127 139 424 37% 30% 33% Copyright AIR, 2015 and used with permission by Ellucian PAGE 14 Senior institutional research (IR)/institutional effectiveness (IE) officers comprised nearly 60% of the response pool (Table 8). Note that 13 respondents failed to answer this item. NOTE: Due to rounding errors, values may appear to not be equal to 100%. Table 8: Respondent's Role Senior IR/IE Officer Associate/Assistant IR/IR Officer IR Analyst/Researcher Senior Assessment Officer Assessment analyst/researcher Other Total # of Respondents 240 43 98 2 15 12 410 % of Respondents 59% 11% 24% 1% 3% 3% Respondents who work at post-­‐secondary institutions in the U.S. were matched with their institutions’ IPEDS sectors. The largest sector represented in the data set was private, not-­‐for-­‐profit, 4-­‐year institutions followed by public 4-­‐year institutions (Table 9.) Table 9: Institutional Sector Private not-­‐for-­‐profit, 4-­‐year or above Public, 4-­‐year or above Public, 2-­‐year Private for-­‐profit, 4-­‐year or above Private not-­‐for-­‐profit, 2-­‐year Total # of Respondents 154 145 87 14 1 401 % of Respondents 38% 36% 22% 4% 0% T e s t i n g o
o f D
D i f f e r e n c e s ii n M
M e a n s We performed t-­‐tests on survey metrics between respondents in different sectors. There were no statistical differences in means by sector except for the metric describing the anticipated change in IR budget over the next three years, as noted. In that metric, respondents from public 4-­‐year institutions rated that item statistically higher than respondents from private, 4-­‐year, not-­‐for-­‐profit institutions. Copyright AIR, 2015 and used with permission by Ellucian PAGE 15 S u r v e y II n s t r u m e n t College or university, in an Institutional Research office College or university, not in an Institutional Research office Which best describes your current place of employment? System office Non-­‐profit organization I am retired Other (please specify) In the U.S./U.S. territories Where is your primary work location? In Canada Outside of the U.S. and Canada How involved should IR offices be in choosing their institutions’ business intelligence solutions? How involved should an IR office be in choosing their institutions’ reporting and analytics tools/software? Should have primary responsibility for decision Should participate in decision-­‐making Should have no involvement Don’t know Should have primary responsibility for decision Should participate in decision-­‐making Should have no involvement Don’t know Reduce by 25% or more How do you expect an institution’s IR office budget to change in the next 3 years? Reduce up to 25% Stay the same Expand up to 25% Expand by 25% or more Don’t know Senior IR/IE Officer Associate/Assistant IR/IR Officer IR Analyst/Researcher Which best describes your role in Institutional Research? Senior Assessment Officer Assessment analyst/researcher Other (please specify) How involved was your IR office in choosing your institution’s business intelligence solution? How involved was your IR office in choosing your institution’s reporting and analytics tools/software? Primary responsibility for decision Participated in decision-­‐making No involvement Don’t know Primary responsibility for decision Participated in decision-­‐making No involvement Don’t know Copyright AIR, 2015 and used with permission by Ellucian PAGE 16 Primary responsibility for decision How involved was your IR office in choosing your institution’s ERP (enterprise resource planning) system (e.g., Ellucian, Jenzabar, PeopleSoft)? Participated in decision-­‐making No involvement Don’t know Institution doesn't have an ERP system How do you see your role changing in the next few years? That is, do you anticipate that it will be more or less strategic, encompass different responsibilities, etc.? Information Technology Finance How frequently do you interact with Registrar each of the following? (Scale = 1) Daily, Academic Department Leaders 2) Weekly, 3) Monthly, 4) Less than Administrative Department Leaders monthly, 5) Rarely or never) Senior Leaders (CEO, CAO, etc.) Cabinet/Board of Trustees How involved was the IR office with the following processes at your institution? (Scale = 1) Not involved at all, 2) Somewhat involved, 3) Very involved, 4) Institution does not use KPIs, 5) Don’t know) Creation of Key Performance Indicators (KPIs) How involved will the IR office expect to be with these processes in the next 3 years? (Scale = 1) Not involved at all, 2) Somewhat involved, 3) Very involved, 4) Institution does not use KPIs, 5) Don’t know) Creation of Key Performance Indicators (KPIs) Data collection to support KPI reporting Reporting on KPIs Strategic planning institution-­‐wide Other institutional performance improvement initiatives Reporting on KPIs Data collection to support KPI reporting Strategic planning Other institutional performance improvement initiatives Are KPIs (Key Performance Indicators) Yes a major responsibility for the IR office? No State Federal Other funding organizations Where do the KPIs used at your institution originate? Please select all that apply. President/Cabinet Academic leadership Institutional Research Internal committees Department committees Other (please specify) Don’t know Copyright AIR, 2015 and used with permission by Ellucian PAGE 17 Institutional Research Information Technology Department Leaders Finance/CFO Who has the primary authority to collect data on KPIs? Operations/COO Academic/Provost Office Other (please specify) Don’t know Institutional Research President Information Technology VP Student Services Who is the primary reporter on KPIs? Department Leaders Finance/CFO Operations/COO Academic/Provost Office Other (please specify) Don’t know Leadership Staff focused on measurement Faculty The public Accrediting bodies Where are KPIs reported? Please select all that apply. Funding agencies Government (state/federal) agencies Website Other (please specify) Don’t know Formal strategic planning Obtain funding resources Accreditation How are KPIs being used at your institution? IPEDS reporting Other external reporting Other (please specify) Don’t know Copyright AIR, 2015 and used with permission by Ellucian PAGE 18 Blackboard Analytics Business Objects (SAP) Crystal Reports What software tools does your institution currently use for tracking KPIs (Key Performance Indicators) and reporting on data? Please select all that apply. Entrinsik Evisions Argos IBM Cognos Microsoft Excel Nuventive TracDat SharePoint Other (please specify) Don't know Please list any other software tools used for IR tasks with which you are familiar. Access to all required data Out-­‐of-­‐the-­‐box integration with ERP Thinking of the ideal software tool for your role, what features would it include? Please select all that apply. Ability to engage all major departments electronically Ability to capture historical data and decisions Built-­‐in KPIs and metrics for tracking performance Other (please specify) Don’t know Recruitment/Admissions Enrollment Student Services/Student Life Please indicate which of the following KPI categories your institution currently tracks. Please select all that apply. Academics Alumni Relations/Advancement Finance and Administration Workforce Management Other (please specify) None of these Don’t know How will the IR office’s involvement with the following KPI categories change in the next 3 years? (Scale = 1) Decreased involvement, 2) Involvement will stay the same, 3) Increase involvement, 4) Institution doesn't use this category, 5) Don't know) Recruitment/Admissions Enrollment Academics Alumni Relations/Advancement Finance and Administration Workforce Management What is the level of importance placed on KPIs by your institution? (Scale = 1) Not at all important, 2) Somewhat important, 3) Very important, 4) My institution doesn't use KPIs, 5) Don't know) Copyright AIR, 2015 and used with permission by Ellucian PAGE 19 How important is having KPIs defined across institutions to support benchmarking? That is, how useful would common definitions across institutions for KPIs be in order to support comparisons? (Scale = 1) Not at all important, 2) Somewhat important, 3) Very important, 4) My institution doesn't use KPIs, 5) Don't know) What will be your IR office’s future involvement with these KPIs categories? Blackboard Analytics Business Objects (SAP) Thinking about the software tools that you use for Institutional Research, how satisfied are you with each? (Scale = 1) Very dissatisfied, 2) Dissatisfied, 3) Neutral, 4) Satisfied, 5) Very satisfied, 6) Don't use this) Crystal Reports Entrinsik Evisions Argos IBM Cognos Microsoft Excel Nuventive TracDat SharePoint What would your ideal software tool be able to do that your current tool doesn't? Please explain. There are no barriers to the tools I currently use The right data aren't available Thinking about the software tools you are currently using for Institutional Research, what are the barriers to using these tools? Please select all that apply. Need more training Software is not user-­‐friendly Cost No enterprise resource planning integration Lack of historical data and decision capture Inability to engage with all institutional departments Other (please specify) Additional services engagement with system vendor Which of the following would help you to retrieve data from your systems? Please select all that apply. Additional services from your Information Technology department Workarounds (e.g., export data to Excel) Manual data collection/entry Other (please specify) Reduce by 25% or more Reduce up to 25% How do you see the budget for your IR office changing in the next 3 years? Stay the same Expand up to 25% Expand by 25% or more Don't know Copyright AIR, 2015 and used with permission by Ellucian PAGE 20 Up to $1,000 What is the approximate value of largest purchase your IR office has authorized in the past 12 months? $1,001 to $5,000 $5,001 to $9,999 $10,000 or more Don't know Less than $1,000 Approximately how much does your IR office spend annually on analytical software tools? $1,000 to $4,999 $5,000 to $9,999 $10,000 or more Don’t know Yes, all Do analytical software expenditures come out of the IR office's budget? Yes, some No Don’t know What kind of influence do you think the IR office will have in the selection of analytical software tools for Institutional Research in the next 3 years? Will have primary responsibility for decision Will participate in decision-­‐making Will have no involvement Don’t know How do you think that the advances in analytical technologies will change the responsibilities of your job? Copyright AIR, 2015 and used with permission by Ellucian PAGE 21