How to Identify Your Impact: The Value of Libraries Joe Matthews June 10, 2013 Megan Oakleaf Outline • Performance Measures • Value • Value of Information • Value of Information Services Outline • Value of a Library – Personal value • Direct measures • Indirect measures – Organizational value – Financial value Few libraries exist in a vacuum, accountable only to themselves. There is thus always a larger context for accessing library quality, that is, what and how well does the library contribute to achieving the overall goals of the parent constituencies? Sarah Pritchard There is no systematic evidence collected which shows the value of academic libraries for teaching and research staff. Claire Creaser and Valerie Spezi Performance Measures Library Services Input Process Resources Capability Output Individual Society Outcomes Outcomes Use Beneficial effects Effectiveness Efficiency Impact Cost Effectiveness Cost benefit VALUE Start with the end in mind: work backwards Refocus from the activity to the impact Impact Library Control How much? How many? How economical? How prompt? Magnitude Magnitude Resources used Cycle times % of change last year Change Units processed Turnaround time Anticipatory % of overall change Cost Library & Customers Decide How valuable? How reliable? How accurate? Effort expended Dependability Completeness Cost Access Comprehensiveness Benefits obtained Accuracy Currency Customers Decide How well? How courteous? How responsive? How satisfied? Accuracy Attentive Anticipatory Expectations met Promptness Welcoming Helpful Materials obtained Empathetic Personal interaction Courtesy Expertise Ease of use Environment Comfort Willingness to return Live by the numbers, …. Challenges Lack of consensus about what should be measured and how Lack of understanding of performance measurement and metrics Organizational structural issues Lack of precision in measuring performance, and alignment issues Determining the “bottom line” is too far away Majority of stakeholders are too far away Library staff find it difficult to see the “big” picture And the survey said …. Lack of a Connection • Budget and outputs (and outcomes) are separated • No “bottom line” measure for libraries • Decision-making process is bigger than the library • Library has neither champions nor foes • Library benefits are not widely self-evident Orr’s Fundamental Questions • How good is the library? • What good does the library do? • How well is the library managed? We should be a bit wary of the “little library” …For when it is good, it is very, very good and when it is bad, it’s a “pretty good library for a town this size.” Eleanor Jo Rodger Levels of Assessment … • Individual student • Course • Departmental/Program • College or University Types of Measures • Direct – Provide tangible, visible and selfexplanatory evidence of what students have & have not learned • Indirect – Capture students’ perceptions of their knowledge & skills; supplement direct measures; sometimes called surrogates Qualitative Tools • Focus groups – open ended • Biography • Phenomenology – capture the “Aha!” moment • Grounded theory • Ethnography • Case study Qualitative Assessment • Provides in-depth understanding of user responses and interactions • Represents part of a long-term strategy of formative evaluative Quantitative Tools • Surveys • Transaction logs • Statistics from systems • Observations (count) Quantitative Assessment • Analyses to determine library impacts on academic performance, retention rates • Describe retention rates and GPAs in defined populations over semesters and users • Compare users & non-users of library services while adjusting for academic preparation and background differences • Conduct quasi-experimental designs employing multivariate analysis of covariance & hierarchical regression techniques Useful Assessment Be cautious about cause-and-effect relationships The Issue • Is it: Use library resources/services and you will get better grades. • Or: I want to do well and so I work hard to achieve better grades - and one way I do that is to use library resources/services. “Not surprisingly, librarians are keen to show that the use of expensive, scholarly materials positively correlates with higher grades, although they cannot prove that this is so.” Deborah Goodall & David Pattern “There is growing pressure on all academic library managers to be more accountable for how they use limited resources and to achieve institutional outcomes perceived as important by college and university stakeholders….” Elizabeth Mezick Value of Information • Expect value-in-use • Library’s collection reflects a “potential value” • Collection also reflects a “future value” • Value of local collection is declining Valuable is not about our professional values; in the paradigm of the value of public libraries, we are the producers, not the consumers of services. Our sense of what is valuable really doesn’t matter much at all unless it matches that our our customers. Eleanor Jo Rodger Fundamental Changes Libraries have changed more in the past two decades than in the prior two centuries. Technology is the major driver . . . We need to recognize that all this change has only begun, and that change is irreversible. Increasingly it is important to remember that libraries provide few unique services. Information is woven into our lives Quality of Information This fast food approach to information consumption drives librarians crazy. “Our information is healthier and tastes better too” they shout. But nobody listens. We’re too busy Googling.” Peter Morville Key Characteristics of Information Uncertainty Knowledge Ambiguity Indeterminacy Redundancy System dependency Sharing Timeliness Compression Presentation Stability Multiple life cycles Leakability Substitutability Criteria for Judging Value Customer Criteria Value Added by the Service Ease of use Browsing, formatting, mediation service, orientation service, ordering, physical accessibility Noise reduction Access (item identification, subject description, subject summary), linkage, precision, selectivity Quality Accuracy, comprehensiveness, currency, reliability, validity Adaptability Closeness to problem, flexibility, simplicity, stimulatory Time savings Response speed Cost savings Cost savings Collections are disrupted Atoms to bits Nature of Information is Changing Information was …. Information is …. Scare, controlled All around us Expensive Cheap or free Shaped by elites Shaped by consumers One-way, mass consumption Designed for sharing, participation & feedback Slow moving External to our worlds Immediate Embedded in our worlds Value of the Academic Library If the physical proximity of print collections had a demonstrable impact on researcher productivity, no university would hesitate to allocate prime real estate to library stacks. Traditional Value Proposition Without a great library, there can be no great university. David Kinly, President of The University of Illiniois 1929 Universities Provide • Private goods & services – Courses exchanged for tuition – Research completed for funding • The value proposition The value to an individual or an organization determines whether payment is made for the service Academic Libraries Provide • Public goods and services Print and online resources are shared by all, usually without the exchange of payment • Value proposition The collective value of all users must be estimated to determine if a good or service should be continued Value is determined by the user and the use of information Astin’s IEO Model Programs Institutional characteristics Input Library Fellow students Faculty Place of residence Environment Output Student Learning Outcomes Model Abstract, Process Oriented Intelligence General Reasoning Broad Abilities Knowledge, Understanding, and Reasoning Concrete, ContentOriented Define, develop, and measure outcomes that contribute to institutional effectiveness ACRL Standards for Libraries in Higher Education Challenge It is not how much a library is used that matters, rather how does the library impact or benefit the customer? Perspectives on Value Impacts Direct Personal Organizational Use Indirect Financial Benefits Option – Preservation of option for future use by me Nonuse Existence – Perceived value and significance to the community Legacy – Value of preservation for future generations Personal Why Use the Library? Reasons For a TASK For PERSONAL reasons To get an OBJECT or INFORMATION Interactions Access RESOURCES Results COGNITIVE results AFFECTIVE results Use of RESOURCES or SERVICES ACCOMPLISHMENTS EXPECTATIONS met OPERATIONS TIME aspects To perform an ACTIVITY ENVIRONMENTS MONEY estimates Generic Learning Outcomes Knowledge & Understanding Skills Attitudes Enjoyment, Inspiration, Creativity Activity, Behavior, Progression Organizational Student Learning is Affected by … • • • • • Full-time students Live on campus Interact more with faculty Study more Collaborate with their peers NSSE • 5 benchmarks of effective educational practice – – – – – Level of academic challenge Active & collaborative learning Student-Faculty interaction Enriching educational experiences Supportive campus environment • Student self-reported gains in intellectual & personal development • No overlap between self-reported data & standardized objective tests NSSE – Use of the Academic Library • 50% never used the library • Use of libraries at small, academically challenging liberal arts colleges are correlated with other purposeful activities • Library use less intensive at larger universities • Students who work harder use library resources Assessing Student Achievement • Direct measures – Capstone experience – Use of a portfolio – Standardized exam (Collegiate Learning Assessment) • Indirect measures Assessment of Higher Ed • Gains in student performance are quite low • Individual learning is characterized by persistence • Notable variation within and across institutions Wabash National Study • Different instrument – CAAP • 2,212 students • • • • Nearly identical results to Academically Adrift 44 percent no gains in the first year 33 percent no gains in 4 years Students only study about 15 hours per week • Opinion surveys • Skills testing • Observed behaviors Bibliographic Instruction • Improvement in basic library skills is the means and not the end • Yet the means is the focus for evaluation efforts • Evaluation efforts focus on – Opinion surveys – Skills improvement – Pre-test & post-test knowledge • Not the impact on student achievement Library experiences do not seem to directly contribute to gains in information literacy, to what students gain overall in college, or to student satisfaction. Kuh & Gonyea “One way to demonstrate the library’s contribution is to assess whether students’ experiences with the library directly or indirectly contribute to desired outcomes of college.” George D. Kuh & Robert M. Gonyea How to Demonstrate Impact in … • • • • • • • • • Student enrollment Student experiences Student learning Student grades (GPA) & achievement Student retention & graduation Student career success Faculty productivity Institutional reputation The environment Student Enrollment Student Enrollment • Recruitment of prospective students • Matriculation of admitted students • Recommendation of current students Student Learning Student Learning • • • • GPA Professional/educational test scores Learning assessments Faculty judgments Meta-analysis • Entering student characteristics – SES – High school GPA – ACT/SAT scores • Environment - Psychosocial and study skill factors – Academic goals, skills and self-confidence – Social support & engagement (acculturation) Student Learning Occurs … • • • • • • • • in the classroom in the laboratory with peers in the student union in the dorm in the library (for some) online and Direct Measures • Capstone Experience • Portfolios • Standardized Test • Locally-developed test Capstone Projects Indirect Measures Surveys Retention, graduation rates Grades Acceptance rates into graduate programs • Job placement rates • Exit interviews • Alumni surveys • • • • Student Grades Non/Low Use Usage of Electronic Resources WAM = Weighted Average Marks (Grades) Other Studies • Hong Kong Baptist University • Georgia State University Hope College University of Minnesota Gym Bags and Mortarboards Use Campus Recreational Facilities At least 25 times, first-year retention increased 1% & 5-year graduation rates increased 2% University of Minnesota Library • 5,368 first-year non-transfer students • Use of library was associated with a .23 increase in students GPA • More use of the library, GPA also goes up University of Minnesota Library Library Instruction and GPA • Surveys of student opinions & habits • Assessing student work for specific skills • Analysis of grade point average • Mixed results Library Instruction and GPA Hong King Baptist University • 45 sample groups – N=31 to 1,223, study majors • Pairs of data • One-fourth (11) had a positive relationship • Results: – 1 or 2 workshops – little impact on GPA – 3 or 4 workshops – ½ show a positive impact – 5 workshops (1 sample group) – 100% had a higher GPA Library Instruction and GPA University of Wyoming Libraries • Analysis of 4,489 transcripts • Slight positive relationship between upperlevel library instruction courses and GPA – 0.075 GPA difference – that’s less than 1/10th of 1 percent • Research statement – 44% • Evaluate Web site – objectivity – 52% – authority – 65% • Presentation to persuade – 12% How scalable is library instruction? Student Retention & Graduation Persistence Attrition Retention Graduation Rates Completion Student Retention • Fall-to-fall retention • Graduation rates Retention Concepts • Institutional retention – Enrolling & graduating from the same institution • Program retention – Enrolling & graduating with the same major/department/school • System retention – Students who leave one university yet continue and complete post-secondary studies elsewhere Measures of Retention • Persistence (Continuation rate) – From first to second year? Entry to graduation? • Completion rate – From entry to graduation (Student goals?) • Graduation rates – Are transfers included? Time period? • Attrition – Leaving university? Leaving higher ed? Measures of Retention • Stopout – Leave university with the intention (and action) of returning later to complete a program • Dropout – Leave university with intention (and action) of NOT returning • Transfer – Change institutions yet persist in higher education – May change type of institution • Voluntary vs. involuntary attrition? Why Students Leave? • Students’ decision to leave University is influenced by many personal factors – – – – Financial reasons Family responsibilities Lack of academic ability Poor fit, etc. • Foundational Theories from Education / Psychology: – Tinto’s “Model of Student Integration” – Bean’s “Model of Student Attrition” Tinto’s Model of Student Integration Pre-entry Attributes Goals / Institutional Commitments Experiences Integration Goals / Commitments Outcome Academic System Performance SES Intentions Skills & Abilities Quality of Education Academic Integration Engagement Intentions Goal Institutional ExtraCommitments Curricular Peer group Social System Social Integration Institutional Match Stay or Leave Bean’s Model of Student Attrition Environmental Variables Personal Variables Organizational Variables Loyalty Attitudes Certainty Practical Value Grades Courses Educational Goals Intent Major & Job Certainty Opportunity to Transfer Dropout Family Approval Student Retention & Graduation • Important because … rankings, revenues, educational achievement, emotional wellbeing • Many reasons for drop-outs are not under the control of the university • Engagement is the key Indicators • Student goal attainment • Course retention • Subsequent course work • Fall-to-fall persistence • Time to degree • Degree completion • Grad school enrollment • • • • • • • Transfer rate & success Employer assessment Academic value add Student satisfaction Professional growth Student involvement Citizenship & engagement Australasian Survey of Student Engagement (AUSSE) Curtin University University of Huddersfield “A high rate of attrition is indicative of a failure on the part of an institution to achieve its purpose.” Elizabeth Mezick Student Engagement “many students don’t develop a personal connection with their institution. And when they don’t, they leave” Gonzales 2010 NSSE & CSEQ Library Retention Studies • Statistically significant relationships between library expenditures, or staffing levels and student retention E.g. Hiscock, 1986 Hamrick, Schuh, & Shelley, 2004 Mezick, 2007 Graduation Rates & Library Expenditures • Used IPEDS data on institutional characteristics & resource allocations • Library expenditures was strongly correlated with graduation rates – 1.77 percent increase in graduation rates • Greatest payoff is attributable to enhanced library expenditures (+0.92) and instruction (+0.80) while increased non-library contributions were quite modest (+0.27) Hamrick, Schuh & Shelley Library Retention Studies • Relationships between library use (collections) and student retention – Student who borrowed books = more likely to persist • E.g. Kramer & Kramer, 1968 • Impact of instruction – Students involved in library skills programs showed lower attrition rates • E.g. Knapp, 1966 Library Retention Studies University of Minnesota – 77% of undergrads made use of the libraries, 85% of grad students made use of the libraries – Students who used the library at least once were 1.54 times more likely to re-enroll Library Retention Studies • Some library involvement in first year experience programs; specific programs for “at risk” groups – NOT proven to have significant effect • E.g. Hollis, 2001 Colton, et al, 2002 Aguilar & Keating, 2009 Love, 2009 Library Retention Studies • Relationship between library employment & retention – Higher completion rate among library student workers • E.g. Wilder, 1990 Rushing & Poole, 2002 “If strong linkages between libraries and student retention can be made, then the perceived value of the library may indeed rise.” Steven Bell • Some groups, some majors & seniors engage in more library-related activities • Academic support expenditures tend to correlate with increased engagement • Institutional academic challenge correlates with library use Student Career Success Grad School Exams Alumni Surveys Student Success • • • • • Job placement rates First-year job salaries Professional/graduate school acceptance Internship success Marketable skills Faculty Teaching • Integration of library resources and services into course syllabi, Websites, lectures, labs, reserve readings, etc. • Faculty/librarian collaborations; cooperative curriculum, assignment, or assessment design Perceived Benefits for Teaching • Savings – Of own time – Of own money – Of other resources • Improvements – Teaching – Course-related materials – Student performance Impact on Faculty • Library is the source for most journal articles (individual subscriptions are way down) • If library subscriptions were unavailable – productivity would decrease 17% • Library is not the source of book readings • 42% of reading material is library provided Time • Academics spend a lot of time reading • Article reading inspires new thinking, improved results, changed focus • Award-winning academics read more • Academics who publish more use more library resources Ithaka Studies • Library services not understood • Library services not valued • The Library is disappearing Faculty Research Productivity • Number of publications, number of patents, value of technology transfer • Tenure/promotion judgments Faculty Grants • Number of grant proposals (funded or unfunded) • Value of grants funded Assessment of Research • • • • • • • • • Payback model – form of ROI Research impact Research utilization ladder Lavis decision-making impact model Weiss logic model HTA organization assessment framework Societal impact framework Research assessment exercise Becker medical library model For Most Impact Models • Indicators of research output • Indicators of knowledge transfer • Indicators of implementation • Indicators of community benefit Faculty Productivity Institutional Reputation & Prestige • Faculty recruitment • Institutional rankings • Community engagement Institutional Reputation • Changes in reputational rankings affects student & faculty recruitment • University budget allocations to libraries have decreased Since the library absorbs a very small percentage of a university budget, the contribution of the library is disproportionately high relative to its cost to the institution. Sharon Weiner University & the Library Can • Attract outstanding faculty • Retain outstanding faculty • Foster innovative research • Align library activities with university goals • Indispensable for their research • Maintain a high-level overview of their field • Value for money is good • Library not available, costs would increase 40% • Take 31% longer to locate same information Financia l At the table or On the menu? What is ROI? ROI Applications • Projects • Services • Organization ROI in Library Contexts • Demonstrating the value of libraries • Evaluating existing services, collections, etc. • Making the case for additional services or resources • Recruiting support for a program or initiative ROI Terminology • Cost-Benefit Analysis (CBA) – Compares cost or purchase price with estimated value of variables that are difficult to measure • Consumer Surplus – Value that consumers place on the consumption of a good or service in excess of what they paid for it • Cost of Time and Effort – Measures time and effort expended by users • Contingent Valuation – Measures value of use and non-use of non-priced goods and services, e.g., a library Cost/Benefit Methodologies Maximize the benefits for given costs Minimize the costs for a given level of benefits Maximize the ratio of benefits over costs Maximize the net benefits (present value of benefits minus the present value of costs) • Maximize the internal rate of return • • • • Ratio of Benefits to Costs Value of benefits divided by Costs ROI = Benefit – Cost Cost University of Pittsburgh ROI • If the library’s journal collection (print & electronic) were not available, faculty would use 250,000 hours and $2.1 million to find alternative sources for the articles • It would cost the university 4.38 times the cost of the current library journal collections for the same amount of information gathering to be carried out University of Illinois, Urbana-Champaign Connected citations to resources in the library’s collection to successful grant proposals, and the income the grants generated ROI = 4.38:1 Other studies – 0.27:1 to 15.54:1 ROI may provide us with calculations that seek to document a financial relationship between action and benefit, but too often in the library community these studies are poorly constructed, ineffectively executed, and naïvely communicated. And in the final analysis, do not respond to the legitimate questions being raised by our administrators and funders, and do not advance the academic library as a critical factor in institutional success. James Neal • • • • • • • Comprehensive assessment of the library ROI of the journal collection & readership ROI for support of teaching & learning ROI of digitized special collections ROI of eBooks Value of library commons Bibliography Bryant University – Faculty Access Book ROI = 3.5:1 Article ROI = 3.2:1 Other publications ROI = 3.2:1 Syracuse University – ROI 4.49:1 (Millions of Dollars) Faculty Students In Person $13.6 $23.1 Remote 19.0 14.5 TOTAL $32.6 $37.6 Remote access $1.6 $3.7 Read not-printed 0.1 0.7 Social ? ? TOTAL $34.3 $42.0 Economic Environmental Journal Collections & Reading ROI for Support of Teaching & Learning Perceived Benefits – • Savings … – Of own time – Of own money – Of other resources – printing, copier • Improvements … – Teaching – Course-related materials – Student performance ROI of Digitized Special Collections User • What is the value to a user in terms of time and money spent? Prestige • What is the prestige to the institution for high visibility digital special collection? Development • What value accrues to the development effort of the institution? ROI of Digitized Special Collections Environmental • What is the value of the environmental savings from limited physical access to unique and often fragile material? Scholars • What value accrues from the role of special collections in attracting graduate students? Collections • What is the value of digital collection in attracting additional special collections? ROI of eBooks The Commons ROI = 26:1 Communicating Value Past Future Building & maintaining collections Focus Engaging students & faculty Internal silos Structure University priorities Incremental Change Transformational Periodic Outreach Deliberative Top down Decision making Shared Traditional Measures of Success Impact Library Value • How integral it is to the community • How well it supports learning and teaching • How well it supports research Value Proposition What your library does well What your customers value The Value Proposition • An offer, not a demand • Not what you value • Only valuable perspective is the customer’s • Valuable in a competitive environment Or The promise that a library makes to its customers about what they can expect to receive in return for their time, their effort, their loyalty, and especially their dollars. The library needs to … Focus on customers and what they want and need as well as how they want and need it. Stories + Stats = Success Is the value in the glass, the wine or the savoring? The financial crisis is looking even worse, but you will pleased to know that the director reports that the library performance went up a half a point on the “library goodness scale” last week. Michael Buckland Joe@JoeMatthews.Org www.joematthews.org Joe Matthews Library Consultant What Are the Results Four Year College Full-time College degree – 19% Four Year College Part-time Two Year College No college Graduate degree – 11% Stopout/Transfer AA degree – 8% No college degree – 60% Valuing the Collection Dewey Subclass Number of Titles 2010-2011 Avg List Total Value ($) 001 - Knowledge $76.71 $0.00 002 - The book $62.45 $0.00 003 - Systems $129.77 $0.00 004 - Data processing. Computer science $89.82 $0.00 005 - Computer programming, programs, data $69.14 $0.00 006 - Special computer methods $83.60 $0.00 010 - Bibliography $73.65 $0.00 011 - Bibliographies $69.46 $0.00 012 - Bibliographies of individuals $0.00 $0.00 013 - Of works by specific classes of authors $0.00 $0.00 014 - Of anonymous and pseudonymous works $55.95 $0.00 015 - Of works from specific places $184.99 $0.00 016 - Of works on specific subjects $134.87 $0.00 017 - General subject catalogs $0.00 $0.00 018 - Catalogs arranged by author, date, etc. $0.00 $0.00 019 - Dictionary catalogs $0.00 $0.00 020 - Library and information sciences $56.06 $0.00 021 - Library relationships $62.60 $0.00 022 - Administration of the physical plant $65.50 $0.00 023 - Personnel administration $56.00 $0.00 025 - Library operations $77.02 $0.00 026 - Libraries for specific subjects $89.99 $0.00 027 - General libraries $66.25 $0.00