LibQUAL+® & Beyond: Applying Your
Survey Results & Other Performance
Measures in Library Practice
LibQUAL+® Canada Workshop
October 24-25, 2007
Ottawa, Ontario, Canada
Martha Kyrillidou, Director, Statistics and Service Quality Programs, ARL
Steve Hiller, Director, Assessment and Planning, UW
Jim Self, Director, Management and Information Services, UVA
Martha Kyrillidou, Director
Statistics and Service Quality Programs
Association of Research Libraries
What’s in a “Library”?
A word is not crystal, transparent and
unchanged; it is the skin of a living
thought, and may vary greatly in color and
content according to the circumstances
and time in which it is used.
--Justice Oliver Wendell Holmes
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What’s in a word?
quality
library
What makes a
?
“Quality much like beauty is in the eye of the beholder”
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Library Assessment and
its Global Dimensions
• Markets and people exposed to economic
and social frameworks unheard of before
• Competing internationally
• Library users exposed to global forces
• Libraries facing similar challenges
• Libraries as the Internet
• Libraries as Google
• Libraries as Collaborative Spaces
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Library Assessment
Library assessment provides a structured process to learn
about our communities, their work and the libraries
connection to what they do
The information acquired through library assessment is
used in an iterative manner to improve library programs
and services and make our libraries responsive to the
needs of our communities.
Academic libraries do not exist in a vacuum but are part of
a larger institution. Assessment within the institution
may take place in individual areas as well as at the
broad institutional level.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Thinking Strategically About Library
Futures: Some Assessment-Related
Questions
• What is the central work of the library and how
can we do more, differently, and at less cost?
• What important services does the library provide
that others can’t?
• What advantages does the research library
possess?
• How is customer behavior changing?
• How do we add value to our customers work?
• What are the essential factors responsible for
library success now and in the future?
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Free speech wall,
Charlottesville, Sept 2006
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
IFLA: Measuring Quality
• Resources, infrastructure: What services
does the library offer?
• Use: How are the services accepted?
• Efficiency: Are the services offered costeffectively?
• Potentials and Development: Are there
sufficient potentials for future
development?
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Assessment at ARL
• A gateway to assessment tools: StatsQUAL®:
–
–
–
–
•
•
•
•
•
ARL Statistics -- E-Metrics
LibQUAL+®
DigiQUAL®
MINES for Libraries®
Library Assessment Conferences
Service Quality Evaluation Academy
Library Assessment Blog
Making Library Assessment Work
ESP Assessment
– Effective, Sustainable, Practical
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Assessment at CARL
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Assessment at SCONUL
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Assessment at CAUL
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Assessing the Value of Networked
Electronic Services
The MINES survey
Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries®
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What Are We Measuring?
Reviewing the ARL Statistics
October 2005, ARL Board approved a study to:
• Determine if there are new ways of describing
research library collections.
– What is it we are currently measuring
– Are they the right data
– Develop alternative models
• Develop a profile of the characteristics of a
contemporary research library
• Determine/develop new meaningful measures to
augment current ones to support this profile
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Quantitative Stats
(Per Bruce Thompson)
• Expenditure Focused Index (EFI)
• Current ARL stats that could be used for
benchmarking
– Collections
– User interactions
• # Participants in group presentations
• # Presentations to library groups
• # Reference transactions
– Collaborative Activities - Interlibrary loan activities
• Borrowed total items
• Loaned total items
• Set of statistics related to the digital library (from
ARL supplementary statistics)
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Qualitative Profile
Developing New Metrics (per Yvonna Lincoln)
•
•
•
•
•
•
•
•
Uniqueness of collections
Defining the value of consortia
Administrative and budgetary efficiencies
Student outcomes/student learning/graduate
success
Contributions to faculty productivity
Social frameworks/intellectual networks
Generating new knowledge
Creating the collective good with reusable
assets
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What Makes a Research Library?
•
•
•
•
•
•
•
•
Breadth and quality of collections and services
Sustained institutional commitment to the library
Distinctive resources in a variety of media
Services to the scholarly community
Preservation of research resources
Contributions of staff to the profession
Effective and innovative use of technology
Engagement of the library in academic planning
Association of Research Libraries ‘Principles of Membership’
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Group discussion
• How do you go about developing a profile
that is succinct and rich?
• Other important areas that should be part
of a qualitative profile?
• Can LibQUAL+® be used in the profiles?
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Library of the Future Will Also
Need . . .
. . . To have it’s own data collection and
management personnel, individuals who
constantly collect, analyze and prepare reports
on data regarding what services are being used,
which portions of the collection are getting the
highest usage, what materials are being lent
through interlibrary loan, and who patrons are.
Documenting the libraries contributions to quality
teaching, student outcomes, research
productivity will become critical.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Making Library Assessment Work
• ARL project approved in 2004
• Funded by participating libraries
• Site visits by Steve and Jim
– Presentation
– Interviews and meetings
– Report to the Library
• 24 libraries in U.S. and Canada visited in 200506
• Succeeded by Effective, Sustainable and
Practical Library Assessment in 2007
– Open to all libraries
– 6 libraries participating in 2007
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What We Found
• Strong interest in using assessment to improve
customer service and demonstrate value of
library
• Many libraries uncertain on how to establish,
maintain, and sustain effective assessment
• Effectiveness of assessment program not
dependent on library size or budget
• Each library has a unique culture and mission.
No “one size fits all” approach works.
• Strong customer-focus and leadership support
were keys to developing an effective and
sustainable assessment
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What are the lessons learned?
• Understanding changes in users approach to
information resources.
• Service quality improvement is a key factor.
• Understanding the impact of e-resources on library
services - TRL.
• Learning how to compete with Google.
• Upfront investment in design and development.
• Making the assessment service affordable, practical,
& effective.
• Assessment needs to be satisfying and fun.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
User Needs Assessment and
Academic Library Performance
Steve Hiller
Director
Assessment and Planning
University of Washington Libraries
An “Aha” Moment
“[Access to online resources] has changed the way
I do library research. It used to be a stage
process: Initial trip, follow-up trip, fine-tuning
trip. Now it’s a continuous interactive thing. I
can follow-up anything at any time. While I’m
writing I can keep going back and looking up
items or verifying information.”
Graduate Student, Psychology (2002 UW Libraries focus group)
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What Do We Need to Know About Our
Customers?
•
•
•
•
•
Who are our customers (and potential customers)?
What are their teaching, learning, and research interests?
How do they work? What’s important to them?
How do they find information needed for their work?
How do they use library services? What would they
change?
• How do they differ from each other in library use/needs?
How does the library add value to their work?
How does the library contribute to their success?
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
How Do We Get Customer Information?
•
•
•
•
•
•
•
•
•
Surveys
Usage statistics
Focus groups
Observation
Usability
Interviews
Embedding
Data mining (local, institutional)
Logged activities
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
University of Washington
(Site of the 2008 Library Assessment Conference!)
• Located in beautiful Seattle
metro population 3.2 million
• Comprehensive public research
university
– 27,000 undergraduate students
– 12,000 graduate and professional
students (80 doctoral programs)
– 4,000 research and teaching
faculty
• $800 million annually in federal
research funds (2nd in U.S.)
• Large research library system
– $40 million annual budget
– 150 librarians on 3 campuses
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
UW Libraries Assessment Priorities
Customer Needs, Use and Success
•
•
•
•
•
•
•
Information seeking behavior and use
Patterns of library use
Value of library
User needs
Library contribution to customer success
User satisfaction with services, collections, overall
Data to make informed and wise decisions that lead to
resources and services that contribute to user success
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
UW Libraries: Assessment Methods Used
• Large scale user surveys every 3 years (“triennial
survey”): 1992, 1995, 1998, 2001, 2004, 2007
– All faculty
– Samples of undergraduate and graduate students
– Research scientists, Health Sciences fellow/residents 2004-
• In-library use surveys every 3 years beginning 1993
• LibQUAL+™ from 2000-2003
• Focus groups/Interviews (annually since 1998)
• Observation (guided and non-obtrusive)
• Usability
• Use statistics/data mining
Information about assessment program available at:
http://www.lib.washington.edu/assessment/
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Our Latest Assessment Method
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
The Qualitative Provides the Key
• Increasing use of such qualitative methods as, comments
interviews, focus groups, usability, observation
• Statistics/quantitative data often can’t tell us
– Who, how, why
– Value, impact, outcomes
• Qualitative provides information directly from users
– Their language
– Their issues
– Their work
• Qualitative provides understanding
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Researchers and Libraries:
3 Recent Studies with Qualitative Focus
• University of Minnesota
– Extremely comfortable with electronic sources
– Interdisciplinary critical in sciences
– Inadequate methods for organizing research materials
• New York University
– Researchers (all disciplines) no longer tied to physical library
– Physical library can play a “community” role
– Expectations for info shaped by Web and commercial sector
• University of Washington (Biosciences)
– Start info search outside library space (virtual and physical)
– All digital all the time
– Could not come up with “new library services” unprompted
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Reasons for UW Libraries Biosciences
Review
•
•
•
•
•
•
Better understand how bioscientists work
Growing inter/multi/trans disciplinary work
Significant change in use patterns
Libraries responsiveness to these changes
Value of research enterprise to the University
Strengthening library connection to research
Ensuring our services and resources support
the work of the biosciences community
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Biosciences Review Process (2006)
• Define scope (e.g. what is “bioscience”?)
• Identify and mine existing data sources
– Extensive library assessment data
– Institutional and external data
• Acquire new information through a customercentered qualitative approach
–
–
–
–
Environmental scan
Interviews
Focus groups
Peer library surveys
NO NEW USER SURVEYS
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Biosciences Faculty Interview Themes
•
•
•
•
•
•
Library seen primarily as E-Journal provider
Physical library used only for items not available online
Start information search with Google and PubMed
Too busy for training, instruction, workshops
Faculty who teach undergrads use libraries differently
Could not come up with “new library services”
unprompted
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Biosciences Focus Group Themes
• Content is primary link to the library
– Identify library with ejournals; want more titles & backfiles
• Provide library-related services and resources in
our space not yours
– Discovery begins primarily outside of library space with Google and Pub
Med; Web of Science also important
– Library services/tools seen as overly complex and fragmented
• Print is dead, really dead
– If not online want digital delivery/too many libraries
– Go to physical library only as last resort
• Data and reference management important to some
– Bioresearcher toolkit, EndNote, JabRef, StatA
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Biosciences Task Force Recommendations
• Integrate search/discovery tools into users workflow
• Expand/improve information/service delivery options
• Make physical libraries more inviting/easier to use
– Consolidate libraries, collections and service points
– Reduce print holdings; focus on services
•
•
•
•
Use an integrated approach to collection allocations
Get librarians to work outside library space
Lead/partner in scholarly communications & E-science
Provide more targeted communication and marketing
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Biosciences Review Follow-up : 2007 Actions
• Appointed a Director, Cyberinfrastructure Initiatives &
Special Asst to the Univ Libr for Biosciences & E-Science
• Libraries Strategic Plan priorities for 2007 include:
– Improve discovery to delivery (WorldCat Local etc.)
– Reshape our physical facilities as discovery and learning centers
– Strengthen existing delivery services, both physical and digital,
while developing new, more rapid delivery services
– Enhance and strengthen the Libraries support for UW’s scientific
research infrastructure
– Do market research before developing & promoting services
• Informed development of Libraries 2007 Triennial Survey
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
In God We Trust:
All Others Must Bring Data
UW Triennial Survey 2007 – Selected Questions
Mode of access/physical library uses and users
Resource type importance
Sources consulted for research
Primary reasons for using Libraries Web sites
Information literacy
Libraries contribution to work and academic success
Useful library services (new and/or expanded)
Satisfaction
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
UW Triennial Library Survey
Number of Respondents and Response Rate 1992-2007
2007
2004
2001
1998
1995
1992
Faculty
1455
36%
1560
40%
1345
36%
1503
40%
1359
31%
1108
28%
Grad/Prof
Students
580
33%
627
40%
597
40%
457
46%
409
41%
560
56%
Undergrads 467
20%
502
25%
497
25%
787
39%
463
23%
407
41%
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Mode of Library Use by Group 2007
(weekly or more often)
Undergrad
Rem ote &
Visit
39%
Visit Only 27%
Grad
Visit
Only 2%
Faculty
Visit
Only 1%
Rem ote &
Visit
19%
Rem ote
Only
14%
Rem ote &
Visit
45%
Rem ote
Only
47%
Rem ote
Only
72%
Non- Weekly
20%
Non- Weekly
6%
Non- Weekly
8%
I only wish I could reproduce the graduate reading room in my home because I do so
much of my reading/research online now. Oh well, at least I can be in my slippers.
Associate
Professor,
Psychology
LibQUAL+® & Beyond,
Ottawa, Ontario,
Canada, October 24-25, 2007
UW Faculty Mode of Use by Academic Area
1998/2007 (w eekly or more often)
Non- Weekly
17%
Rem ote
Only
45%
Non- Weekly
5%
Rem ote
Only
87%
Non- Weekly
25%
Rem ote
Only
26%
Rem ote &
Visit
7%
Visit Only10%
Health Sci
1998
Health Sci
2007
Rem ote
Only
47%
Rem ote &
Visit
51%
Rem ote &
Visit
42%
Rem ote &
Visit
17%
Science-Engin Science-Eng
1998
2007
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Non- Weekly,
9%
Rem ote
Only
72%
1%
0%
Non- Weekly
15%
Rem ote
Only
23%
Rem ote &
Visit
39%
Rem ote &
Visit
32%
Visit Only 6%
Non- Weekly
9%
Visit Only
10%
Hum-Soc Sci
1998
1%
Hum-Soc Sci
2007
Undergrad Mode of Library Use 1998/2007
(w eekly or more often)
Non- Weekly
23%
Non- Weekly
20%
Rem ote
Only 7%
Rem ote
Only
14%
Rem ote &
Visit
35%
Visit Only 35%
Undergrad 1998
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Rem ote &
Visit
39%
Visit Only 27%
Undergrad 2007
Physical Library Users by Group for
Selected Libraries (2005 In-Library Use Survey)
100%
90%
Undergrads 70%, Grads 25%, Faculty/Staff 5%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Main
Art
Business
Undergrad
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Health Sci
Grad
Math
Faculty
Music
Other
Undergrad
Physical Library Use by Academic Area
(2005 In-Library Use Survey)
Libraries Used
Biology Undergrads (n=126)
Other
Other
Sci
Chem
9%
Health Sci Library Use - Grad by Area (n=186)
Health
Sci
Unknown
Main
33%
Other
UGL
44%
Health Sci
0%
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
20%
40%
60%
80%
100%
What Do They Do in the Library?
Activities by Group (UW 2005 In-Library Use Survey)
70%
60%
50%
Undergrads
Grad Students
40%
Faculty/Staff
30%
20%
10%
0%
Ask for help
Look for
material
Copy
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Work alone
Work in groups
Use Lib
Computer
Use printer
In-Person Visits1998-2007
(% From "triennial" surveys)
80%
Grad
70%
Undergrad
Undergrad
60%
50%
Grad
Faculty
40%
30%
Faculty
20%
1998
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
2001
2004
2007
Off-Campus Remote Use 1998-2007
(Percentage using library services/collections at least 2x week)
70%
70%
Grad
60%
60%
50%
50%
Faculty
40%
40%
30%
30%
Undergrad
20%
20%
10%
10%
0%
0%
1998
2001
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
2004
2007
Importance of Books, Journals, Databases
Academic Area
(2007, Faculty, Scale of 1 “not important” to 5 “very important)
5
4.75
Journals
>1985
4.5
4.25
Books
4
Journals
<1985
3.75
Bib
Databases
3.5
3.25
3
Health Sciences
Books
Science-Engineering
Journals<1985
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Bib Databases
Hum-Soc Science
Journals>1985
Book/Older Journal Importance
By Selected College/School (2007, Faculty)
4.25
OceanFish
Older Journal
4
Science
3.75
Public Health
Engineering
Business
3.5
Social Sci
Forestry
Fine Arts
Humanities
Education
Medicine
3.25
3
Nursing
2.75
3
3.25
3.5
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
3.75
4
Book
4.25
4.5
4.75
5
Overall Collections Satisfaction
in Selected Hum/Soc Sci Colleges (2007, Faculty and Grads)
4.6
Faculty
Faculty
4.5
Grad
4.4
Grad
4.3
4.2
4.1
4
3.9
3.8
Architect
Fine Arts
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Humanities
Social Sci
Business
Education
Sources Consulted for Information on
Research Topics (2007, Scale of 1 “Not at All” to 5 “Usually”)
Undergrad
Open Internet
Search
Grad
Faculty
Open Internet Ref
Source
Bibliographic
Databases
2.5
2.75
3
3.25
3.5
3.75
4
4.25
4.5
“If it’s not on the Internet, it doesn’t exit.” My students at all levels behave this way.
They also all rely on Wikipedia almost exclusively for basic information.
Associate Professor, English
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Primary Reasons for Using Libraries
Web Sites 2007 Faculty (at least 2x per week)
75%
65%
55%
45%
35%
25%
15%
Health Sci
Library Catalog
Science-Engin
Bib Database
Hum-Soc Sci
Online journal articles
Theabiltyocsfu-xrpdeahticlsonrugebay
subcriptonmyasefhlibrndctaomyesrh.
NeurobilgyGadStn
The ability to access full-text or PDF research articles online through the library
subscriptions is my primary use of the library and is central to my research. Neurobiology
Grad Student
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Information Literacy: Importance to Undergrad
Success & Rating Student Performance
(% of Faculty marking 4 or 5 on scale of 1 “Low” to 5 “High” in 2007 Triennial Survey)
Knowing about
plagiarism
Cite sources correctly
Critically evaluating
info sources
Performance
Importance
Finding Info on Web
Finding Info in Library
Finding and Refining
Research Topics
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
It is difficult to help students understand how to use sources, what the libraries can provide
them, and help them appreciate the resources available to them beyond Google. Do you have
any suggestions? Assistant Professor, Art
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Undergrad Rating of Usefulness
(Mean score on scale of 1 “Not Useful” to 5 “Very Useful”)
Course reserves
Library subject
guides
Library reference
assistance
Research consult
with librarian
Librarian class
presentation
3
3.25
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
3.5
3.75
4
4.25
4.5
Usefulness of New/Expanded Services
Faculty and Grad (% responding yes for each service)
Scan on Demand
Digitize collections
Office Delivery of
Books
Grad
Faculty
Integrate services into
campus Web sites
Manage your info and
data
0%
10%
20%
30%
40%
50%
60%
70%
80%
You’re considering a free scanning service for journal articles? That would change my life!
Wow! I didn’t even know I could want that. Now I want that!
Post-Doc, Oceanography
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Usefulness of New/Expanded Services
Undergrads (Physical Library Services in Red)
Quiet work/study areas
Increase weekend hours
More library computers
Integrate services into
campus Web sites
Book Self-Check out
Group/Presentation
Spaces
Consult on finding info
0%
10%
20%
30%
40%
50%
60%
Odegaard needs a facelift. The lighting in terrible and the workspaces are old--not somewhere
that you want to spend hours studying. I live in Suzzallo, however and I love it. Undergrad
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Libraries Contribution to: (Scale of 1 “Minor” to 5 “Major”)
Being a more
productive
researcher
Keeping current in
your field
Finding info in new or
related areas
Efficient Use of Time
Grad
Faculty
Academic Success
3
3.25
3.5
3.75
4
4.25
4.5
4.75
The UW libraries and librarians are the BEST. Our ability to access the system from the road (or home) and to review/download
current articles is absolutely super. The resources on HealthLinks have helped train many young doctors and saved COUNTLESS
lives. --Associate Professor, Medicine
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
2007 Triennial Survey Key Findings
• Library satisfaction exceptionally high
• Long-term changes in mode of use continue
– Sharp increase in off-campus remote use by faculty/grad
– Library as place still important to undergraduates
• Open Internet gains as primary discovery medium
– Library provided bibliographic databases decline in importance
• Users want content delivered to them in their space &
desired format
• Faculty see information literacy as important to student
success
– Student performance in this area is rated low
– Student evaluation of effectiveness is mixed
• Libraries is major contributor to faculty research
productivity and grad student academic success
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What We’ve Learned about the UW
Community
• Libraries are still important source of information;
however library less integrated into work/learn “flows”
• Library needs/use patterns vary by and within academic
areas and groups
• Remote access is preferred method for faculty and grad
students and has changed the way they use libraries
• Faculty and students find information and use libraries
differently than librarians prefer them too
• Library/information environment is perceived as too
complex; users find simpler ways (Google) to get info
• Customers cannot predict the Libraries future
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
How UW Libraries Has Used Assessment
•
•
•
•
•
•
•
•
•
•
Extend hours in Undergraduate Library
Create more diversified student learning spaces
Eliminate print copies of journals
Enhance usability of discovery tools and website
Provide standardized service training for all staff
Stop activities that do not add value
Consolidate and merge branch libraries
Change/reallocate collections allocations
Change/reallocate staffing
Support budget requests to University
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Closing the Loop:
Using Data Effectively in Management
• Use multiple assessment methods
• Focus on user work and how they find & use information
• Increase reliance on qualitative info to identify issues
from user perspective
• Learn from our users
• Partner with other campus programs/institutions
• Mine/repurpose existing data
Decisions based on data not
assumptions -“assumicide”
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Our Challenge: Maintain High Value and
Satisfaction (UW Overall Satisfaction 1995-2007)
4.6
4.6
Faculty
4.56
4.5
4.44
4.4
4.33
4.3
4.33
4.34
Faculty 4.25
Grad 4.36
Undergrad 4.36
4.26
4.32
4.2
4.1
4.4
4.3
4.2
Grad 4.18
4.11
4.22
4.1
4
3.9
4.5
4
Undergrad 3.97
3.99
3.9
3.8
3.8
1995
1998
2001
2004
2007
You guys and gals rock!!!!!! We need to invest in our library system to keep it the best system
in America. The tops! My reputation is in large part due to you. Professor, Forest Resources
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Measures that Matter: Designing a
Balanced Score Card
Jim Self, Director,
Management Information Services, UVA
The University of Virginia
• 14,000 undergraduates
– 66% in-state, 34% out of state
– Most notable for liberal arts
– Highly ranked by U.S. News
• 6,000 graduate students
– Prominent for humanities, law,
business
– Recent expansion in sciences
• Located in Charlottesville
– Metro population of 160,000
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
The University Libraries
•
•
•
•
•
•
5 million volumes
15 libraries
350 FTE staff
$35 million budget
Top 20 in ARL
2005 ACRL Academic Library of the Year
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
U.Va. Library Innovations
•
•
•
•
•
•
•
Electronic Text Center -- 1992
Customer Surveys – 1993, 1994
LEO Faculty Delivery -- 1994
MIS unit – 1996
Library café -- 1998
Balanced Scorecard – 2002
Scholars’ Lab -- 2006
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Management Information Services
•
•
•
•
•
•
•
MIS committee formed in 1992
Evolved into a department 1996-2000
Currently three staff
Coordinates collection of statistics
Publishes annual statistical report
Coordinates assessment
Resource for management and staff
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Collecting the Data at U.Va.
•
•
•
•
•
Customer Surveys
Staff Surveys
Mining Existing Records
Comparisons with peers
Qualitative techniques
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Corroboration
• Data are more credible if they are
supported by other information
• John Le Carre’s two proofs
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
UVa Customer Surveys
• Faculty
– 1993, 1996, 2000, 2004
– Separate analysis for each academic unit
– Response rates 59% to 70%
• Students
–
–
–
–
1994, 1998, 2001, 2005
Separate analysis for grads and undergrads
Undergrad response rates 43% to 50%
Grad response rates 54% to 63%
• LibQUAL+® in 2006
– Response rates 14% to 24%
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Analyzing U.Va. Survey Results
• Two Scores for Resources, Services,
Facilities
– Satisfaction = Mean Rating (1 to 5)
– Visibility = Percentage Answering the Question
• Permits comparison over time and among
groups
• Identifies areas that need more attention
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Reference Activity and Visibility
in Student Surveys
7,000
6,008
75%
Visibililty
Reference Questions
Recorded per Week
64%
Visibility
39%
Visibility
34%
Visibility
1,756
10%
1,000
1993
1994
1995
1996
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
1997
1998
1999
2000
2001
2002
2003
2004
Constructing a Balanced
Scorecard
• Select a limited number of meaningful and
measurable indicators for each dimension
• Select targets for each indicator
• Four dimensions:
– User perspective
– Internal processes perspective
– Finance perspective
– Future/growth perspective
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Importance of Targets
• Measure quantitatively
• Set challenging, but achievable targets
• Consider two sets of targets:
– Complete success
– Partial success
• Aggregate regularly to provide feedback
• Address problems that are revealed
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
The BSC at the U.Va. Library
•
•
•
•
•
Implemented in 2001
Results tallied FY02 through FY07
Completing metrics for FY08
Reporting results for FY07
A work in progress
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Choosing the Metrics
--Reflecting Values
• What is important?
• What are we trying to accomplish?
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Choosing the Metrics
--Diversity and Balance
• Innovations and operations
• Variety of measurements
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Choosing the Metrics
--Ensuring validity
• Does the measurement accurately
reflect the reality?
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Choosing the Metrics
--Being Practical
•
•
•
•
Use existing measures when possible
Use sampling
Collect data centrally
Minimize work by front line
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
What Do We Measure at U.Va.?
•
•
•
•
•
•
Customer survey ratings
Staff survey ratings
Timeliness and cost of service
Usability testing of web resources
Success in fund raising
Comparisons with peers
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Reviewing the Perspectives
•
•
•
•
User
Internal Processes
Finance
Learning and Growth
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Balanced Scorecard
UVA Fiscal Year 2007
Target1
Target2
Not Met
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric U.1.A: Overall rating in
student and faculty surveys
• Target1: An average score of at least
4.25 (out of 5.00) from each of the
major constituencies.
• Target2: A score of at least 4.00.
FY07 Result: Target2
– Graduate students 4.08
– Undergraduates 4.11
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric I.1.A: Processing time for
routine acquisitions
• Target1: Process 90% of in-print books
from North America within one month.
• Target2: Process 80% of in-print books
from North America within one month.
• Result FY07: Target1.
– 94% processed within one month.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric I.2.A. Staff Rating of
Internal Communications
• Target1: Positive scores (4 or 5) on 80% of
responses to internal communications
statement in biennial work life survey.
• Target2: Positive scores on 60% or
responses.
• Result FY07: Did not meet target.
– 48% or responses were positive.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric F.1.B. Library spending
compared to University
expenditures
• Target1: : The University Library will
account for at least 2.50% of the
University’s academic division
expenditures.
• Target2: : The Library will account for at
least 2.25% of expenditures.
• Result FY07: Target1.
– 2.71% ($26.2M of $963M)
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric F.1.C. Amount of
unrestricted development receipts.
• Target1: Increase unrestricted (or
minimally restricted) giving by 10% each
year.
• Target2: Increase of 5% per year.
• Result FY07: Target1.
– FY07 unrestricted receipts were $871,000; target was
$411,000.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric F.2.A: Unit Cost of Electronic
Serial Use
• Target1: There should be no increase in
unit cost each year.
• Target2: Less than 5% annual increase in
unit cost.
• Result FY07: Target1.
– Cost per journal article downloaded in FY07 was
$1.98, compared to $2.10 in FY06.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric L.2.C. Comparing librarian
salaries to peer groups.
• Target1: Average librarian salaries should
rank in the top 40% of average salaries at
ARL libraries.
• Target2: Rank in top 50%.
• Result FY07: Target1.
– Ranked 33 of 113. (Top 28%)
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Trying your hand at a Scorecard
• Devise one or two metrics per dimension
– Should be something that matters
– How would you measure it?
– How do you define success?
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Two more metrics from U.Va.
• Representing values of the Library
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric U.3.A: Circulation of new
monographs
• Target1: 60% of all newly cataloged print
monographs should circulate within two
years.
• Target2: 50% should circulate within two
years.
• Result FY07: Target1.
– 63% of monographs purchased in FY05 circulated
within two years.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
Metric U.4.B: Turnaround time for
user requests
• Target1: 90% of user requests for new
books should be filled within 7 days.
• Target2: 80% of user requests for new
books should be filled within 7 days.
• Result FY07: Target1.
– 77% filled within 7 days.
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007
To summarize…
The Balanced Scorecard
• Reflects the organization’s vision
• Clarifies and communicates the vision
• Provides a quick, but comprehensive,
picture of the organization’s health
LibQUAL+® & Beyond, Ottawa, Ontario, Canada, October 24-25, 2007