Quants, Quals, and Pals
Library Assessment in the 21st Century
Steve Hiller
University of Washington Libraries
CUNY Assessment Conference
June 6, 2014
Four Useful Assessment Assumptions
•
•
•
•
Your problem/issue is not as unique as you think
You have more data/information than you think
You need less data/information than you think
There are useful methods that are much simpler than
you think
Adapted from Douglas Hubbard, “How to Measure Anything” (2010)
What Will We Assess and Measure
• Institutional assessment efforts should not be
concerned about valuing what can be measured,
but instead about measuring what is valued. A.
Astin
• Not everything that can be counted counts, and
not everything that counts can be counted. Attributed
to Albert Einstein
Some Questions Before You Start
What Do You Need to Know and Why?
• What do you need to know
• Why do you need to know it
• How will you collect data (is it already available?)
• When do you need the data
• How will you analyze the data
• How will you use data
• What outcomes are expected
The answers will determine methods used
Assess What’s Important
• Contribution to student and faculty success
• Contribution to institutional mission/visibility
And, perhaps (depending on library and institution type)
• Accountability/Efficiency/Effectiveness
• Use
• Revenue generation (including fund raising)
• Benchmarking and comparisons with others
Not how good is this library? Rather, 'How much
good does it do?' (R.H Orr, 1973)
Recent Trends in Library Assessment and
Performance Measurement
• Greater reliance on external measures and user
impacts; aligned with planning
• Customer-centered library concept
• Outcomes-based assessment and metrics that made
use of multiple methods, including qualitative
• Collaboration with institutional and other partners
• Demonstrating library impact and value on
individuals and communities
Changing the Focus:
Quants, Quals, Pals
• Quantitative
– Focus changes from counting to analysis and action
• Qualitative
– Focus changes from research study to practical methods
which emphasize understanding
• Pals (Partners)
– Focus changes from using our own data and internal
assessments to working with other on broader collaborative
assessments
Quants
• First mentioned in science fiction for those with
exceptional quantitative skills and analysis
• SF Quants were often androids, or enhanced with
implants or through breeding
• SF Quants, especially androids, had a high
termination rate
• Quants became human (?) in the late 20th century as a
name for those with high analytical skills, especially
in business
Quants in the 20th Century:
We Measure What We Can Count
• Inputs - Size
–
–
–
–
–
Collections
Budget
Staff
Facilities
Discovery tools
• Size used as indicator of
excellence
• Outputs - Usage
–
–
–
–
–
Collections
Reference services
Instruction sessions
Facilities usage
Web sessions
• May indicate if inputs
used but don’t tell us
what users were able to
accomplish
Data Accuracy
Many Library Statistics Are Self-Reported
and Lack Independent Verification
The Challenge for Libraries
• Traditional statistics/measures are no longer sufficient
– Emphasize inputs/outputs – how big and how many
– Do not tell the library’s or customers’ story
– Usually not aligned with organizational goals and plans or
support library strategic directions and objectives
– Do not capture the impact and value of the library
• Need to demonstrate the difference the library makes
– To the individual, community and the organization
Why your kids will want to be data scientists
Big money, lots of jobs, challenging
and they’ll take care of you in old
age!
Quants in the 21st Century:
Data Analysis and Action
Finding meaning and application in:
• Big data sets
• Data mining
• Data exchange
• Actionable data
• Data formats
• Metadata
Supported by sophisticated and robust data analysis
tools
Some Quant Examples
• Infosphere use and application
• Relationship between library use and student, faculty
and research success
• Data trends
• Data comparability and integration
• Analytics
Quals - Definition
An abbreviation for the word Quality. Quals is used
when the item being described surpasses the quality of
everything else, by far, leaving only one word left Quals.
Only to be used by the super-cool and often used more
frequently when drunk and rowdy.
Source: Urban Dictionary
Classic Qual(itative)s
• Social Science research methodology using timeconsuming protocols
• Interviews and observations (ethnographic)
• Focus groups
• Relatively few library qualitative studies until 25
years ago
Quals: The Past 25 Years
• Widely employed in libraries and for outcomes-based
efforts
• Shorter, more focused and actionable studies
• Usability
• User-centered design
• Ethnographic
• Advocacy (narratives)
• Qualitative work integral part of market research
FutureQUAL®
• Merging quantitative and qualitative as data
collection and analysis tools incorporate digital
technology
• Understand individual and group behavior beyond
information, libraries and organizations
• Narratives and personas closely linked to customized
services, marketing and advocacy
Pals (Partners)
Partners and Collaborators
• Partners
– Close, recognized
relationships
– Work towards similar
goals and outcomes
– Value each others
contributions
– Watch each others backs
– Ride off in the sunset
together
• Collaborators
– Brought together for
specific efforts
– May have different goals
– Bring special expertise
– Watch your back!
– Often shot at the end of
spy/war movies
Partners and Collaborative Assessment:
• Collaboration with other institutional programs/units
• Collaboration with other libraries/consortia/
organizations
• Linked to strategic planning, accountability, advocacy
• Fostering assessment & performance measurement that
focuses on community success
What Libraries Bring to Academic
Partnerships
• Big picture understanding of higher education, scholarly
communication, the institution and its programs
• Close links to academic programs through liaisons
• Service to entire community
• Student-centered approach and research collaboration
• Diverse skills and expertise
From Library-Centered to External Partnerships
Research
Instruction
Traditional: Collection
Development
Traditional: Bib Instruction
•
Collections most valued library
research support. Remains important
Now: Information Management
•
•
•
Now: Teaching & Learning
•
Embed library experts within research
•
programs
•
Support research cycle from discovery
through communication and archiving •
Collaborate with a variety of partners
•
•
Library and tool focused
Not always tied to course work or
outcomes
Evaluation by pre- and post-tests
Focus on student learning
outcomes and authentic assessment
Partnerships & contribution to
teaching/program outcomes
Broad use of educational
technology
Catalyst for Partnerships Was Often External
• Association and information standards
• Integrated approach to accreditation (professional and
institutional)
• Consortia
• Value and impact studies, advocacy
Library Value & Impact
• Value of Academic Libraries (Oakleaf, 2010)
• Lib-Value Project (ARL, Tennessee)
• Library Impact Data Project (Huddersfield/JISC)
• Methods and procedures for assessing the impact of
libraries (ISO 16439)
• Discovering the Impact of Library Use and Student
Performance (Wollongong)
• ACRL Assessment-in-Action project
• University of Minnesota Library Data & Student
Success
Value & Impact: Key Themes
• Connecting
institutional &
library data
• Aligning library’s role with core mission & values of
institution: “allow institutional missions to guide
library assessment” (Oakleaf 2010, p, 30).
• Communicating value to stakeholders
U.S. Regional Accreditation Standards for
Institutions of Higher Education NWCCU 2010
http://www.nwccu.org/index.htm
• Institutional mission, core themes and expectations
• Resources and capacity
–
–
–
–
–
–
Governance
Educational resources
Library and information resources
Financial resources
Leadership and management
Infrastructure
• Planning and implementation
• Assessment
• Mission fulfillment, adaptation and sustainability
U.S. Regional Higher Education Accreditation:
Library Standards (NWCCU)
The institution holds or provides access to library and information resources
with an appropriate level of currency, depth, and breadth to support the
institution’s mission, core themes, programs, and services
Planning for library and information resources is guided by data that
include feedback from affected users and appropriate library and information
resources faculty, staff, and administrators.
The institution provides appropriate instruction and support to enhance
efficiency and effectiveness in obtaining, evaluating, and using library and
information resources that support its programs
The institution regularly and systematically evaluates the quality, adequacy,
and utilization of library and information resources and services, wherever
offered and however delivered.
ACRL Standards for Libraries in Higher
Education http://www.ala.org/acrl/standards/standardslibraries
The Standards assume that libraries:
•Develop user-centered, measurable outcomes that articulate
specifically what the user is able to do as an outcome
•Conduct assessments that may be quantitative and/or
qualitative;
•Collect evidence from assessments that demonstrate degree
of success
•Use assessment data for continuous improvement of library
operations.
ACRL Standards for Libraries in Higher
Education (2010)
Institutional Effectiveness
• The library defines, develops and measures outcomes in the
context of institutional mission and aligned with
institutional outcomes
• The library develops and maintains a body of evidence that
demonstrates its impact
• The library articulates how it contributes to student learning,
ACRL Standards: Libraries Contribution to
Institutional Effectiveness
The library contributes to student recruitment,
retention, time to degree, and academic success.
Students improve their academic performance over their
college experience through their contact with the library.
Students who have more contact with the library show higher levels
of improvement in performance on standardized tests.
Students are able to describe the role of the library in improving
their academic performance from freshman to senior.
Partnerships in the 21st Century
• Institutional teams working on institutional level
issues with functional specialists
• Multi-institutional level networking and expertise
sharing
• Defining institutional and programmatic goals,
priorities, and outcomes
• Blurring of administrative and academic boundaries
• Computational savvy staff
Lorcan Dempsey’s 3 Challenges (2013)
• Engagement
– Libraries work to create distinctive value in the research, learning and
teaching workflows of their users in ways which go beyond the
provision of collections
• Rightscaling
– Libraries moving from “institution scale” to collaborative, shared
systems – especially through functional and regional consortia. The
need for local infrastructure declines. As libraries want to emphasize
impact and engagement, and de-emphasize activities which do not
create distinctive local value, rightscaling becomes a key question.
• Institutional innovation
– Library as an organization which reconfigures to map changes in the
user environment and expectations. Libraries have to develop new and
routine ways of collaborating to achieve their goals, which involves
evolution of organizational, cultural and communication approaches.
– The learning that flows from institutional innovation
Four Library Assessment Questions
• What do we need to know about our communities and
customers to make them successful?
• Who are our partners in collaborative assessment?
• How do we measure the effectiveness of our services,
programs and resources and how they contribute to
institutional and user success.
• What do our stakeholders need to know in order to
provide the resources needed for a successful library?
2014 Library Assessment Conference
August 4-6, University of Washington
http://libraryassessment.org/