Analyzing Data,
Getting Results
Some practical, not-tooburdensome tips and tricks
Jenn Riley
University of North Carolina at Chapel Hill
3/5/13
SCELC Research Day
Evidence-driven
decisions are a
powerful guide for
library operations.
2
3/5/13
SCELC Research Day
“The plural of anecdote
is not data.”
3
Sometimes attributed to Frank Kotsonis.
After a quote with the opposite meaning, by Raymond Wolfiger.
3/5/13
SCELC Research Day
“There are three kinds
of lies – lies, damned
lies, and statistics.”
4
Mark Twain, perhaps after Benjamin Disraeli.
Materials to
buy/license/accept/
digitize/keep/preser
ve
Effectiveness
of/satisfaction with
procedures/services
Evaluating a pilot
service or project
Projecting future
expenditures
SCELC Research Day
Existence/hours of
service points
Designing web sites
and other online
resources
3/5/13
Using data for planning library
operations
5
Proposes definitions of value for cataloging:
Discovery success
Use
Display understanding
Data interoperability
Support for FRBR user tasks
Throughput/timeliness
Support administrative goals
SCELC Research Day
ALCTS Heads of Technical Services in Large
Research Libraries Interest Group, Task
Force on Cost/Value Assessment of
Bibliographic Control (2010)
3/5/13
Both cost and value are key
6
• Benefits of manually enhanced metadata
for images
• Comparing effort to utility for specific
EAD elements
See Chapman, Joyce. “Metrics & Management: Cost & value of metadata
workflows.” SAA 2011.
http://www.academia.edu/1708422/Return_on_Investment_Metadata_met
rics_and_management
SCELC Research Day
• By Joyce Chapman, then at North
Carolina State University
3/5/13
Example studies
7
Change over time
Identifying unmet
needs
SCELC Research Day
Cost per unit
produced
Predicting impact of
a change
3/5/13
Some common analyses
Error/problem rate
8
3/5/13
SCELC Research Day
Back to library
scenarios
9
• Staff time
• Facilities management costs
• Benefits
• Number and type of visitors, and how they use it
• Service transactions completed
• Specific services used at the location
• Other data to collect
• Usage by time of day
• Calculate cost per transaction
SCELC Research Day
• Who is using what and when?
• How can we most effectively staff them?
• Costs
3/5/13
Existence/hours of service
points
10
• Initial purchase/license
• Ongoing license/maintenance
• Staff for
cataloging/processing/digitizing/ingesting/preserving
• Software
• Hardware/storage
SCELC Research Day
• Should we acquire, make more accessible, or keep this?
• Costs
3/5/13
Materials to
buy/license/accept/digitize/
keep/preserve
• Benefits
• Current and predicted future use
• Opportunity for transformative use
11
• Is the pilot achieving its aims?
• Does this [whatever] do what we thought it
would?
• What collateral effects will it have?
• Were the assumptions we made correct?
• Data collection will be varied for this task
SCELC Research Day
• Is the cost/benefit ratio appropriate?
• What is the raw cost?
• But it’s not all about cost/benefit:
3/5/13
Evaluating a pilot service or
project
12
SCELC Research Day
• A/B testing
• User-centered design
• Satisfaction surveys with previous
iterations, similar sites, or prototypes
• Web stats for previous iterations or similar
sites
• Task-based usability testing
• Don’t forget the cost of sustaining it once
you have it up!
3/5/13
Designing web sites and other
online resources
13
•
•
•
•
•
•
User surveys
Ratio of potential to actual users
Ratio of returning to non-returning users
Error/failure rates
Time from request to delivery
Time tracking during staff activity
SCELC Research Day
• What parts of our current service are users most and
least happy about?
• What are the ineffieciences in our procedure for
[whatever]?
• Some data collection ideas
3/5/13
Effectiveness of/satisfaction
with procedures/services
14
Projecting future
expenditures
Define its lifecycle
Amortize purchase cost
Add in maintenance costs
Compare to use as context
• Staff
• Educated guess at raises, turnover, benefit costs
changes
SCELC Research Day
•
•
•
•
3/5/13
• Equipment
• Consider:
• Inflation
• Past trends
• Upcoming sea changes
15
3/5/13
SCELC Research Day
Strategies for getting
data that can be
analyzed
16
SCELC Research Day
• Circulation
• COUNTER/SUSHI
• Physical visitors
• Web hits
• Social media engagement
• Attendance at events/sessions
3/5/13
Tracking use
17
SCELC Research Day
• Can be effective when collected as a
representative snapshot
• Options for data collection
• Clipboard next to a clock
• Spreadsheet
• Free time tracking apps
3/5/13
Tracking time
• Make it as simple as possible
18
• Initial purchase
• Maintenance contracts for big-ticket items
• Amortize big costs over time in service
SCELC Research Day
• Staff time
• 2080 hours per year is full time
• Standard benefit percentages
• Materials (including software)
3/5/13
Calculating costs
• Overhead
• Universities typically have standard rates
19
SCELC Research Day
• Both objective and subjective criteria
• Typically best when done as a sample
• Consider both automated and manual
means to locate errors for study
3/5/13
Calculating error rates
20
Categorization
• Can be done at time of data collection, or
afterwards
• Good idea to have some sense of
categories at the beginning of the study
SCELC Research Day
• Compare size of groups to one another
• Compare effort spent on one group to another
• Compare priority/value of one group to
another
3/5/13
• Putting things into like groups
21
Calculating benefit
• Survey – ask about knowledge level before
and after
• Pre- and post-tests
• Indirect measures
• Number of people reached
• Use
SCELC Research Day
• Over time
• After an interaction
3/5/13
• Change in knowledge or status
22
3/5/13
SCELC Research Day
Additional data analysis
strategies
23
SCELC Research Day
• Code qualitative data to make it
processable
• Make sure you pick a representative and
consistent sample
• Extrapolate based on known data when
you need to
• ALWAYS do a sanity check
• Spreadsheets are your friend
3/5/13
Mechanics
24
SCELC Research Day
• Context is key
• Don’t be paralyzed by a perceived need for
perfection
• Know your basic analysis plans before you
collect/identify data
• Utilize pilot projects to generate data where
there is none
• Use the right tool for the job
• Document your assumptions
• It’s OK to use “napkin math”
3/5/13
More advice
25
It will make your next
decision easier.
3/5/13
SCELC Research Day
Get in the habit of
collecting data.
26
SCELC Research Day
Questions and
discussion
3/5/13
Thank you!
jennriley@unc.edu
27