Measuring the New Measures
Culture, Counting and Plumbing
One Library’s Take on “New Measures”
ALCTS: Collection Management and Development Section
Atlanta, June 2002
Joe Zucca
Assessment, Planning and Publications Librarian
University of Pennsylvania Library
Measuring the New Measures
Topics For This Presentation:
1. Placing the “New Measures” into Penn’s Context
2. Building the Plumbing for Measurement: The Penn Library Data Farm
3. Mining Data for Information
4. Can Librarians Count?
New Measures…Old Ways, and a High Altitude View From the Field
Measuring the New Measures
1. The Penn Context
The Proliferation of Electronic Resources
Article indexes, e-journals and other full-text resources
7000
6875
6000
4932
Number of Titles
5000
4000
3000
2438
2000
1394
1000
0
3
9
1991
1993
86
1996
1999
2000
2001
2002
Measuring the New Measures
1. The Penn Context
The Growth of Expenditures for Electronic Resources
Annual Growth of Expenditures for Electronic Information Based on 1991
1000%
900%
PCT Increase in Expenditure
800%
700%
600%
500%
400%
300%
200%
100%
0%
1991
1993
1996
1999
2000
2001
E-Resources as a percent of acquisitions budget
1991
3.7%
1993
3.2%
1996
5.5%
1999
13.2%
2000
13.9%
2001
15.7%
Measuring the New Measures
1. The Penn Context
Strategy-Formation
1996-2000 Strategic Plan: Develop new ways of conceptualizing and
measuring library effectiveness … to promote within the Library growing
accountability and responsiveness to changing information environments
and to track progress toward our goals.
Measuring the New Measures
1. The Penn Context
Administrative Imperative
1999/2000 External Review Report: The Committee did not address
the difficult questions about quantitative indices and performance
measures…because no satisfactory answers exist. …we recommend
that the Penn Library become an active participant [in the ARL new
measures initiative].
Measuring the New Measures
1. The Penn Context
Accountability
From: The Council of Deans, UPenn
To: Vice Provost and Director of Libraries, Paul Mosher
1.
Who uses the library(ies)—breakdowns by school, status (student/standing
faculty/research faculty/adjuncts/staff/ full vs part time)
2.
What types of uses—books, journals, reference data, study space, etc.
3.
Intensity of use—time and volume measures for both onsite and remote uses
4.
Relationships among the above—type of user by type of use by intensity etc.
5.
Patterns in the above over time
6.
Comparisons with peer institutions re the above
7.
Patterns/trends in intellectual areas of interest to users
8.
Patterns/trends in information resource use beyond the library—where do
Penn people go for what types of information and how?
9.
How do other universities develop library budgets and apportion costs among
peers?
Measuring the New Measures
1. The Penn Context
Utilitarian Attitude/Willingness to Experiment
Leverage the data you have; usually they’re rigorous enough to
validate and inform organizational experience and judgment
Measuring the New Measures
User
transaction
Building the Plumbing for Measurement
Tools That Deliver Service Also
Enable Measurement
Vendor
Reports
Web & Other
Servers
Server Logs
Library Data Farm
Log archive
raw log output
parsed & stored
Perl
Data harvest &
processing
systems
Processed data files
Reports and Analysis
Knowledge base:
statistical repository,
data files, assembled
for access
Web host
Staff query interface
Voyager/DLXS
SAS/Excel/MS Access
Measuring the New Measures
The Library Data Farm: Mining Data for Management Information
Measuring the New Measures
Circulation Statistics
Measuring the New Measures
Penn Library Franklin (OPAC) Activity Statistics
Measuring the New Measures
Image Collection: Use Analysis
Measuring the New Measures
Ebook Use
Measuring the New Measures
OUP/CUP Ebook Sessions
Sample Months 2001-2002
1800
535 Titles
Penn Site Sessions
533 Titles
Public Site Sessions
1600
452 Titles
Tripod Sessions
Number of Sessions with Individual Ebooks
Total Sessions
1400
418 Titles
The labels indicate the number of individual
titles accessed. While only three books are
available on the public site, authorized
visitors can and do access the complete
258 Titles
1200
1000
800
600
167
Titles
400
200
0
January 01
May 01
November 01
January 02
March 02
April 02
Measuring the New Measures
Increase in OUP/CUP Ebook Activity Per Session,
Sample Months 2001-2002
10.5
10.0
Files Views & Saves Per Session
9.5
9.0
8.5
8.0
7.5
7.0
6.5
6.0
January 01
May 01
November 01
January 02
March 02
April 02
Measuring the New Measures
Managing the Collection of Public Web Sites
Measuring the New Measures
Information Base, Data, & E-Journal Usage Statistics
Measuring the New Measures
Proxy Service Statistics
Measuring the New Measures
Dynamic Report Generation
Measuring the New Measures
Use of Licensed Resources
What Are the High Use E-Journals, Data for FY2001
Title
Log-ins
Pct Total
Log-ins
Log-ins
On Campus Off Campus
Science
4,232
1.5%
3,114
1,057
Nature
4,081
1.4%
2,880
1,173
Journal of Biological Chemistry
2,408
0.8%
1,883
519
Journal of the American Chemical Society
2,405
0.8%
2,153
247
New England Journal of Medicine
1,994
0.7%
1,359
620
Angewandte Chemie (international edition)
1,836
0.6%
1,665
167
Journal of Organic Chemistry
1,660
0.6%
1,504
150
Proceedings of the National Academy of Sciences
1,608
0.6%
1,246
360
Tetrahedron Letters
1,361
0.5%
1,218
143
Organic Letters
1,308
0.5%
1,208
99
Proceedings of the National Academy of Sciences, U.S.
1,285
0.5%
1,017
266
Journal of Molecular Biology
1,060
0.4%
850
210
JAMA: The Journal of the American Medical Association
1,023
0.4%
650
352
Journal of Chemical Physics
992
0.3%
819
172
Journal of Finance
887
0.3%
423
378
Lancet
867
0.3%
637
227
American Journal of Sociology
860
0.3%
384
373
Medicine
849
0.3%
580
263
Applied Physics Letters
834
0.3%
751
83
Physical Review B
826
0.3%
727
98
Measuring the New Measures
Use of Licensed Resources
How Much Bang Do We Get on the Dollar For E-Journals?
E-Journal Subscription Costs Per Log-In, FY2002 (July-April)
Publisher
Log-ins
Pct of Total
Cost Per Login
ScienceDirect
139,727
27.1%
$0.63
ECO
70,730
13.7%
$0.09
JSTOR
48,668
9.4%
$0.35
Wiley
38,255
7.4%
$0.09
ACS
31,865
6.2%
$0.12
Ideal
30,568
5.9%
$5.51
Blackwell/Munksgaard
28,940
5.6%
$0.27
Journals@Ovid
26,982
5.2%
n/a
Oxford
14,819
2.9%
$0.20
SpringerLINK
13,507
2.6%
n/a
ABI/Inform
12,785
2.5%
$3.08
Project Muse
11,438
2.2%
$1.22
AIP
7,873
1.5%
$5.01
Cambridge
7,835
1.5%
n/a
Annual Reviews
7,215
1.4%
$0.08
IEEE
7,132
1.4%
$6.73
RSC
5,661
1.1%
n/a
Others†
11,451
2.2%
515,451
100%
Total
† 11 publishers
Measuring the New Measures
Use of Licensed Resources
What Databases Do Our Clients Use at What Cost?
15 Most Frequently Used Index/Abstract/Full-text Databases in FY 2001
Database
MEDLINE
Log-ins
Pct Total
Cost Per Login
205,150
22.9%
$
0.10
LEXIS/NEXIS
63,817
7.1%
$
0.42
Academic Index
52,407
5.9%
$
0.58
Dow Jones
39,828
4.5%
$
0.68
ISI Citation Indexes
39,753
4.4%
$
2.75
ABI/Inform
36,190
4.0%
$
1.09
PsycINFO
Investext
27,636
17,695
3.1%
2.0%
$
$
0.89
0.68
Business & Industry
16,797
1.9%
$
0.55
CINAHL/Nursing
16,232
1.8%
$
0.36
PubMed
15,610
1.7%
$
-
MLA International
13,359
1.5%
$
0.41
Multex
12,196
1.4%
$
0.10
ERIC
10,852
1.2%
$
0.54
EconLit
8,940
1.0%
$
0.80
Hoovers Online
8,905
1.0%
$
0.22
Inter Bibliog Soc Science
8,152
0.9%
$
0.38
Sociological Abstracts
7,703
0.9%
$
1.58
S&P Industry Surveys
7,346
0.8%
$
0.63
D&B Million $ Database
6,376
0.7%
$
1.74
894,416
100.0%
All others
Measuring the New Measures
Use of Licensed Resources
How Does Use Scatter Across Databases
Use Measured in Log-ins for FY 2001
100%
90%
80%
PCT of Use
70%
60%
50%
40%
30%
20%
10%
0%
0%
10%
20%
30%
40%
50%
PCT of Titles
60%
70%
80%
90%
100%
Measuring the New Measures
Use of Licensed Resources
ScienceDirect Articles Viewed, FY 2001 [vendor-supplied data]
100%
90%
80%
PCT of Titles
70%
60%
50%
40%
30%
20%
10%
0%
0%
10%
20%
30%
40%
50%
60%
PCT of Science Direct Titles
70%
80%
90%
100%
Measuring the New Measures
Use of Licensed Resources
Academic Press (Ideal) Articles Viewed, FY 2001 [vendor supplied data]
NERL Pct used
100%
Penn Use
90%
80%
PCT of Use (full text views)
70%
60%
50%
40%
30%
20%
10%
0%
0%
10%
20%
30%
40%
50%
PCT of Ideal Titles
60%
70%
80%
90%
100%
Measuring the New Measures
Use of Licensed Resources
Where Do Our Clients Access Information?
Database Log-ins by Domain, FY2001
Campus Residences
10%
Off-Campus
15%
In-Library
25%
On-Campus Depts
50%
Measuring the New Measures
Database Use by Penn’s
Schools & Centers
Use of Licensed Resources
School
How Does Database Use Distribute By Communities?
Per Capita Use of Databases by Penn’s Schools and Centers, FY 2001
Medicine
23.2%
Arts & Sci
20.4%
In-Library
12.8%
Wharton
12.3%
55
Admin
4.8%
50
Enginrng
4.4%
Nursing
3.5%
Veterinary
2.5%
40
Education
1.4%
35
Social Wrk
1.8%
Commnctn
1.3%
Fine Arts
1.0%
25
Dental
0.9%
20
Law
0.5%
Dorms
9.3%
45
Log-ins Per Capita
Pct of Log-ins
30
15
10
5
0
School and Center Domains
†Does not include resources licensed by the Law Library for Law school affiliates
Measuring the New Measures
Use of Licensed Resources
Where Do Communities of Clients Work?
Database Log-ins from Off Campus as a Percent of Total Log-ins, FY2001
100.0%
90.0%
Pct. of Log-ins
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
School or Center
On Campus
Off-Campus
Measuring the New Measures
Use of Licensed Resources
When Are They Working?
Database Use by Time of Day, FY2001
60,000
50,000
40,000
30,000
20,000
10,000
0
Schools
Library
Student Residences
Dialin
Other
Measuring the New Measures
Use of Licensed Resources
How Does Audience Composition Change Through the Day?
Database Use by hour, FY2001
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Schools
Library
Student Residences
Dialin
Other
Measuring the New Measures
Intelligence for Bibliographers
Return-Path: <olson@pobox.upenn.edu>
Subject: Again, testing general databases
To: sblack@asc.upenn.edu
Date: Wed, 10 Apr 2002 16:54:11 -0400 (EDT)
From: olson@pobox.upenn.edu
Dear Sharon -Just a second quick note begging you, please, keep trying to look
at those three databases!
Data farm usage logs indicate that one-quarter of all
database logins from Annenberg IP addresses in 2001 were
pointing to Academic Index (followed by Lexis-Nexis and
PsycInfo, both with about 10-percent of all Annenberg
database logins).
Also, 15-percent of all Academic Index school-based logins last
year came from Annenberg IP addresses, more than from all schools
except Arts and Sciences (at 30-percent).
Considering how much Annenberg people use the general database -and you must know best how they can raise Holy Ned over the least
change, I hope
Measuring the New Measures
Impact of Behavior on Facilities Planning
vetlib1.vet
vetlib1.vet
vetlib1.vet
vetlib1.vet
vet119.lib
vet119.lib
vet119.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet118.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
vet115.lib
11:20:46
CAB Abstracts
17:14:22
ISI Citation Indexes
17:17:32
MEDLINE
9:19:50
CAB Abstracts
18:08:49
OVID Multi-Database Searching
20:01:37
OVID Multi-Database Searching
17:58:17
OVID Multi-Database Searching
12:50:03
jake
14:13:43
BIOSIS Preview s
14:38:48
Bryologist
14:41:30
Invitro Animal Cellular & Developmental Biology
14:43:24
Condor
14:52:22
Evolution
14:53:29
Journal of Eukaryotic Microbiology
14:54:27
Journal of Mammalogy
11:51:39
OVID Multi-Database Searching
14:57:26
Journal of Medical Entomology
16:28:24
WorldCat
16:32:44
RLIN
19:02:16
OVID Multi-Database Searching
17:08:02
Journal of Biological Chemistry
14:44:11
CAB Abstracts
14:21:51
Journals@OVID Full Text
14:04:40
MEDLINE
15:07:49
CAB Abstracts
15:08:33
MEDLINE
18:46:13
MEDLINE
11:06:02
Journal of Veterinary Medical Education
11:06:23
Veterinary Research
11:08:03
Vet On-line: the International Journal of Veterinary Medicine
11:09:54
MEDLINE
15:06:20
CAB Abstracts
10:52:24
CAB Abstracts
14:57:19
CAB Abstracts
Measuring the New Measures
Studying Technology Transfer
High impact chemistry journals: regression of online use on in-library use
160
Journal of the American Chemical Society
Journal of Organic Chemistry
Tetrahedron Letters
140
R2 = 0.2211
Reshelves
120
100
80
60
40
20
0
0
50
100
150
Log-ins
200
250
300
Measuring the New Measures
Can Librarians Count?
Measuring the New Measures
Plus Ga Change, Plus C’est La Meme Chose
Measuring the New Measures
New Measures…Old Ways?
 The Metrics are Faulty
 The Assessment Rationale is Vague
 The Data are Inexact
 We Need a Formal Study Process to Address the Problem
 Management’s Data Need is Imperative (and the managers are impatient)
 The Result is Entropy
Measuring the New Measures
Measuring the New Measures: A High Altitude View From the Field
 Impetus for Culture Change (help librarians to embrace counting)
 Creation and Testing of Models/Methods (e.g. LibQual)
 Channels for Learning and Collaboration
 Further Cultural Change
Measuring the New Measures
Culture, Counting and Plumbing
One Library’s Measure of the “New Measures”
Joe Zucca
University of Pennsylvania Library
zucca@pobox.upenn.edu
Penn Library Data Farm:
http://metrics.library.upenn.edu/prototype/datafarm