Demonstrating Value and Creating Value:
Evidence-Based Library Management
through
MINES for Libraries™
Martha Kyrillidou
Director of the ARL Statistics and Measurement Program
Association of Research Libraries
Brinley Franklin
Vice Provost for University Libraries
University of Connecticut
Scholars Portal Forum, Ontario Council of University Libraries
February 1 2006
Toronto
http://www.arl.org/stats
www.arl.org/stats/
http://www.minesforlibraries.org
Today
is
Tomorrow
- Peter Melinchok expressing dismay at his
parents’ distorted sense of time
http://www.arl.org/stats
http://www.minesforlibraries.org
• Bangor University considers removing librarians posted by
Blake on Thursday January 27, @07:30AM -753 hits
Ms Information writes "News from the University of Wales Bangor in
the UK. senior management no longer feel that subject librarians /
academic liaison librarians are needed in the modern academic
library. They have made restructuring proposals which include
removing all bar one of the subject librarians and a tier of the library
management, including the Head of Bibliographic Services. The
university management thinks that technology has 'deskilled'
literature searching. As far as I know, this proposal is unprecedented
in the United Kingdom.
In essence, there will remain 4 professional librarians serving a
'research-led' university of 8,000 plus FTEs and with 8 library sites.
These will be the university librarian, cataloguing librarian,
acquisitions librarian and Law librarian.
Has anything like this happened anywhere that you know of? If so,
what have been the effects?
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
Rhodes University – Anne Moon
http://www.arl.org/stats
http://www.minesforlibraries.org
Circulation
TotalTotal
Circulation
600,000
550,000
500,000
450,000
19
91
19
92
19
93
19
94
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
400,000
Note. M. Kyrillidou and M. Young. (2003).
ARL
Statistics 2002-03. Washington, D.C.: ARL, p.8.
http://www.arl.org/stats
http://www.minesforlibraries.org
Reference Transactions
Reference Transactions
170,000
160,000
150,000
140,000
130,000
120,000
110,000
100,000
19
91
19
92
19
93
19
94
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
90,000
Note. M. Kyrillidou and M. Young. (2003).
http://www.arl.org/stats
ARL
Statistics 2002-03. Washington, D.C.: ARL, p.8.
http://www.minesforlibraries.org
ARL Overall
http://www.arl.org/stats
http://www.minesforlibraries.org
Libraries Remain a Credible Resource in
21st Century
98% agree with statement, “My … library
contains information from credible and
known sources.”
Note. Digital Library Federation and Council on Library and Information
Resources. (2002). Dimensions and Use of the Scholarly Information Environment.
http://www.arl.org/stats
http://www.minesforlibraries.org
Changing Behaviors
Recent Survey:
Only 15.7% agreed with the statement “The
Internet has not changed the way I use the
library.”
Note. Digital Library Federation and Council on Library and Information
Resources. (2002). Dimensions and Use of the Scholarly Information Environment.
http://www.arl.org/stats
http://www.minesforlibraries.org
The Internet Goes to College
Early data from ethnographic interviews
– “I use Google because I heard it searches for more
–
–
–
–
–
–
things” (than other sources).
“I believe I can find anything on the Internet. There
hasn’t been anything I haven’t been able to find.”
“Because I’m lazy.”
Books have “so much information that no one can go
through it all.”
I use “the Internet first is because it is more convenient.”
I go to the library “because that’s what teachers like.”
“Google has gotten me through college.”
Source: Steve Jones, The Internet Goes to College, ARL Talk
http://www.arl.org/stats
http://www.minesforlibraries.org
… a revolution in making
Il est plus nécessaire d'étudier les hommes
que les livres
—FRANÇOIS DE LA ROCHEFOUCAULD (1613–1680)
http://www.arl.org/stats
http://www.minesforlibraries.org
Into the future …
• StatsQUAL+™
– ARL Statistics
– E-Metrics
– LibQUAL+™
– DigiQUAL+™
– MINES for Libraries™
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES for Libraries
The MINES Survey Methodology
Brinley Franklin
Vice Provost for University Libraries
University of Connecticut
brinley.franklin@uconn.edu
http://www.minesforlibraries.org
www.arl.org/stats/
What is MINES?
• Action research
– Set of recommendations for research design
– Set of recommendations for web survey
presentation
– Set of recommendations for information
architecture in libraries
– Plan for continual assessment of networked
electronic resources
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES for Libraries
TM
• MINES is a transaction-based research methodology consisting
of a web-based survey form and a random moments sampling
plan.
• MINES typically measures who is using electronic resources,
where users are located at the time of use, and their purpose of
use.
• MINES was adopted by the Association of Research Libraries
(ARL) as part of the “New Measures” toolkit in May, 2003.
• MINES is different from other electronic resource usage
measures that quantify total usage (e.g., Project COUNTER, EMetrics) or measure how well a library makes electronic
resources accessible (LibQual+TM).
http://www.arl.org/stats
http://www.minesforlibraries.org
TM
MINES for Libraries Survey Form
Five Questions and a Comment Box
http://www.arl.org/stats
http://www.minesforlibraries.org
Questions Addressed By MINES for Libraries™ for the
OCUL Scholars Portal
•
How extensively do sponsored researchers use
OCUL’s Scholars Portal? How much usage is
for non-funded research, instruction/education,
student research papers, and course work?
•
Are researchers more likely to use the Scholars
Portal from inside or outside the library? What
about other classifications of users?
•
Are there differences in Scholars Portal based on the user’s location (e.g.,
in the library; on-campus, but not in the library; or off-campus)?
•
Could MINES, combined with usage counts, provide an infrastructure to
make Scholars Portal usage studies routine, robust, and easily integrated
into OCUL’s administrative decision-making process for assessing
networked electronic resources?
http://www.arl.org/stats
http://www.minesforlibraries.org
OCUL/MINES Methodological
Considerations
•
The sampling plan was determined at the outset. Surveys were conducted
once a month for two hours a month between May, 2004 and April, 2005
•
The selection of the monthly survey periods were weighted based on usage
counts by time of day and were chosen randomly.
•
Participation was mandatory, negating on-respondent bias, was based on
actual use in real-time, and was brief (to minimize user inconvenience).
•
OCUL designed the local questions, mounted the survey, collected data and
sent it to ARL for tabulation in aggregate and by individual institution.
•
If more than one search was conducted by a user, the survey form was
auto-populated with initial responses as the default.
http://www.arl.org/stats
http://www.minesforlibraries.org
OCUL/MINES Methodological
Considerations (continued)
•
Each participating library explained the survey and its confidentiality
provisions to their local constituency.
•
Research ethics officers and/or Ethics Review Boards, where necessary,
reviewed and approved the survey instrument and methodology.
•
OCUL determined that individual institutions and their institution-specific
data collected during the survey periods would not be disclosed. Individual
data was anonymous.
•
The mandatory nature of the survey required discussion on some campuses
and caused one OCUL member library to withdraw from the study.
•
Two institutions pre-tested the survey in January, 2004. Data collection
programming and configurations/links had to be revised in February and
March, 2004.
•
After completing the survey, users were connected to their desired Scholars
Portal networked electronic resource.
http://www.arl.org/stats
http://www.minesforlibraries.org
Issues with web surveys
• Research design
– Coverage error
• Unequal access to the Internet
• Internet users are different than non-users
– Response rate
• Response representativeness
– Random sampling and inference
– Non-respondents
• Data security
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES strategy
• A representative sampling plan, including sample size, is
determined at the outset. Typically, there are 48 hours of
surveying over 12 months at a medical library and 24
hours a year at a main library.
• Random moment/web-based surveys are employed at
each site.
• Participation is usually mandatory, negating nonrespondent bias, and is based on actual use in real-time.
• Libraries with database-to-web gateways or proxy rewriters offer the most comprehensive networking solution
for surveying all networked services users during survey
periods.
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES strategy (continued)
• Placement
– Point of use
– Not remembered, predicted or critical incident
• Usage rather than user
– What about multiple usages
– Time out ?
– Cookie or other mechanism with auto-population
• Distinguish patron association with libraries.
– For example, medical library v. main library.
– But what if the resources are purchased across
campus for all. Then how to get patron affiliation?
http://www.arl.org/stats
http://www.minesforlibraries.org
Web Survey Design Guidelines
• Web survey design guidelines that MINES followed:
– Presentation
• Simple text for different browsers – no graphics
– Different browsers render web pages differently
•
•
•
•
•
•
Few questions per screen or simply few questions
Easy to navigate
Short and plain
No scrolling
Clear and encouraging error or warning messages
Every question answered in a similar way - consistent
– Radio buttons, drop downs
• ADA compliant
• Introduction page or paragraph
• Easy to read
– Must see definitions of sponsored research.
• Can present questions in response to answers – for example if
sponsored research was chosen, could present another survey
http://www.arl.org/stats
http://www.minesforlibraries.org
Quality Checks
• Target population is the population frame – surveyed the patrons
who were supposed to be surveyed - except in libraries with
outstanding open digital collections.
• Check usage against IP. In this case, big numbers may not be
good. May be seeing the survey too often.
• Alter order of questions and answers, particularly sponsored and
instruction.
• Spot check IP against self-identified location
• Spot check undergraduates choosing sponsored research –
measurement error
• Check self-identified grant information against actual grants
• Content validity – discussed with librarians and pre-tested.
• Turn-aways – number who elected not to fill out the survey
• Library information architecture -- Gateway v. HTML pages – there is
a substantial difference in results.
http://www.arl.org/stats
http://www.minesforlibraries.org
Mandatory – UConn Libraries (3 months)
Mandatory
Sponsored
All Networked Electronic
Services Use
Research
Instruction
Other
Total
In-Library
Off-Campus
21
49
186
138
90
146
297
333
19.04%
On-Regional Campus
35
77
22
134
8.59%
174
498
124
796
51.03%
279
17.88%
899
57.63%
382
24.49%
1560
100.00%
100.00%
On-Main Campus
Total
Total as a Percentage
http://www.arl.org/stats
21.35%
http://www.minesforlibraries.org
Optional – UConn Libraries (3 months)
Optional
Sponsored
All Networked Electronic Services
Use
Research
In-Library
Off-Campus
On-Regional Campus
On-Main Campus
Total
Total as a Percentage
http://www.arl.org/stats
Instruction
Other
Total
19
12
106
148
37
66
162
226
19.13%
8
41
29
78
9.21%
66
262
53
381
44.98%
105
12.40%
557
65.76%
185
21.84%
847
100.00%
100.00%
26.68%
http://www.minesforlibraries.org
Issues with web surveys:
brief bibliography
•
•
•
Cook, Colleen; Heath, Fred; and Russell L. Thompson. 2000 (December). “A Meta-Analysis of
Response Rates in Web- or Internet-Based Surveys.” Educational and Psychological
Measurement 60(6): 821-836.
Couper, Mick P.; Traugott, Michael W.; and Lamias, Mark J. 2001. "Web Survey Design and
Administration," Public Opinion Quarterly, 65 (2): 230-253.
Covey, Denise Troll. . 2002. Usage and Usability Assessment: Library Practices and Concerns.
CLIR Publication 105. Washington DC: Council on Library and Information Resources.
–
•
•
Dillman, D.A. 2000 (December). Mail and Internet Surveys, The Tailored Design Method. 2nd
Ed. New York: John Wiley & Sons.
Gunn, Holly. 2002. “Web-based Surveys: Changing the Survey Process.” FirstMonday 7(12).
–
•
•
•
http://www.libqual.org/documents/admin/ARL_Notebook2004.pdf
Schonlau, Matthias; Fricker Jr., Ronald D.; and Elliott, Marc N. 2002. Conducting Research
Surveys via E-Mail and the Web. Santa Monica, CA: RAND.
Tenopir, Carol, with the assistance of Brenda Hitchcock and Ashley Pillow. 2003 (August). Use
and Users of Electronic Library Resources: An Overview and Analysis of Recent Research
Studies. Washington DC: Council on Library and Information Resources.
–
•
http://firstmonday.org/issues/issue7_12/gunn/index.html
LIBQUAL+ ™ Spring 2004 Survey. 2004. Cook, Colleen, and others.
–
•
http://www.clir.org/pubs/reports/pub105/contents.html
http://www.clir.org/pubs/reports/pub120/contents.htmls
Thomas, Susan J. 2004. Using Web and Paper Questionnaires for Data-Based Decision Making:
From Design to Interpretation of the Results. Thousand Oaks, Corwin Press.
Thompson, Bruce.; Cook, Colleen.; Thompson, Russell L. 2002. Reliability and Structure of
LibQUAL+™ scores: Measuring Perceived Library Service Quality. portal: Libraries and the
Academy.3-12.
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES for Libraries
OCUL MINES
What do the data tells us?
Martha Kyrillidou
Director of the ARL Statistics and Measurement Program
Association of Research Libraries
http://www.minesforlibraries.org
http://www.arl.org/stats
www.arl.org/stats/
http://www.minesforlibraries.org
Questions Addressed By
MINES for Libraries™ for the
OCUL Scholars Portal
•
How extensively do sponsored researchers use
OCUL’s Scholars Portal? How much usage is for non-funded research,
instruction/education, student research papers, and course work?
•
Are researchers more likely to use the Scholars Portal from inside or
outside the library? What about other classifications of users?
•
Are there differences in Scholars Portal based on the user’s location
(e.g., in the library; on-campus, but not in the library; or off-campus)?
•
Could MINES, combined with usage counts, provide an infrastructure to
make Scholars Portal usage studies routine, robust, and easily integrated
into OCUL’s administrative decision-making process for assessing
networked electronic resources?
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES for LibrariesTM Survey Form
Five Questions and a Comment Box
http://www.arl.org/stats
http://www.minesforlibraries.org
Analysis
• Web deliverables:
– Crosstabulations in html for all OCUL data
– Interactive crosstabs for all OCUL and
institutions
• Print deliverables:
– summary tables for OCUL
– summary tables for each institution
– Final report
http://www.arl.org/stats
http://www.minesforlibraries.org
Add them up and down …
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
OCUL Scholars Portal Usage
Affiliation
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
Affiliation by Purpose of Use
Purpose of Use
Coursework
Other
Activities
Other
Research
Applied Sciences
24.0%
7.6%
17.7%
Business
34.8%
7.6%
Education
40.9%
Environmental
Studies
Affiliation
Patient
Care
Sponsored
Teaching
0.6%
46.3%
3.7%
100.0%
30.0%
0.9%
10.8%
16.0%
100.0%
5.4%
17.1%
0.8%
11.8%
24.0%
100.0%
43.5%
2.5%
24.0%
0.3%
23.3%
6.3%
100.0%
Fine Arts
56.3%
6.9%
20.6%
1.3%
5.6%
9.4%
100.0%
Humanities
51.5%
10.8%
21.0%
0.5%
9.5%
6.7%
100.0%
Law
67.5%
6.8%
12.8%
0.9%
2.6%
9.4%
100.0%
Medical Health
29.7%
5.5%
18.4%
8.6%
32.0%
5.7%
100.0%
Other
51.9%
22.8%
10.9%
2.1%
7.4%
5.0%
100.0%
Sciences
44.6%
9.7%
11.1%
0.4%
31.8%
2.4%
100.0%
Social Sciences
62.6%
4.5%
14.4%
0.7%
13.6%
4.2%
100.0%
42.0%
7.5%
16.2%
2.4%
26.2%
5.6%
100.0%
Total
http://www.arl.org/stats
Total
http://www.minesforlibraries.org
User Status by Purpose of Use
Purpose of Use
User Status
Faculty
Coursework
Other
Activities
Other
Research
Patient
Care
Sponsored
Teaching
Total
1.5%
4.7%
21.2%
4.4%
42.6%
25.6%
100.0%
Graduate Professional
19.5%
3.9%
25.5%
2.5%
45.4%
3.2%
100.0%
Library Staff
23.5%
24.1%
13.1%
16.5%
17.7%
5.2%
100.0%
Other
6.0%
35.2%
20.8%
8.7%
26.8%
2.5%
100.0%
Staff
3.5%
9.5%
20.6%
2.1%
51.6%
12.7%
100.0%
75.8%
7.8%
7.7%
0.9%
5.9%
1.9%
100.0%
42.0%
7.5%
16.2%
2.4%
26.2%
5.6%
100.0%
Undergraduate
Total
http://www.arl.org/stats
http://www.minesforlibraries.org
Location by Purpose of Use
Purpose of Use
Location
Coursework
Other
Activities
Other
Research
Patient
Care
Sponsored
Teaching
Total
Library
52.8%
14.9%
10.8%
1.2%
12.3%
7.9%
100.0%
Off-campus
47.2%
7.0%
17.3%
4.1%
19.9%
4.6%
100.0%
On-campus
29.2%
4.0%
17.9%
0.9%
42.2%
5.7%
100.0%
42.0%
7.5%
16.2%
2.4%
26.2%
5.6%
100.0%
Total
http://www.arl.org/stats
http://www.minesforlibraries.org
Reason for Use
Reason for Use (n=20293)
Important Journal
Frequency
10219
Reason for Use (n=20293)
Percent
Important Journal
50.4%
Recommended Colleague
2436
Recommended Colleague
12.0%
Reference/Citation
6090
Reference/Citation
30.0%
Recommended Librarian
620
Recommended Librarian
3.1%
Course Reading
925
Course Reading
4.6%
Other
http://www.arl.org/stats
4388
Other
21.6%
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
How extensively do sponsored researchers use
OCUL’s Scholars Portal? How much usage is
for non-funded research,
instruction/education, student research
papers, and course work?
MINES for Libraries™shows that the Scholars
Portal resources are heavily used by faculty
and students in all OCUL. The majority of
the use is from the sciences and the medical
field and particularly in those fields the
majority of the use is for sponsored research
purposes.
http://www.arl.org/stats
http://www.minesforlibraries.org
Are researchers more likely to use the Scholars
Portal from inside or outside the library?
What about other classifications of users?
Most faculty, graduate professionals and
undergraduates uses of the Scholars Portal
are from outside the library building.
Undergraduates though do show many uses of
the Scholars Portal from within the library as
they are probably becoming more exposed to
these resources by having more physical
contact with the library.
http://www.arl.org/stats
http://www.minesforlibraries.org
Are there differences in Scholars Portal
based on the user’s location (e.g., in the
library; on-campus, but not in the
library; or off-campus)?
Most of the faculty and graduate
professionals use Scholars Portal either
from on-campus locations outside the
library or from other off-campus
locations. Most of the uses from these
locations outside the library are for
sponsored research purposes.
http://www.arl.org/stats
http://www.minesforlibraries.org
Could MINES, combined with usage counts,
provide an infrastructure to make Scholars
Portal usage studies routine, robust, and
easily integrated into OCUL’s
administrative decision-making process for
assessing networked electronic
resources?
Yes
http://www.arl.org/stats
http://www.minesforlibraries.org
Discussion
• How do we allocate expenditures for
electronic resources?
• How do we allocate indirect costs for
electronic resources?
• What is the appropriate balance between
electronic and print?
• What is the appropriate balance between
centralized and distributed purchasing?
• How are electronic resources affecting
learning and research outcomes?
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES for Libraries
OCUL MINES
Possible Future Directions
Brinley Franklin
Vice Provost for University Libraries
University of Connecticut
http://www.minesforlibraries.org
http://www.arl.org/stats
www.arl.org/stats/
http://www.minesforlibraries.org
OCUL MINES
Possible Future Directions
• Individual Institutional Analysis
• Longitudinal Data Collection
• Beyond the Scholar’s Portal
http://www.arl.org/stats
http://www.minesforlibraries.org
OCUL Scholars Portal Users
by Purpose of Use
(n=20,300)
2%
26%
Sponsored Research
6%
8%
Coursework
Other Research
Other Activities
16%
42%
Teaching
Patient Care
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES for LibrariesTM
Location of Electronic Resources Users
Inside the Library
Outside the Library
64%
20%
21%
36%
79%
80%
U.S. Main Libraries
Total Users
U.S. Medical Libraries
Total Users
n = 25,698
n = 31,883
OCUL (Canada)
Libraries
Total Users
n = 20,300
http://www.arl.org/stats
http://www.minesforlibraries.org
MINES for LibrariesTM
Demographics by Location of User
Ontario Council of University Libraries
Graduate Students
Faculty, Staff,
Research Fellows
Undergraduate
Students
All Other Users
3%
14%
5%
2%
14%
31%
29%
45%
48%
69%
16%
24%
Inside the Library
n = 4,047
http://www.arl.org/stats
On Campus, Not in the Library
n = 7,090
Off-Campus
n = 9,163
http://www.minesforlibraries.org
Beyond the Scholar’s Portal
• DD/ILL
– ILLiad – enable at the ILLiad logon screen
• Ask Reference
– Enable at the Ask Reference page or icon
• Digital libraries
– Represent an enormous investment
– Primary clientele is outside the library.
– Introduces non-authenticated group
http://www.arl.org/stats
http://www.minesforlibraries.org
Beyond the Scholar’s Portal
• Online catalog
– 856 field
• Serials solutions
– List of ejournals
• Referrer server
– Create a passthrough gateway
• Mirrored web server
– Drop in mirrored HTML page with survey links
at survey period
• Mirrored HTML pages enabled by scripts
http://www.arl.org/stats
http://www.minesforlibraries.org
http://www.arl.org/stats
http://www.minesforlibraries.org
Beyond the Scholar’s Portal
• Because of the point of use requirement,
libraries that have either a virtual gateway in
library web architecture (or mount their own
files like OCUL or OhioLink) succeed the
best.
• Rewriting proxy server
• Database-to-web solutions
• Serials Solutions
• OpenURL solutions are a gateway.
http://www.arl.org/stats
http://www.minesforlibraries.org
Beyond the Scholar’s Portal
•
•
•
•
•
•
•
•
Networked electronic resources are accessible from many different web
pages and web servers
Patrons bookmark networked electronic resources locally on their own
workstations.
Academic departments, librarian liaisons, anyone with a web page copies
and pastes library links into their own web sites
The survey data must be collected and commensurable for all networked
electronic resources, including e-journals, e-books, online databases or
traditional library request services offered in the online environment, such
as Interlibrary Loan.
The results of the survey have to be uninfluenced by caching issues, both
local, web browser caching and proxy server or Internet Service Provider
caching.
The survey has to be meaningful for networked electronic resources, no
matter how they were implemented.
Different authentication methods have to be accommodated, whether the
institution used IP, password, referring URL, or an authentication and
access gateway.
Remote usage has to be measured, regardless of the channel of
communication, whether locally implemented proxy server, modem pool, or
other institutional service.
http://www.arl.org/stats
http://www.minesforlibraries.org
September 25-27, 2006
Charlottesville, VA, USA
http://www.arl.org/stats
http://www.minesforlibraries.org
OCUL MINES
Possible Future Directions
brinley.franklin@uconn.edu
martha@arl.org
http://www.minesforlibraries.org
http://www.statsqual.org
http://www.arl.org/stats
http://www.minesforlibraries.org