MINES for Libraries
ACRL: Outcome Assessment Tools for the
Library of the Future: MINES at OCUL
Toni Olshen
York University
Association of College and Research Libraries
ACRL Conference 2005
Minneapolis
April 7, 2005
www.arl.org/stats/
Ontario Council of University Libraries
• OCUL is a consortium of twenty university
libraries in the province of Ontario
• The member libraries cooperate to enhance
information services through: resource
sharing, collective purchasing, document
delivery and many other similar activities.
www.minesforlibraries.org
OCUL Members – Those in Green are ARL Libraries
Brock University
Queen's University
Carleton University
Royal Military College of
Canada
University of Guelph
Ryerson Polytechnic University
Lakehead University
University of Toronto
Laurentian University
Trent University
McMaster University
University of Waterloo
Nipissing University
University of Western Ontario
Ontario College of Art & Design
Wilfrid Laurier University
University of Ontario Institute of
Technology
University of Windsor
University of Ottawa
York University
www.minesforlibraries.org
Member Institution Enrolments (2003)- Total
Undergraduate and graduate students
•
•
•
•
•
•
•
•
•
University of Ontario Institute of
Technology
936
Royal Military College of
Canada
1,941
Ontario College of Art and
Design
3,062
Nipissing University 5,478
Lakehead University 7,304
Trent University
7,388
Laurentian University 8,751
Wilfrid Laurier University
12,426
Brock University
15,527
•
•
•
•
•
•
•
•
•
•
•
University of Windsor 16,266
University of Guelph 19,096
Queen's University
20,034
McMaster University 22,064
Carleton University
22,535
University of Waterloo 25,029
Ryerson University
27,221
University of Ottawa 30,948
University of Western Ontario
32,784
York University
46,794
University of Toronto 68,290
•
•
•
Total 391,933
90% undergrads 10% graduate
12,500 faculty
www.minesforlibraries.org
Scholars Portal – What is it?
• A unique set of shared information resources
and services
• Resources acquired and managed through
OCUL with funding support from a 5-year
grant from the Ontario Innovation Trust (OII),
a provincial funding body
• Resources are made available to researchers
and students in Ontario through their own
university libraries
www.minesforlibraries.org
Scholars Portal
Ontario Council of University Libraries (OCUL)
• Ontario Information Infrastructure (OII)
funded by the Ontario Innovation Trust
in 2001 for five years
• Consortia-purchased electronic
resources offered through the Ontario
Scholars Portal
• March 2004, we began the evaluation
phase of $7.6 million dollar OII project
www.minesforlibraries.org
Scholarly Information Resources
• As of the end of March, contains 7,547,904 full text
articles from 6,783 full text journals published by 12
academic publishers
• Coverage of most disciplines but concentration in
sciences
• Current and historic coverage
• One of the largest collections of electronic
journals available to researchers anywhere
www.minesforlibraries.org
Scholars Portal Resources
•
•
•
•
•
•
•
•
•
•
•
•
Academic Press,
American Psychological Association,
American Chemical Society,
Berkeley Electronic Press,
Cambridge University Press,
Emerald Publishing,
Elsevier Science (Elsevier Science, Harcourt Health Sciences),
Kluwer (Kluwer Academic Publishers, Kluwer Law International and
Kluwer/Plenum),
Oxford University Press,
Project MUSE,
Springer-Verlag, and
John Wiley & Sons.
www.minesforlibraries.org
Scholars Portal – Project Goals
• Centrally mount and deliver information
resources acquired through OCUL
consortia purchases to ensure rapid
and reliable access
• Provide for the long term, secure
archiving of resources to ensure
continued availability
www.minesforlibraries.org
Scholars Portal – Project Goals
• Ensure that the resources and services
provided meet the needs of faculty,
students and staff.
• Ensure that resources and services can
be seamlessly integrated to the local
library and information systems
www.minesforlibraries.org
Measuring Success
• OCUL provides a sophisticated statistical
report mechanism. (see next slide).
Download statistics are a rough measure of
value but we need more to properly assess
impacts.
• Need to measure also the significance for
research of access to e-journals
• Employing ARL MINES Survey methodology
to capture information on how resources are
being used (from where, by whom, and for
what purposes)
www.minesforlibraries.org
SP Statistics and Report Generator
www.minesforlibraries.org
Why Evaluation?
• Feedback to OII and University funders
• Understand who, where, and why the digital
resources are used
• Supplement usage numbers to answer the
key question:
What is the impact of Portal content on
research at Ontario academic libraries?
www.minesforlibraries.org
Evaluating Success
• Evaluating Scholars Portal from user and
staff points of view
• Use a mix of quantitative and qualitative
tools for a richer assessment: MINES, focus
groups, staff survey
• Are OII projects improving research
services?
• Does Scholars Portal meet OCUL user and
staff expectations?
www.minesforlibraries.org
MINES (Measuring the Impact of Networked Electronic Services)
• MINES survey is one of a new breed of
assessment tools that did not exist
before because services were not
digital.
www.minesforlibraries.org
MINES (Measuring the Impact of Networked Electronic Services)-Desired
Outcomes
• To capture in-library and remote web
usage of the Scholars Portal in a sound
representative sample using MINES
methodology;
• To identify the demographic differences
between in-house library users as
compared to remote users by status of
user;
www.minesforlibraries.org
MINES (Measuring the Impact of Networked Electronic Services)-Desired
Outcomes
• To identify users’ purposes for accessing Scholars Portal
electronic services (funded research, non-funded research,
instruction/education use, student research papers and course
work);
• To assist with the evaluation of the project as well as to capture
information for OCUL about indirect research costs; and
• To develop an infrastructure to make studies of patron usage of
networked electronic resources routine, robust and integrated
into the decision-making process.
www.minesforlibraries.org
MINES Methodology
•
•
•
•
•
What user groups use SP?
What specific resources are used?
From where?
How do users learn about SP?
Are there differences in the use of digital
resources based on the user's location?
• Why use SP? (sponsored research?
Instruction? patient care?)
• Does use differ by discipline? user group?
location?
www.minesforlibraries.org
MINES Methodology
• Web-based surveys conducted over the
course of a year for each institution
• Activated during randomly selected 2-hour
survey periods each month as users access
one of SP’s journals
• Mandatory, short, and anonymous
www.minesforlibraries.org
ARL/MINES – Jan. ’04-Dec. ‘05
• ARL developed random schedule of two-hour sessions per
month
• OCUL designed local questions, mounted survey, collects and
sends data to ARL
• ARL compiles survey results for all sites
• ARL reports findings on a semi-annual basis
• ARL presents findings and final report to project participants
on an aggregated and individual institution basis
www.minesforlibraries.org
Development of survey form
• Finding balance between simplicity,
ease and richness of data elements
• Bilingual – University of Ottawa,
Laurentian University, Glendon College
at York University
• Ultimately a change in focus to the
creation of a unique data set
www.minesforlibraries.org
MINES Survey Form – Five Questions and a
Comment Box
www.minesforlibraries.org
Survey Form
• Survey form determined :
– users’ status
– Discipline (affiliation)
– location or where accessed from
– purpose of use (sponsored research,
instruction, patient care, course work)
– how the resource was identified
(bibliography, colleague, librarian,
important journal in field etc.)
www.minesforlibraries.org
OCUL Definition of Usage for MINES
• A successful search connecting the user to
an article of interest for viewing, printing or
downloading
• Unique to Scholars Portal because of
consortia server setup and archiving of all
journals
www.minesforlibraries.org
MINES Methodology
• Random sampling plan and the mandatory
nature of the questions are both required to
create a statistically sound study
• If the survey is not mandatory, the group of
non-respondents is likely to be different from
the group of respondents, and we will not
know what that difference is
• One of the strengths and innovations of this
survey technique is that it is based upon
actual use, not on predicted, intended, or
remembered use
www.minesforlibraries.org
OCUL Implementation of MINES
• Once the survey is completed, the
respondent's browser is forwarded to
the desired networked electronic
resource
• If more than one search is carried out,
the survey form is auto-populated with
user’s responses as defaults which
only have to change if response is
different
www.minesforlibraries.org
Informed Consent
• Because this is a Web-based survey,
the respondents consent to participate
by electing to fill out the survey
questionnaire
• It is the participating library’s
responsibility to provide an explanation
of the survey and information
pertaining to its confidentiality
www.minesforlibraries.org
Confidentiality of Data
• Institutional data are confidential. Individual
institutions and/or their specific data will not
be identified.
• Individual data are anonymous. The
respondent’s privacy is protected because
only very indirect information is captured,
which would be difficult to trace back to an
individual.
www.minesforlibraries.org
Ethics Review
• A major step was contacting research ethics officers
and/or Ethics Review Boards to get approval, where
necessary, to run the survey
• Purpose of ethics reviews for human subjects is to
prevent putting subjects at risk
• Officers/Boards on 16 OCUL campuses accepted
that no physical or psychological harm would come
to library users who are asked to fill out a brief
mandatory anonymous survey before they are
connected to the title of their choice.
www.minesforlibraries.org
Ethics Review
• Reference to interesting opinion piece
by J. Paul Grayson. “How Ethics
Committees are Killing Survey
Research on Canadian Students”.
University Affairs, January 2004.
http://www.universityaffairs.ca/issues/200
4/jan/print/opinion.html
www.minesforlibraries.org
Mandatory Survey
• If individuals chose to avoid filling out
the brief anonymous survey, they might
be inconvenienced for a maximum of a
two-hour period, but they would not be
harmed
• We needed to balance good data for
making decisions and the
inconvenience caused to the user.
www.minesforlibraries.org
Ethics Review – Issues and Problems
• Mandatory nature of the survey required discussion
on some campuses
• Several campuses did not require approval because
the survey fell into quality assurance guidelines and
was seen as a library management tool (8)
• Several schools received approval after an
application process (8)
• One Library and Review Board did not support the
mandatory nature of the methodology so that school
dropped out of the project.
www.minesforlibraries.org
Pre-testing and False start – January –
March 2004
• ARL prepared a schedule for the random two-hour
monthly runs.
• A test run was planned at York and Wilfrid Laurier in
January with the real survey commencing at the end
of February.
• The pilot in January failed at York and highlighted
the need for all institutions to be using a link
resolver URL when connecting to SP journals from
their catalogues or eResources databases.
• Each site reviewed their configuration and
necessary changes were made.
www.minesforlibraries.org
Pre-testing and False start – January –
March 2004
• Survey form and the explanatory material were
translated into French for bilingual Ottawa,
Laurentian, and Glendon College at York.
• February run highlighted concerns about the data
collection.
• The technical infrastructure was capturing only
access through library catalogues or eresource
databases, but not from the use of the SP directly.
• There were some technical problems with the
February and March runs and the validity of the data
was under question. The data-collection
programming was revisited.
www.minesforlibraries.org
Lessons Learned
• Early runs taught us a great deal about
the different ways OCUL libraries
access the SP
• We needed to reflect that in the data
gathering
www.minesforlibraries.org
Lessons Learned
• As originally planned, we now capture as
much usage as possible that comes from :
– local eresource databases
– library catalogues
– Scholars Portal browse and search functions.
www.minesforlibraries.org
New Definition of Usage for MINES
• A successful search is now defined as
connecting the user to an article of interest
for viewing, downloading or printing
• Definition is unique to Scholars Portal
because of consortial server setup and
archiving of content
• We cancelled the April 20 run and reset the
dates of the survey from May 2004 through
April 2005, considering the February and
March runs as tests.
www.minesforlibraries.org
New Definition of Usage for MINES Innovation
• We continue to build on the unique opportunity we
have to gather useful data that is not open to other
types of library groups. By the end of March about
22,500 surveys have been completed. One more
month to go!
• By implementing the MINES survey, OCUL is ahead
of other projects in that we are not held "hostage" to
the limitations and inconsistencies of vendor
statistics
• We have opportunities to disseminate research on
measurement of networked resources through
conferences and publications
www.minesforlibraries.org
MINES Very Preliminary Output: MAY –AUGUST 2004
5223 respondents
www.minesforlibraries.org
Very Preliminary Findings – 4 months of data -Subject
Affiliation
•
•
•
•
•
•
•
•
•
•
•
Applied Sciences
Business
Education
Environmental Studies
Fine Arts
Humanities
Law
Medical Health
Sciences
Social Sciences
Other
•
•
•
•
•
•
•
•
•
•
•
804
146
176
160
22
93
21
1341
1031
673
129
www.minesforlibraries.org
17.5%
3.2
3.8
3.5
.5
2.0
.5
29.2
22.4
14.6
2.8
Very Preliminary Findings – 4 months of data
User Status
• Faculty
• 764
16.6%
• Graduate/Professional
• 2068
45.0
• Undergraduate
• 1039
22.6
• Library Staff
•
47
1.0
• Staff
•
427
9.3
• Other
•
251
5.5
www.minesforlibraries.org
Very Preliminary Findings –
4 months of data - Location
• Library
• 578
12.6%
• Off-Campus
• 1978
43.6
• On-Campus ( but
not in the library)
• 2040
44.4
www.minesforlibraries.org
Very Preliminary Findings4 months of data - Purpose of Use
• Sponsored research
• Other nonsponsored research
• Teaching
• Course work
• Patient care
• Other activities
• 2189
47.6%
•
•
•
•
•
20.0
6.0
14.9
3.1
8.3
919
278
686
143
381
www.minesforlibraries.org
Cross Tabulations
• Purpose of use by affiliation, user status, location,
why
• Location by affiliation, user status, purpose of use,
why
• Why by affiliation, user status, location, purpose of
use
• Which titles used by which users for which purposes
www.minesforlibraries.org
Location and Purpose of Use
www.minesforlibraries.org
Additional Qualitative Data
• MINES Survey respondent comments
• Staff Survey: What does the range of
institutional experiences reveal?
• Focus Groups: What anecdotal data can
faculty and students add to the development
of the Scholars Portal?
www.minesforlibraries.org
Thank you for your attention!
• Questions?
• Toni Olshen: tolshen@yorku.ca
www.minesforlibraries.org