RBMS Metrics and Assessment Task Force
Subgroup 4: Use & Users, Web Stats, Circulation, Gate Counts, and Community
Impact
Report Section Draft—Christian Dupont (01/21/13)
INTERLIBRARY LOAN SERVICES
Fulfillment of requests forwarded from ILL/document delivery services for copies or
physical loans, fill rates of such requests, etc.
Measuring Operational Activities, Efficiencies, Etc.
1. Describe the activity or process to be measured.
Special collections and archives at academic libraries have opportunities to collaborate
with the interlibrary loan and document delivery offices in their libraries to provide
greater access to their collections through the physical lending or digitization-on-demand
of materials requested by researchers at other institutions.
In January 2012, ACRL approved an updated set of best-practice guidelines to help
libraries develop appropriate policies and procedures to manage the lending of special
collections materials: ACRL/RBMS Guidelines For Interlibrary And Exhibition Loan Of
Special Collections Materials.1
OCLC Research has also published a series of reports that offer guidance on developing
and implementing processes for managing digitization-on-demand services for special
collections materials, including Scan and Deliver: Managing User-initiated Digitization
in Special Collections and Archives.2 The “Scan and Deliver” report was published as
part of a larger “Streamlining Photography and Scanning” OCLC Research initiative that
also resulted in other content, including a webinar.3
The specific activities that can be tracked in this area include:
1
See: http://www.ala.org/acrl/standards/specialcollections
See: http://www.oclc.org/research/publications/library/2011/2011-05r.html
3
See: http://www.oclc.org/research/activities/photoscan.html
2
RBMS Metrics Task Force, Subgroup 4


ILL Services / 2 of 8
the lending of physical “returnable” materials from special collections to another
institution for research use via interlibrary loan services
the delivery of digital images/files, photocopies, or other reproductions of special
collections materials to another institution for research use via interlibrary
loan/document delivery services
Depending on the institution and its local policies, the specific functions that special
collections and interlibrary loan/document delivery staff respectively perform in fulfilling
such requests may vary (for example, at some institutions ILL staff will scan the special
collections materials while at others special collections staff will do the scanning), but for
the activity as such to be counted in this category, an interlibrary loan or document
delivery outside of special collections must be involved in receiving and fulfilling the
request for the end-user. Thus reproduction orders received and fulfilled by special
collections staff alone are not considered or counted as interlibrary loan or document
delivery requests for special collections materials.
2. Identify existing metrics (i.e., measures or counts), noting any references to
international, national, or community standards, guidelines, or definitions.
The Sharing and Transforming Access to Resources Section (STARS) of the Reference
and User Services Association (RUSA), a division of the American Library Association
(ALA) maintains the Interlibrary Loan Code of the United States.4 Although the Code
does itself define any metrics for assessing ILL standards of performance, it does define
and describe the process that libraries measure. Since ILL services are now automated at
most institutions, ILL software reports generate the statistical data that libraries use to
measure and evaluate their ILL performance.
The four principal metrics that libraries have used in assessing ILL performance are5:




4
Transaction Costs
Fill Rate
Turnaround Time
User Satisfaction
See: http://www.ala.org/rusa/sections/stars/resources
See See also J. B. Hill, “Assessment and Optimization: Using Data to Focus on Continuous Improvement
in Interlibrary Loan and Delivery Services,” 2011 ILLiad International Conference, Virginia Beach,
Virginia, March 2011, slide 10. Retrieved from: http://www.atlassys.com/ILLiadConf/Presentations/AssessmentandOptimization.pdf. Hill adds “Service Trends” as a fifth
domain. Hill’s PowerPoint presentation also includes helpful bibliographies on ILL benchmarking studies
(slide 28) and user satisfaction surveys (slide 41).
5
RBMS Metrics Task Force, Subgroup 4
ILL Services / 3 of 8
The first three—transaction costs, fill rate, and turnaround time—can be measured
quantitatively through statistical reports generated by ILL management software systems.
User satisfaction is generally measured through user surveys and other qualitative
methods.
In 1995, the Association of Research Libraries (ARL) undertook a comprehensive study
of interlibrary loan and document delivery performance in academic libraries. According
to ARL:
The 1995-1997 ARL ILL/DD Performance Measures Study provided 1996
baseline data to enables librarians to identify and understand local performance of
mediated ILL/DD operations and compare the performance of their operations to
other participants' operations. Funded by a generous grant from The Andrew W.
Mellon Foundation, the study tracked borrowing and lending unit costs,
borrowing and lending fill rates, borrowing turnaround times, and borrowing user
satisfaction for 97 research and 22 college libraries. The findings have been
distributed through a publication, in 12 ARL-sponsored workshops, and more
than a dozen other workshops and speeches. The methodology was adapted for
use in studies in Australia and the Nordic countries. Libraries of all types and
sizes have incorporated many of the characteristics of high-performing borrowing
and lending operations identified in the study.6
The 2011-12 ARL Statistics Questionnaire includes two questions (21 and 22) that focus
specifically on interlibrary loan services. According the questionnaire instructions,
libraries are to:
Report the number of requests for material (both returnables and non-returnables)
provided to other libraries and the number of filled requests received from other
libraries or providers. For both of these figures, include originals, photocopies,
and materials sent by fax or other forms of electronic transmission. Include
patron-initiated transactions. Exclude requests for materials locally owned and
available on the shelves or electronically. Do not include transactions between
libraries covered by this questionnaire.7
Note that the questionnaire is asking only for numbers of filled borrowing and lending
transactions, not the total number of requests received or fill rates or turnaround times.
Association of Research Libraries, “Measuring the Performance of Mediated and User-Initiated ILL/DD
Services: A Proposal,” May 17, 2002, p. 1. Available from:
http://www.arl.org/arldocs/stats/aboutstats/may2002/a9_ILLproposal.pdf
7
See the instructions for completing the 2011-12 ARL Statistical Survey, available from:
http://www.arlstatistics.org/About/Mailings/stats_2011-12
6
RBMS Metrics Task Force, Subgroup 4
ILL Services / 4 of 8
Expenditures for interlibrary loan and document delivery services are to be aggregated
with “miscellaneous expenditures” in the category of expenditures for “collection
support” in question 7c, and are thus not reported separately. Being a quantitative
instrument, ARL survey does not include any metrics or questions related to user
satisfaction with interlibrary services.
The ARL statistical survey is based on definitions for library operations derived from the
NISO Z39.7-201X standard for Information Services and Use. According to the Data
Dictionary for the “Metrics & Statistics for Libraries and Information Providers”
comprised by the standard, interlibrary loan and document delivery services are defined
as follows:8
7.6.3 Interlibrary Loan
Loan of a document in its physical form or delivery of a document (or part of it)
in copied form, from one library to another library not under the same
administration. Note: Mediated transmission of documents in electronic form is
counted as electronic document delivery.
See Section 7.6.2, Electronic Delivery.
7.6.3.1 Materials Obtained from Other Libraries
These are library materials, or copies of the materials, received by one library
from another library upon request. The libraries involved in interlibrary loans are
not under the same library administration.
7.6.3.2 Materials Provided to Other Libraries
These are library materials, or copies of the materials, provided by one library
to another upon request. The libraries involved in interlibrary loans are not under
the same library administration.
Source: library
Applicability: international, local, national, state
Library Type: academic, cooperative, network, public, school, special, state
Aggregate: yes
7.6.2 External Document Supply Emetrics
National Information Standards Organization, “NISO Z39.7-201X, Information Services and Use: Metrics
& statistics for libraries and information providers - Data Dictionary,” retrieved from:
http://www.niso.org/dictionary/toc/section7/.
8
RBMS Metrics Task Force, Subgroup 4
ILL Services / 5 of 8
Document or part of it in print or electronic form delivered from outside the
library collection by non-library suppliers (not through interlibrary lending) with
the library being involved in the transaction and/or the payment. Note 1: It is
irrelevant whether a number of individual transactions are paid per view or a
certain number of transactions have been prepaid. Note 2: Please see 7.8.1: Use,
Electronic Collections.
Source: vendor
Applicability: international, local, national, state
Library Type: academic, cooperative, network, public, school, special, state
Aggregate: yes
Although national standards and statistical reporting for interlibrary loan services address
only counts of filled requests, many libraries have libraries have established local
benchmarks and targets for ILL performance that pertain to the four principal
performance metrics cited above. For example, the CIC libraries have conducted a threeyear study of ILL performance across the consortium that addresses fill rates as well as
overall ILL activity.9
3. Comment on:
a. the adequacy and the inadequacy of existing metrics, especially those that
reference international, national, or community standards, guidelines, or
definitions;
The definitions and metrics that the interlibrary loan and resource-sharing communities
have developed for interlibrary loan and document delivery activities and performance
appear to be adequate and commonly employed. Although there are no national standards
for ILL performance, local institutions and consortia have devised metrics and
benchmarks that are meaningful for their purposes and are generally comparable across
institutions and institutional groupings because they are based on similar definitions and
methods of data gathering and analysis.
b. methods commonly used to collect data required by the metric; and
A variety of software applications are available for managing interlibrary loan and
document delivery operations and most libraries have automated their operations using
one of them. The applications are able to produce a range statistics on transactions,
Anne K. Beaubien, Jennifer Kuehn, Barbara Smolow, and Suzanne M. Ward, “Challenges Facing Highvolume Interlibrary Loan Operations: Baseline Data and Trends in the CIC Consortium,” College &
Research Libraries 67 (January 2006): 63-84.
9
RBMS Metrics Task Force, Subgroup 4
ILL Services / 6 of 8
including counts of filled requests as well as data that can be used to determine fill rates,
turnaround times, transaction costs, and inform other types of assessment. If desired, the
applications can report separately lending transactions involving special collections
materials based on collection location codes.
c. the priority that RBMS should place on developing a better metric and
related definitions and guidelines for use.
Since most academic library special collections and archives have not yet engaged in
intentional collaborations with their interlibrary loan and document delivery offices to
systematically increase access to their collections, it seems premature to suggest that
RBMS should place a priority on developing metrics and benchmarks or related
definitions and guidelines to aid libraries in assessing their efforts in this area.
At present, libraries that wish to assess their performance with respect to filling requests
for special collections materials can adapt the metrics and benchmarks commonly used
for measuring ILL performance overall to assess performance for special collections
specifically. Three recent conference presentations provide examples and insights into
how this can be done:
1) Julia Gardner (University of Chicago), presentation for the seminar “Yes, We
Scan!: Innovative Approaches to User-initiated Digitization,” held at the 52
Annual RBMS Precconference, June 21-24, 2011. Gardner also adapted her
presentation for the OCLC Research webinar, “Scan and Deliver: Creative Userinitiated Digitization in Special Collections and Archives,” recorded on
September 22, 2011.10 Gardner’s presentation shows how the Special Collections
Research Center at the University of Chicago worked with the library’s ILL office
to develop a scan-on-demand workflow that enabled them to double the number
of ILL requests filled annually for reproductions of special collections materials.
2) Kristine Shrauger (University of Central Florida), presentation for the panel
session “When Worlds Collide: Interlibrary Loan and Special Collections,” held
at the ACRL 2011 National Conference, April 1, 2011.11 Shrauger’s presentations
show ILL cancellation statistics for materials in special collections drove an
initiative to create a scan-on-demand workflow with special collections to digitize
requested materials from defined collections to fulfill ILL requests and to add
them to a digital repository. The program is also described in Kristine J. Shrauger
and Lee Dotson, “Scan by Numbers: Interlibrary Loan Lending Statistics Shape
10
The Powerpoint presentation, audio recording, and chat transcript from the webinar are available from:
http://www.oclc.org/research/events/webinars.html
11
See:
http://www.academia.edu/1684550/When_Worlds_Collide_Interlibrary_Loan_and_Special_Collections
RBMS Metrics Task Force, Subgroup 4
ILL Services / 7 of 8
Digital Initiative,” Journal of Interlibrary Loan, Document Delivery & Electronic
Reserve 20 (2010):135-48. For the same ACRL panel session, Sandra Stelts and
Barbara Coopey (Penn State University) described procedures they developed to
lend rare published materials from special collections to improve their ILL fill
rates.
3) Anna Dulin and Ellen Makaravage (Wake Forest University), “Preserving and
Sharing: Bridging the Gap Between ILL and Special Collections,” presentation at
the 2012 ILLiad International Conference, Virginia Beach, Virginia, March 23,
2012.12 Dulin and Makaravage described how they collaborated on developing a
streamlined workflow between ILL and special collections to receive digitization
requests for rare materials via a web request form linked to their ILLiad ILL
management software that has enabled them increase their ILL fill rate for special
collections materials.
RBMS should look to these and other such examples and share the “best practices” with
the larger special collections and archives community.
Assessing Value and Impact for Users
When considering the evaluation of a program to increase access to special collections
through interlibrary loan and documentary delivery channels, the assessment of impact
and value can be measured according user satisfaction with the service provided. Was the
service more convenient and productive for the user than an alternative that might mean
having to travel to the owning repository to use the material onsite or foregoing access to
the material altogether?
While there do not seem to be any user satisfaction surveys that are geared specifically to
assessing the impact and value of services that offer access to special collections
materials via interlibrary loan or document delivery, it is likely that user satisfaction
surveys that have been developed to assess user satisfaction with ILL services in general
could be adapted for this more specific purpose. Likewise, quantitative methods for
assessing ILL performance could be adapted to assess the performance of ILL services
that are specifically geared to special collections materials and used to support
comparative strategies for analyzing metrics that have been found to be useful indicators
of end-user satisfaction with ILL services, notably fill rates and turnaround time.
12
Powerpoint presentation available from: http://www.atlas-sys.com/conference/2012-illiad-internationalconference-session-archive/
RBMS Metrics Task Force, Subgroup 4
ILL Services / 8 of 8
The methodologies that have been commonly used in the interlibrary loan and resourcesharing communities relating to user satisfaction—which is generally regarded as an
indicator of value and impact—appear to be adequate and feasible for most institutions to
measure as they can be derived from the analysis of fill rates and turnaround times
reported by ILL management software or from user surveys that can be easily
administered. These same methodologies can be adapted for assessing ILL services
involving special collections materials.
To the extent that interlibrary loan and resource-sharing communities have made strong
use of key metrics like fill rates and turnaround times to increase efficiencies and increase
user satisfaction, it seems that ILL services involving special collections materials can
take advantage of these same approaches to evaluate and improve such services and
ultimately increase user satisfaction with special collections and archives.