Best Practices for Evaluating a
New Electronic Resource
Sponsored by The Committee on Relations with Information Vendors (CRIV)
Prepared by:
Rob Myers
Associate Director for Collections,
Acquisitions and Planning
The Judge Ben C. Green School of Law
Case School of Law
Case Western Reserve University
Cleveland, Ohio
With assistance from:
Judith Kaul
Electronic Resources &
IP Reference Librarian
The Judge Ben C. Green Law Library
Case School of Law
Case Western Reserve University
Cleveland, Ohio
Presented by:
Brandi Ledferd
Technical Services Librarian
K&L Gates LLP
Seattle, Washington
With:
Shaun Esposito
Head of Public Services
University of Arizona Law Library
Tucson, Arizona
Ed Edmonds
Associate Dean for Library and Information
Technology, Professor of Law
Kresge Law Library
Notre Dame Law School
Notre Dame, Indiana
Best Practices Defined



Best practices is a concept that migrated from the business
community to the academic world as part of knowledge
management.
Organizations gather information about the practices of other
organizational units of comparable size and scope. They compare
the outside organizations with each other and with themselves to
identify the best practices in their areas of interest.
The term "best practices" is often used interchangeably with other
terms such as "effective practices" or "guidelines" or "standards.“
From “Library Best Practice for Print and Online Periodicals.” Montgomery College Libraries,
Montgomery County, Maryland, viewed July 4, 2011.
http://www.montgomerycollege.edu/~nnyland/Periodicalsbestpractices.html
Focus of Presentation is on Evaluating
Online Commercial Databases





Many of the principles also transfer to e-books, CD-ROMs, and other media.
The presentation will not address specifics of license agreements as they are covered
in two other CRIV sponsored programs: G4 “Anatomy of a License Agreement” on
Monday at 2:15 pm and H1 “Getting to Yes for Your Library: Negotiation Contracts in
Your Favor” on Tuesday at 9:00 am and co-sponsored with PLL.
The presentation will not discuss at any length the design and development of an eresource Collection Development Policy as this topic merits an entire program in itself
and tends to be very library specific based on the subject specialties of the
institution.
Discussion of electronic resource management is limited to only those aspects
impacting the selection process. We will not discuss at any length organization and
access to e-resources, Content Management Systems, Electronic Resource
Management Systems, or preservation of e-resources.
While much of the criteria in this program can be applied to e-book collections, ebooks are addressed in the program “E-books and the Future of Legal Publishing” on
Tuesday, from 10:45 to 11:30.
What are the Best Practices for
Evaluating a New Electronic Resource?
1.
Develop an e-resource Collection Development Policy that fits
the needs and goals of your institution and integrates
electronic resources within the array of other collection formats
in your library.
2.
“Try before you Buy” – Always get a trial from the e-publisher
before agreeing to purchase.
3.
Obtain stakeholder involvement before deciding to purchase.
4.
Use an “Electronic Resource Evaluation Checklist” to ensure
you look at all facets of the particular e-resource.
Best Practices Continued
5.
Perform a cost benefit-analysis.
6.
Benchmark- Before, during and after the acquisition is made.
7.
Monitor usage statistics regularly (and especially before
renewing a subscription).
8.
Develop and implement library workflows, policies, and
procedures for evaluating, selecting, and accessing e-resources
that are tailored to the goals and needs of your library and
promote the use of your collection of e-resources.
Life Cycle of an E-Resource
Seven Phases
1. Discovery- Awareness of a new or existing e-resource originates from
an attorney, paralegal, faculty member, student, librarian, or publisher
brochure. Awareness may also arise out of some recognized need and
then researched in a directory of e-resources, other selection tool, or
posed as a question to a listserv to find a fitting e-resource solution.
2. Trial- It is a good practice to always try out an e-resource before
deciding to subscribe or purchase. A trial may include additional
stakeholders other than just librarians. Nearly all publishers now
accommodate trials of their e-resources before deciding to subscribe.
3. Evaluation- The process of gathering and analyzing information gained
from the trial, completing an Electronic Resource Evaluation Checklist,
reading/evaluating the license agreement, and performing a cost
benefit analysis.
Life Cycle of an E-Resource
Seven Phases Continued
4. Selection- Based on results from the trial, the E-Resource Evaluation
Checklist, the cost benefit analysis, and satisfaction with the clauses in
the license agreement, a decision is made whether to subscribe or
purchase the e-resource. (You may be evaluating competing eresources in which case you would select the one which best suits your
needs and budget.)
5. Acquisition- The process of informing the publisher that you would like
to purchase the database. While negotiation of price and clauses in
the license agreement can take place at any point along the life cycle,
it is at this point that final negotiation takes place. A publisher is most
likely to relent somewhat on price and minor changes to the license
agreement when they know that a deal is nearly in hand.
Life Cycle of an E-Resource
Seven Phases Continued
6.
Access- This is the phase at which you inform patrons of the
availability of the new e-resource, provide points of access for
patrons to use it, place information regarding the e-resource in the
library’s ERM system or homegrown equivalent, and passwords if
needed. The points of access you use depend on each library’s
bibliographic access policies. Examples include placing
bibliographic records and links in the library catalog, placement in
an a A-Z list or subject list on the library’s webpage, and/or
placement in a database of library databases. Examples of ways to
inform the patron base of new resources include placing the eresource on the library’s new acquisition list, notification in the
library’s newsletter, broadcast e-mail, or simply by placing “new”
next to the name of the e-resource in the library’s database list.
Life Cycle of an E-Resource
7.

Decision to Renew or Cancel- As the time to renew an e-resource
nears, a decision to renew or cancel the subscription must be
made. Factors influencing the decision include usage, reliability of
the interface (technical issues), receptivity of the patron base,
responsiveness of the vendor in addressing issues, and whether
better or more cost-effective competing e-resources have emerged
during the subscription period
We will examine the first four phases of the life cycle during this
presentation though not nearly in the strict order presented above.
Loosely adapted from Sadeh, Tamar and Mark Ellingsen. “Electronic
Resource Management Systems: The Need and the Realization.”
New Library World 106, no. 5/6 (2005).
Stakeholders in the Selection Process

Librarians – It is essential to have your co-workers onboard!





Reference/Public Services Librarians/Subject Specialists- They must use
the resource for reference, for teaching, and for training purposes.
Electronic Services Librarians- They must answer and handle all resulting
complaints and technical issues.
Catalogers/Bibliographic Access Librarians- They need to address how
the database can be cataloged and presented on the library’s website.
(e.g. top level linking v. title level deep-linking)
Patron base- Attorneys, paralegals, faculty members, students, and
public patrons must like it and have a good enough experience
obtaining relevant results to be repeat users.
IT Staff- IT may or may not need to be involved in the selection
process though it is best to include them up front should something
unforeseen happen such as firewalls blocking access.
Essential E-Resource Discovery Tools

Gale Directory of Databases, 34th ed. Detroit: Gale, Cengage Learning, 2011. (Vols.
1-3 are devoted to online databases.)

Law and Legal Information Directory, 23rd ed. Detroit, Gale Cengage Learning, 2011.
(Contains section on Electronic Resources within chapter on Legal Publishers.)





Ken Svengalis’ Legal Information Buyer’s Guide and Reference Manual , N. Stonington,
CT: Rhode Island LawPress, 2010. (Contains a chapter on computer-assisted legal
research. Note: the 2011 edition has just been released.)
Websites- Both publisher/aggregator websites and other library/consortia websites.
Blog Posts - Reviews of e-resources appear on numerous blog posts (Nina Platt’s
Strategic Librarian, LLRX.com, Three Geeks and a Law Blog, Law Librarian Blog, to
name a few).
Library trade magazines (Choice, Booklist, College and Research Libraries News,
Library Journal, Spectrum, etc.)
E-newsletters and e-zines (e.g. the Scout Report, http://scout.wis.edu and DLibMagazine www.dlib.org.)
The E-Resource Evaluation Checklist

Reasons for using an E-Resource Evaluation
Checklist:
Ensures that you are making the best selection
decision.
 Serves as a guide so that all necessary questions
regarding an e-resource get asked and answered.
 Enables and fosters fiscal responsibility and aids in
budgeting.
 Keeps all the information about an e-resource in
one location (if not using an ERMS).
 Provides documentation that you are doing your
job and fill a need within the institution.

The E-Resource Evaluation Checklist

E-Resource Evaluation Checklists Generally Focus on
Five Areas:






Content and authority
Usability and design
Technical issues, training and support
License agreement issues
Cost and budgetary considerations
Ways in which E-Resource Evaluation Checklists are
maintained:


Electronically (ERMS or other database, PDF form,
Word document, Spreadsheet)
As paper forms
The E-Resource Evaluation Checklist
Specific Criteria to Include









Content Appropriateness- originality, uniqueness, authority
Currency and Archives- also embargos and number of
documents or records
User Interface- ease of use, intuitiveness
Search Capability- Basic, advanced, Boolean, full-text,
sorting search results, etc.
Browsing Capability- index, table of contents, linking to
previous/subsequent sections
Display Formatting- HTML, PDF, etc.
Printing & Downloading – PDF, copy and paste
Clipping and/or e-mail services
Compatibility with Mobile Devices and Mobile Apps
The E-Resource Evaluation Checklist
Specific Criteria to Include










Copyright for ILL and Course Pack Purposes
Access- Passwords, IP Recognition, and VPN
End-User Support- phone support, online support availability
End-User Training- In person, telephone, online, online
tutorials, user guides
Technical Support- hours of availability
User Statistics- availability and type provided
Bill Back Mechanisms- Hourly and/or transactional
Marc Records Availability
Deep Linking- Allows for cataloging at the title level
Cost and Type of Subscription: Annual Subscription,
Perpetual Access, Digital Ownership, etc.
The E-Resource Evaluation Checklist
Specific Criteria to Include


Bench Marking- What others have said about the resource
(other firms, during library association meetings and
lunches, literature reviews, blogs, guides, posts to law-lib
and other lists, etc.)
And Lastly, the License Agreement- Discussed in depth in
other programs but need to recognize that a license
agreement is very influential in the decision-making process
especially where it is overly restrictive (e.g. not allowing for
ILLs or Course Pack copies).
The E-Resource Evaluation Checklist
A sample E-Resource Evaluation Checklist is provided in the
handouts on AALL2GO
Electronic Resource Evaluation Checklist Form.pdf
Selection and Coordination of an EResource Trial Focus Group
Reasons for using a Trial Focus Group:
Obtain stakeholder buy-in for the e-resource.
 Due diligence:




End-users (Attorneys, Students, Faculty members) may
discover something, good or bad, about the resource
that might otherwise have been overlooked.
IT Staff might uncover a technical issue that otherwise
might not have been found. Even if found after the
purchase/subscription has been entered, IT Staff tend to
be more receptive to helping if they were included on the
front end rather than just when help is needed.
It has a marketing effect (albeit limited) of
promoting use of the e-resource.
Selection and Coordination of an EResource Trial Focus Group
Reason for not using a Trial Focus Group
or at least limiting it to just a few select
librarians and close others:

You may find that certain demanding faculty,
students, attorneys, etc. try the database and say
they must have it even though, on balance, the
resource is not very good and not worth the
money.
Selection and Coordination of an EResource Trial Focus Group
It is a good idea to invite at least one member of each of
the stakeholder groups. (They may decline, but at least
they can’t complain they were not informed or invited.):


Librarians (especially reference and bibliographic
access librarians)
End-users (attorneys, faculty members, students,
judges, clerks)



May want to use attorneys on the firm library committee
May want to use students and faculty members on the law
school library committee.
IT Department Staff
Selection and Coordination of an EResource Trial Focus Group
Types of E-Resource Trial Focus Groups:



Limit to just librarians.
Limit to librarians and select IT staff.
Limited to librarians, select IT staff, and select attorneys,
select faculty members, and select students)




An example of this is to use the members of your existing library
committee.
Another example is to create a standing committee made up of
librarians and attorneys/faculty member/students whose purpose is to
trial e-resources.
A final example is a standing committee of librarians with an
interchanging mix of attorneys or faculty members (the liaison model in
many academic libraries.)
Open the survey to entire localized library community via
web and provide link to online survey. (Example: University
of South Carolina Coleman Karesh Law Library)
Selection and Coordination of an EResource Trial Focus Group
Selection and Coordination of an EResource Trial Focus Group
Selection and Coordination of an EResource Trial Focus Group
Selection and Coordination of an EResource Trial Focus Group
Suggestions for devising questions for the eresource focus group trial:
Have the questions available prior to running the
trial. (SurveyMonkey or Zoomerang are useful
tools for gathering and analyzing answers.)
 Make the process simple for the members of the
trial group so that they will want to continue in the
role.
 Use multiple choice and/or close-ended questions
for simplicity.
 Keep questions broad and simple. You are looking
for the reactions of the average user.

Selection and Coordination of an EResource Trial Focus Group
A sample questionnaire for participants of an e-resource
trial is provided in the handouts on AALL2GO
Sample Questions for Participants of E-Resource Trials.pdf
The Cost-Benefit Analysis

Factors to consider:





The database you are interested in versus competitors’ offerings
(In this case you may want to run similar database trials
simultaneously.)
The database you are interested in and there are no competitors.
The database you are interested in versus print formats and other
formats already in the library.
The database you are interested in versus group subscription rates
through a formal consortium such as NELLCO or OhioLINK or an
informal grouping of libraries.
The database you are interested versus free on the web (e.g. Free
EDGAR versus a commercial version).
The Cost-Benefit Analysis

Additional Factors to consider:

Just in Case v. Just in time:
Subscribe just in case someone needs it (mainly Academics)
vs.
Being able to obtain an item just at the time someone needs it.

Examples of Just in time:
 Pay per use subscriptions (i.e., PACER)
 Pay via Credit Card (Lexis stopped this practice in May 2011,
Westlaw and many public record websites continue to offer it.)
 Interlibrary Loans
The Cost-Benefit Analysis
Subscription v. Perpetual Access v. Digital Ownership v. Purchase Plus Access


Subscription- This is the traditional license agreement to “lease” licensed
material for a specified term at a specified cost. When the subscription term
ends, access to all the material also ends. The typical subscription period is
for one year with an annual recurring subscription fee.
Perpetual Access- Typically, when an e-resource license lapses, access to all
content is lost. Perpetual access provides the ability to retain access to the eresource materials after the contractual agreement for these materials has
passed. It is similar to maintaining access to the back run of a print journal
after cancelling the print journal subscription. Ownership of the content never
passes to licensee but access to all the material published up to the time of
cancellation remains in perpetuity. Perpetual access usually requires a lump
sum payment in advance plus an annual access fee. The annual access fee is
lower than a typical annual subscription fee thus making perpetual access
attractive for some libraries.
The Cost-Benefit Analysis
Subscription v. Perpetual Access v. Digital Ownership v. Purchase Plus Access


Digital Content Ownership- This is a purchase agreement whereby the library
obtains limited ownership rights to the contents of the e-resource. The
content is often provided to the library on a storage device (hard drive, CDROM or Digital Linear Tape) and may or may not include metadata files.
Ownership typically does not include the sale of the delivery platform or any
method for access (i.e., user interface, search engine, etc.). Ownership is a
large, one-time lump sum payment (though financing for a term of years is
often available.)
Purchase Plus Access- This is an agreement for digital content ownership
along with a subscription license agreement granting access to the content
and delivery platform on the publisher’s servers. There is a large one-time
lump sum payment for ownership and an annual “access” or “maintenance”
fee for the subscribed access to the content on the publisher’s server. The
annual access fee is usually substantially lower compared to the annual
subscription fee a library would pay to access the material.
The Cost-Benefit Analysis
Misc. Observations on Subscriptions, Perpetual Access,
Digital Ownership and Purchase Plus Access



The Smith, Leers, Roncevich’s article titles “Database Ownership: Myth or
Reality?” Law Library Journal 103, no. 2 (2011) 233-247 provides an
excellent case study on the difficulties of having to construct a database using
the files supplied by a publisher upon the cancellation of the access portion of
a Purchase Plus Access agreement.
Purchase Plus Access makes sense when you have a sizeable amount of onetime only money left toward the end of a fiscal year or when a donor provides
a one-time, non-endowment donation to the library.
The traditional subscription license agreement allows for predictability in
budgeting akin to the annual recurring costs of print serials.
The Cost-Benefit Analysis
Additional Observations on Subscriptions, Perpetual Access,
Digital Ownership and Purchase Plus Access



The cost of usage under a traditional subscription license agreement can be
more readily ascertained and billed onto clients. Passing costs onto clients is
a much more difficult endeavor with perpetual access, purchase plus access,
or digital content ownership.
It may be worthwhile to subscribe to an e-resource for a limited time to
measure usage before deciding to go forward with an ownership or perpetual
access pricing model.
The big question is what happens to the perpetual access if an e-resource
publisher goes out of business or is sold?
The Cost-Benefit Analysis

So what is the cost-benefit analysis?
The cost of the subscription license or purchase agreement plus annual access fees
balanced against










The need for and uniqueness of the content contained in the e-resource.
Whether the cost can be recovered through billing on to clients.
Whether print material can be cancelled in reliance of the e-resource.
Whether space will be saved by removing cancelled print.
Whether loose-leaf filing and other processing costs will be saved.
Whether there is an anticipated large audience of users to justify the cost.
Whether the e-resource can be used for instructional purposes.
Whether the e-resource will save researchers significant amounts of time.
Whether the e-resource can be accessed by multiple simultaneous users.
Whether the e-resource can be accessed from home or other offsite location.
Cost Recovery Through Billing Clients




This topic is only briefly covered here as a full program, “To Recover
or Not Recover: Trends, Solutions, and Alternatives for Taming
Online Research Costs,” is devoted to the topic on Monday, July 25th
from 2:15 to 3:30.
ABA Comm. on Ethics and Professional Responsibility, Formal Op.
93-379 “authorizes lawyers to charge reasonable costs associated
with online legal research as long as the charges are disclosed to
clients. The guiding principle around ethical practice is that the
client charge should be commensurate with the service charge and
not be marked up for firm practice.”
In other words, firms should not make a profit through online legal
research mark-ups.
Many states also have opinions concerning online legal research
cost recovery.
Cost Recovery Through Billing Clients
Percentage of firms charging for online research:



According to a survey of 81 firms (50 responses of which were from firms
with more than 500 attorneys) conducted by Mattern & Associates in 2010,
73% of respondents charge clients for online legal research. That is a drop
from the 97% of firms that said they charged for online legal research
when the study was conducted in 2008. (See: Mattern, Robert. “Cost
Recovery Breeds Client Frustration in New Survey.” Law Technology News,
April 21, 2011)
According to a study conducted by the ABA General Practice, Solo &
Small Firm Division in 2008, 50% of small and solo firms recover all
or part of their online legal research costs and 50% do not.
According to the 2008 study, the cost recovery process varies widely
among firms.
While cost recovery is on the decline, it can still be a very influential
factor in e-resource selection for many firms.
Cost Recovery Through Billing Clients
Articles on low cost alternatives to Westlaw and Lexis:


Justiss, Laura K. “A Survey of Electronic Research Alternatives to
LexisNexis and Westlaw in Law Firms.” Law Library Journal 103, no.
1 (2011): 71-90.
Svengalis, Kendall F. “Chapter 25: Computer-Assisted Legal
Research” in Kendall F. Svengalis. Legal Information Buyer’s Guide
and Reference Manual 2010. N. Stonington, CT: Rhode Island
LawPress, 2010, 143-159.
THANK YOU!
A Bibliography for this Program is available on AALL2GO.