ASSESSMENT PLAN FOR REFERENCE SERVICES NSM 7/10/2012
The assessment plan was developed by the Reference Services Committee in order to evaluate the different outcomes of the Reference Services implementation, a New Service Model
program set forth in July 2011 and it covers all formats of reference that are currently available (virtual reference, embedded services, office hours, etc.) and it can be applied to all units that
provide reference services to library users. The assessment plan will also be used for ongoing evaluation of Reference services offered in the library, and as a guide for the Reference Services
Committee in their ongoing work. The Assessment Coordinator and LAWG will be involved in planning assessment, data collection, and analysis for all activities.
Assessment Goals:







Test the viability of the NSM implementation (see NSM-specific section of the grid)
Determine staffing, operating hours, and location of reference points based on patterns of user need
Collect comparative data across service points and hubs (qualitative data, user satisfaction, etc.)
Identify best practices amongst service points and hubs
Improve service quality
Facilitate access to subject specialists
Inform training needs
NSM Assessment
Assessment target
User satisfaction…
…with service at hubs
…with access to specialists
Continuing education needs for library
faculty/staff/students providing reference and
information service
Satisfaction among library faculty/ staff/students
with continuing education
Referral process
…effectiveness of the process
…user satisfaction with the process
…library personnel satisfaction with the process
Baseline data
Post-change data options
Sourcesi
Issues
LibQual
2006 Faculty/AP survey
2005 Undergrad survey
2004 Grad student survey
Unit records
SDTC records
GA orientation records
GA orientation feedback
LibQual
Focus groups
In-house surveys
WOREP
RSC
Hubs and Library units
Baseline data not targeted to Central
Reference.
Self-assessments
Supervisory assessments
Annual review of emerging needs
(e.g., new tools/services)
In-house survey (frequency?)
RSC
STDT
SAC and Admin Council
Morale survey
2009 SDTC customer service survey
Desk Tracker
Unit records
Online referral form
Desk Tracker
Library faculty/staff focus groups
Patron focus groups
RSC
SDTC
RSC
Hubs
Library units
How to evaluate completion of referral
process?
ASSESSMENT PLAN FOR REFERENCE SERVICES NSM 7/10/2012
Ongoing Assessment of Reference Services
Assessment target
Baseline data
Post-change data options
Who
Initial training /orientation needs for newly hired
staff
GA Orientation records
Service provider satisfaction with consolidated
public service points (hubs)
…serving as well as previous model?
…meeting unit needs (hubs and others)?
Library personnel (all levels) satisfaction with
referral process
Unit records
Specialists records
2009 SDTC customer service survey
Morale survey
Unit records and procedures
Specialists records
Self-assessments
Supervisory assessments
(as people are hired)
Survey
Focus groups
RSC
Hubs and Library units
SDTC
RSC
Hubs and Library Units
Subject specialists
Qualitative survey of referrals from
referral database
Focus groups
RSC
Hubs and library units
Subject specialists
Robustness of expertise within the library; i.e.,
number of people capable handling (most or
many) questions in a given specialty. Identify gaps
in knowledge or training
Recommend standard minimum Desk Tracker
implementation
Conduct of reference service interactions (face-toface or digital) at hubs, in outlying units, and direct
to specialists for accuracy, adherence to best
practices outlined in the literature.
Mode of service employed by users to access hubs
Subject specialist lists (historical)
Referral database
RSC
Hubs and Library units
READ Scale
RSC
Hubs and Library units
RSC
SAC and Admin Council
Many variations in fields used
May be using same fields differently
Need standard method for complaint
tracking
RSC
Hubs
RSC
Hubs and Library units
Subject specialists
RSC
Hubs and Library units
RSC
HR
SAC and Admin Council
Lack of standardized data collection in
Desk Tracker
Lack of standardized data collection in
Desk Tracker
Mode of service employed by users to access
specialists
Type, content, and complexity of questions asked
through available services
Alignment of available human resources with
reference and information service needs
Bob Burger’s study
Unit records
Admin records
WOREP
Complaint tracking
“Secret shopper”
Desk Tracker
Unit records
Desk Tracker
Unit records
Desk Tracker
Analytics on website
Desk Tracker
Referral Database
Desk Tracker, READ Scale
Desk Tracker, READ Scale
Desk Tracker
Unit records
Desk Tracker
Issues
Relies on self reporting of satisfaction
with old model for comparison
ASSESSMENT PLAN FOR REFERENCE SERVICES NSM 7/10/2012
Assessment target
Baseline data
Post-change data options
Who
Alignment of technology tools with reference and
information service needs
Clickthroughs to/from
SFX, Easy Search, Primo, etc.
IM Collaborator logs
Desk Tracker
Clickthroughs to/from
SFX, EasySearch, Primo, etc.
IM Collaborator logs
Desk Tracker
Computer-generated searches
Easy search management team
(ESMT)
ESMT
Primo?
RSC
CAPT
IT
Acquisitions (vendor use data)
SAC and Admin Council
ESMT
Use of electronic reference resources
Click through data
Same
Issues
Would this work on Primo
Can this metric be expanded to other
services?
?
Glossary
CAPT – Content Access Policy and Technology http://www.library.illinois.edu/committee/capt/charge.html
GA – Graduate Assistants
HR – University Library Human Resources http://www.library.illinois.edu/administration/human/
IT – University Library Office of Information Technology Planning and Policy http://www.library.illinois.edu/administration/it/index.html
LAWG – Library Assessment Working Group http://www.library.illinois.edu/assessment/lawg.html
READ Scale – Reference Effort Assessment Data Scale http://www.library.illinois.edu/assessment/readscale.html
RSC – Reference Services Committee http://www.library.illinois.edu/committee/ReferenceServices/ReferenceServicesCommitteeCharge.html
SAC – Services Advisory Committee http://www.library.illinois.edu/committee/advise/charge.html
SDTC – Staff Development and Training Committee http://www.library.illinois.edu/committee/staffdevelop/charge.html
WOREP – Wisconsin-Ohio Reference Evaluation Program http://worep.library.kent.edu/
i
Named groups may provide data, analyze data, and/or make recommendations. The role of SAC and Admin Council will be advisory in the planning process, and they are also key stakeholders who may make recommendations for
action.