UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN LIBRARY
EVALUATION AND SUGGESTIONS FOR EFFECTIVE AND
SUSTAINABLE ASSESSMENT
July 2005
A Report by Steve Hiller and Jim Self
Visiting Program Officers, Association of Research Libraries
INTRODUCTION
The University of Illinois at Urbana-Champaign Library (UIUC Library) is one of an initial
group of seven libraries participating in the Association of Research Libraries (ARL) project,
“Making Library Assessment Work: Practical Approaches to for Developing and Sustaining
Effective Assessment.” ( http://www.arl.org/stats/newmeas/VPO_Hiller_Self.html). This two
year project is under the aegis of the ARL Statistics and Measurement Program, Martha
Kyrillidou, Director. ARL Visiting Program Officers Steve Hiller and Jim Self conducted the
evaluation of the UIUC Library assessment effort which was capped with a visit on May 23-24.
In addition to a presentation on effective library assessment, they met with library faculty and
staff and discussed a wide range of assessment and measurement related issues. The UIUC
Library also responded to a request for information about assessment activities, needs, and issues
before the visit. The following report summarizes the current environment and offers options
and suggestions for developing effective and sustainable assessment at the UIUC Library.
THE VALUE OF LIBRARY ASSESSMENT
An ongoing, sustainable assessment program can help supply the type of information that
enables the UIUC Library to make good decisions in uncertain environments. Assessment is not
just the collection of data but a process that can assist in better understanding the teaching,
learning and research needs of the academic community and the library resources and services
that support them. In an era of tight or flat budgets the margin for error narrows and assessment
can be a critical tool in assuring the wise allocation of resources that provide the most value and
beneficial outcomes for the University. Assessment can help give the Library the necessary
agility and nimbleness to maneuver quickly in this era of rapid changes.
Assessment should not be seen as the sole responsibility of one area in the organization or a
single library. For assessment to work effectively it needs to be part of the overall library culture
and integrated into planning, decision-making, and operations. Lakos and Phipps have defined a
culture of assessment in libraries as:
A Culture of Assessment is an organizational environment in which decisions are based
on facts, research, and analysis, and where services are planned and delivered in ways
that maximize positive outcomes and impacts for customers and stakeholders. A Culture
of Assessment exists in organizations where staff care to know what results they produce
and how those results relate to customers’ expectations. Organizational mission, values,
structures and systems support behavior that is performance and learning focused.
(Lakos, A. and S. Phipps, “Creating a Culture of Assessment: A Catalyst for
Organizational Change, portal: Libraries and the Academy, Vol. 4 (3), July 2004, pp
345-361, quote on p. 352).
2
We believe that assessment can flourish in a variety of organizational cultures as long as there is
leadership and commitment. In a recent article we note how assessment has worked effectively
in four academic research libraries (Universities of Arizona. Pennsylvania, Virginia and
Washington) with very different organizational cultures:
Administrative leadership and support are critical to fostering an environment that equips
staff with the vision, tools, and understanding necessary to make data-based decision
making an integral organizational value. Yet there is no one way to attain this. Each of
the four libraries discussed above takes a different approach to achieving this value, one
that is aligned with the culture, mission, goals and objectives of each institution. It is also
important to recognize that each of these libraries did not wait until all the organizational
pieces were in place but started incrementally. In the long run, success will be measured
by how effective each library is in using data wisely to provide the services and resources
that support the learning, research and information needs of its community. (Hiller, S.
and J. Self, “From Measurement to Management: Using Data Wisely for Planning and
Decision-Making,” Library Trends, Vol. 53 (1), Summer 2004, pp. 129-155), quote on p.
150)
CURRENT ASSESSMENT ENVIRONMENT AND ACTIVITIES AT UIUC
The University of Illinois at Urbana-Champaign Library is one of the world’s great academic
research libraries. The breadth and depth of its collections and information resources, the quality
and dedication of the library faculty and staff, and the scope of services provided to the campus
community all have been major factors in contributing to the prominence and excellence of the
University. Yet the funding environment which supported the Library’s greatness has changed
substantially in the past decade as well as the information environment and the way people find
and use library resources and services. As Paula Kaufman, University Librarian, told library
faculty and staff at her 2004 State of the Library address:
If we don’t change some of the ways we operate, and if we don’t change our expectations
about support from the University, we will not continue to be successful.
Now the new demands and expectations are coming faster and more furiously than ever
before, and just as we can’t muddle through financially hoping that the budget will reach
levels that are truly unattainable, we can’t muddle through by clinging to traditions and
modes of operating that no longer make sense.
Budget constraints and reductions during the past couple of years have led to the loss of 25 fulltime positions and 50 FTE student workers and graduate assistants. The University Library
Annual Report (January 28, 2005) noted:
It is clear that the Library cannot operate its system of 42 units as it has in the past. Our
current strategy is moving us towards consolidating units and services, both physically
and virtually, which, combined with strategies to leverage collection dollars is designed
to use our financial resources as efficiently and effectively as possible.
3
The Advisory Committee to the Associate University Librarian for Services (Services Advisory
Committee) provides guidance and advice on a number of service-related issues, including “How
to identify with precision what expectations our users have for our services.” This group has
taken the lead on several assessment initiatives including the recent large-scale surveys of
graduate students (2004) and undergraduates (2005). The Services Advisory Committee made
general survey data (frequency responses) available, and provided a brief summary of the
graduate results and later a series of recommendations for library action based on the results.
Previous general user surveys occurred in 1998 and 1999 and LibQUAL+™ was applied in 2001
and 2002. Service quality and satisfaction scores were exceptionally high in the LibQUAL+™
surveys, but it was felt that this survey instrument did not adequately capture data relevant to the
42 departmental libraries.
The Strategic Planning Committee (SPC) has also recognized the need for assessment in
developing the UIUC Library’s Strategic Plan for 2005-09. In addition to articulating a specific
goal of “assessing service quality”, they developed a number of action items under each goal,
and called for ways to measure progress through assessment, evaluation, bench-marking, and
measurement. While these action items were later removed from the plan for being too detailed
at this stage, the Library plans to develop performance measures and benchmarks to track
progress in achieving articulated goals.
Other assessment activities in the past few years have been undertaken by specific libraries and
units or programs. These have included a number of studies on the use of physical spaces and
the results have been used to adjust hours and staffing levels. Other efforts have reviewed use of
collections, especially for serial retention decisions and more recently for relocation to the new
Oak Street storage facility. It appears as though a number of libraries and units keep statistics of
various activities, but except for those done for ARL, there seems to be little coordination or
uniformity in what and how data is collected.
While UIUC library faculty conduct high quality research, in part due the rigor of their
promotion, tenure and merit processes, much of the research and results pertaining to UIUC do
not appear to be integrated into overall library planning and decision-making.
IDENTIFIED ISSUES AND CONCERNS
Prior to the site visit, the UIUC Library identified several assessment-related issues and concerns
in the response to our request for information. These included:
Help in using data that’s collected
Ability to discuss data intelligently
Presenting and reporting assessment results
Creating an evidence-based culture
During the site visit, these topics were discussed with a number of different library groups.
However, it quickly became apparent that a major issue was how to develop and sustain an
assessment program in a library culture that traditionally has valued a decentralized operation
with considerable autonomy for library faculty and the many library units. While some
consolidation and reorganization of these service points have taken place recently, the unit
4
perspective continues to be a major focal point for a number of library faculty and staff. Dean
Kaufman noted in her state of the library address:
We must stop clinging to all of those service points we provide, especially in the Main
Library. Whether we’re able to do some facility reconfiguration or not, we must find
ways to offer our services at fewer locations with fewer staff. Our patrons are
increasingly frustrated, our staff are increasingly stressed by high demands and bored by
routine work, and it’s just too expensive. . . We must also look at the nature, number and
mode of departmental libraries, retaining their traditional value and values but
reconfiguring them to be as effective in the 21st century as they were in the 20th.
Reporting of assessment methodology and results tends to take place within the rigorous
research and publication culture, where emphasis is placed on publication within peer-reviewed
journals. This practice works against rapid and widespread dissemination within the library as
the publication process can take considerable time. One library faculty member told us that she
reviewed an excellent article for publication, and then realized that it was UIUC Library
assessment-related research done by a colleague that she was unaware of. We were also told by
several library faculty that the method for funding research makes it easier to collaborate with
those outside the library than with library faculty colleagues although others said that wasn’t
true. While we didn’t determine whether this was actually the case, the perception that it is by at
least several faculty indicates a potential concern.
SUGGESTIONS AND OPTIONS FOR MOVING ASSESSMENT FORWARD
We offer the following suggestions and options for moving assessment forward at the UIUC
Library. They are not intended to be prescriptive but rather outline best practices for establishing
and sustaining a viable assessment program that can flourish at Illinois.
1. Provide leadership for assessment
Assessment cannot work effectively without a strong commitment from library leadership.
Library administration has to step up and address that challenge and articulate assessment as a
library priority for assessment to be effective and sustainable at Illinois. A critical first step
would be the appointment of a person who can lead the program, help develop a culture of
assessment and work with library faculty and staff to build a viable assessment program.
This person (hereafter referred to by the convenient name of “assessment coordinator”) would
not necessarily conduct assessment activities, but would help coordinate efforts, advocate for
assessment, serve as a resource person, know best practices, and promote understanding of the
value of assessment throughout the organization. An assessment team or a committee with
assessment as part of its charge (such as the Services Advisory Committee) can assist in support,
building staff buy-in and helping to conduct a specific assessment project, but it cannot run a
sustainable assessment program. An assessment coordinator can help in moving efforts forward,
integrating data into the decision making stream, and providing both guidance and support to the
assessment team.
5
Assessment should not be added on to existing responsibilities with no additional time or
resources provided. Assessment responsibilities can be part-time but support should be provided
with additional resources or a real reassignment of some or all of an individual’s current
responsibilities. Given the UIUC library culture, we believe the assessment coordinator should
be a tenured library faculty member, who has conducted and published assessment-related
research and has earned the respect of colleagues. We recommend an initial half-time
appointment to get the assessment program going, especially with the size and diversity of the
UIUC Library. There are several UIUC library faculty who have an interest in assessment and
also have requisite skills and expertise. Other characteristics that would be desirable for an
assessment coordinator include:
Commitment to library users
A broad perspective on issues that affect academic libraries and higher education
Understanding the value of assessment in improving library services
Skill in summarizing and presenting results effectively to diverse audiences
Ability to work effectively with faculty, staff, managers and administrators throughout the
organization
The assessment coordinator should be an ex officio member of the Services Advisory
Committee.
2. Develop an assessment program with realistic assessment goals and activities that focus
on how and where the library adds value to the academic community
It is increasingly important to measure how the library makes a positive difference to the success
of the academic enterprise. Assessment goals and activities should help identify those areas and
measure the value in ways that can be presented effectively to the University community and
other stakeholders. Progress towards achieving these goals should be benchmarked and
reviewed on a periodic basis. Priority for assessment activities should be given to efforts that
provide this type of information. However, activities need to be realistic and aligned with
organizational goals, resources, support and expertise.
The development and administration of large-scale surveys to graduate students (2004) and
undergraduates (2005) and one planned for faculty in 2006 provide an excellent start to building
a successful assessment program. Initiating a manageable and focused program lays the
foundation for successful long-term assessment and avoids unreasonable expectations. A
reasonable set of activities for the first two years would include:
Appoint an individual with responsibility for assessment
Work initially with an existing group such as the Services Advisory Committee
Take primary responsibility for survey analysis and reporting
Examine the feasibility of forming a separate assessment-related group
Identify and prioritize assessment needs through widespread consultation within the organization
Develop an assessment plan with clearly defined goals, objectives, support and benchmarks
Obtain training and skills necessary to understand and do good assessment
Promote a “culture of assessment” within the UIUC Library
Participate in appropriate professional venues
Help develop performance measures and benchmarks to support the Strategic Plan and other
programs
Ascertain best methods and support needed for staff involvement and participation in assessment
6
Initiate assessment activities that address priorities and can be accomplished in a timely fashion
Evaluate the success of the program and resources needed
3. Utilize assessment to move the strategic planning process forward
Assessment will be a crucial component in fostering the success of the new UIUC Library
Strategic Plan 2005-2009 (Unlocking Our Past Building Our Future). This strategic plan
implicitly calls for a culture of assessment to inform and validate library programs, services, and
vision. Assessment is a critical component of the first major area, “Expanding Our Service
Orientation,” with a specific goal for assessing service quality but is also mentioned in other
major areas. Additionally, assessment should be an important element in measuring progress
toward achieving strategic planning goals. Measures and benchmarks should be clearly
articulated, tied to specific actions and goals, and reviewed on a periodic basis
4. Leverage the research strengths of library faculty to better support assessment efforts
The UIUC Library faculty have established an enviable record of high quality research and their
efforts are well represented in noteworthy peer-reviewed publications. While library faculty
should be free to pursue research areas of interest, the Library should establish opportunities and
incentives to encourage convergence of faculty research abilities with areas of organizational
research interest and need. Faculty participation in this type of research would not affect their
ability to publish findings.
5. Review how and why internal data are collected and used
Currently, a significant amount of time is spent collecting a variety of internal statistics and data
not needed for reporting to external agencies. Many of these internal statistics are related to the
traditional physical library. However, it’s not clear if there still is a need to collect these data or
how they’re used. In addition to reviewing why they’re collected and used, an analysis of the
costs involved in data collection with the benefits of use should be conducted. Examination of
alternate methodologies such as sampling and automatic data capture should also be encouraged.
6. Establish a central repository for assessment-related data, statistics and research
A substantial amount of assessment-related data and research has been done by the Library,
library units and library faculty. However, at this time there is no mechanism for archiving the
data and making it accessible throughout the organization. It is difficult to discover and use what
has been collected and done. An initial inventory of data collected centrally and by library units
can be followed by one that pulls together library faculty research that pertains to the institution,
including methodology used. Efforts should be made to standardize institutional data in a format
that is accessible and usable by library faculty and staff. Another advantage of such an
electronic repository is that the availability of data may preclude the need for additional data
collection.
7
7. Report assessment-related results to internal and external communities and
stakeholders
Library initiated assessment-related efforts should be reported in a timely and understandable
manner to library faculty and staff, the campus community and external stakeholders. Making
the data available is helpful, but it is more useful to present results in a manner that identifies key
findings, what they mean, and how results will be used to improve library services. Planning for
such major assessment activities as the 2006 faculty survey should include an outline of how and
when results will be disseminated and presented. It is especially important that this information
is readily available to those who participated in the assessment effort.
Conclusion
We believe that the UIUC Library is ready to move forward with developing an effective and
sustainable assessment program. There is general recognition that the information and funding
environments have changed substantially during the past five years and the Library needs to
better understand how to support and partner with the campus community in maintaining
institutional excellence. Because of the unique library culture it is essential that this program
have not only strong administrative support but also the buy-in from the other library areas. The
person selected as assessment coordinator must be able to work effectively with library
administrators, faculty, staff and the many library units.