Overview of
Evaluation Methods
Barbara M. Wildemuth
Professor, SILS, UNC-CH
Questions that might be asked
Questions about the
current users of the
collection
Purposes, methods,
impact
Questions about
potential future
audiences
Purposes, methods,
impact
NC sea rescue crew, NC State Archives
Focus of the study
Evaluation versus
basic research
Object of study
Univ. of Michigan football teams, 1884, 1994,
Bentley Historical Library, Univ. of Michigan
Evaluation criteria
Appalachian Trail,
Shenandoah
National Park,
National Park
Service Archives
Effectiveness
Utility/ usefulness/
impact
Satisfaction
Pack train,
Keystone-Mast
Collection,
1870-1963,
Online Archive
of California
Using the criterion for evaluation
Criterionreferenced
evaluation
Normreferenced
evaluation
Ladies Hockey Team, ca. 1910,
Collecting Toronto: Through the Eyes of Larry Becker,
City Archives of Toronto
Data collection
Watching the
users
Asking the
users questions
Nazaret ‘Chick’ Cherkezian and NET camera,
National Public Broadcasting Archives,
Univ. of Maryland
Methods for watching the users
Direct observation
during use
Think-aloud
protocols
Library of Congress Manuscript Division
Transaction logs
Archdiocese of Milwaukee Archives
Methods for asking questions
Questionnaires
Surveys
Measures
Interviews
Focus groups
Campaign poster for
UNC Daily Tar Heel editor, 1954
Univ. of North Carolina
Southern Historical Collection
Sample satisfaction measure
McKinney, Yoon, & Zahedi, 2002
After using this Web site, I am…
Very
dissatisfied
1 2 3 4 5 6 7 8 9 10 11
Very
satisfied
After using this Web site, I am…
Very
displeased
1 2 3 4 5 6 7 8 9 10 11
Very
pleased
Using this Web site made me…
Frustrated
1 2 3 4 5 6 7 8 9 10 11
Contented
After using this Web site, I am
Terrible
1 2 3 4 5 6 7 8 9 10 11
Delighted
Will never
use it again
1 2 3 4 5 6 7 8 9 10 11
Will definitely
use it again
After using this Web site, I…
Study sample
Define the
population of
interest
Consider sample
size
Develop a
sampling plan
Probabilistic sampling
Nonprobabilistic
sampling
Caroline Virginia Still Anderson (1848-1919),
Presbyterian Historical Society
Research design
Controlling
extraneous variables
Achieving internal
validity
Achieving external
validity
Alexander’s Ragtime Band,
Historic American sheet music,
Rare Book, Manuscript & Special Collections Library,
Duke University
Data analysis: quantitative
Descriptive
statistics
Inferential
statistics
Quarantine Station, D. A. McLaughlin,
National Archives of Canada,
Canadian Genealogy Centre
From results to conclusions
Check for
statistical
significance
Check for
meaningfulness
of results
Draw conclusions
University College, University of Toronto
Archives & Records Mgmt. Services
Data analysis: qualitative
Qualitative content
analysis
Specifying a unit of
analysis
Recognizing themes
Looking for negative
evidence
Boy’s hat, Documenting the
Southeast Asian Refugee Experience,
Univ. of California, Irvine,
Special Collections and Archives
From results to conclusions
Identify themes
for further
analysis
Eleanor Roosevelt,
1937 Farm & Home Week,
Cornell University Archives
Weigh positive
against negative
Draw
conclusions
Richard Nixon &
Elvis Presley, 1970,
National Archives &
Records Admin.
Review
The research/
evaluation question
Evaluation criteria
Data collection
Study sample
Research design
Data analysis
Drawing conclusions
Bank, bottle, bumper sticker,
Coca-Cola Company Archives
Photo/ image credits (1)
Archdiocese of Milwaukee Archives,
www.archmil.org/aboutus/dept.asp?ID=469
Bentley Historical Library, University of
Michigan, www.umich.edu/~bhl/
Canadian Genealogy Centre,
www.genealogy.gc.ca
City Archives of Toronto,
www.city.toronto.on.ca/archives/
Coca-Cola Company Archives, www2.cocacola.com/heritage/ourheritage.html
Cornell University Archives,
rmc.library.cornell.edu/collections/subjects/
archives.html
Duke University, Rare Book, Manuscript, and
Special Collections Library,
scriptorium.lib.duke.edu/
Cunard Passenger’s log book,
Mystic Seaport Library
Photo/ image credits (2)
Mystic Seaport Library,
www.mysticseaport.org/library/home.cfm
Library of Congress Manuscript Division,
www.loc.gov/rr/mss/
National Archives and Records
Administration, www.archives.gov
National Park Service Archives,
www.cr.nps.gov/history/hisnps/NPSarchive
s.htm
National Public Broadcasting Archives,
University of Maryland,
www.lib.umd.edu/NPBA/
North Carolina State Archives,
www.ah.dcr.state.nc.us/sections/archives/
arch/
Online Archive of California,
www.oac.cdlib.org
Back cover of “the state of
Connecticut vs. new haven 9”
Yale University Manuscripts & Archives
Photo/ image credits (3)
Presbyterian Historical Society,
www.history.pcusa.org
University of California, Irvine, Special
Collections and Archives,
www.lib.uci.edu/libraries/collections/
special/special.html
University of North Carolina Southern
Historical Collection,
www.lib.unc.edu/mss/shcabout.html
University of Toronto Archives &
Records Management Services,
www.library.utoronto.ca/utarms/arch
ives/
Yale University Manuscripts &
Archives, www.library.yale.edu/mssa/
Winston Churchill to
Lady Randolph Churchill, May 1882
Churchill Archives Centre, Cambridge, UK,
Churchill and the Great Republic (exhibit),
Library of Congress Manuscript Division