Overview of Evaluation Methods Barbara M. Wildemuth Professor, SILS, UNC-CH Questions that might be asked Questions about the current users of the collection Purposes, methods, impact Questions about potential future audiences Purposes, methods, impact NC sea rescue crew, NC State Archives Focus of the study Evaluation versus basic research Object of study Univ. of Michigan football teams, 1884, 1994, Bentley Historical Library, Univ. of Michigan Evaluation criteria Appalachian Trail, Shenandoah National Park, National Park Service Archives Effectiveness Utility/ usefulness/ impact Satisfaction Pack train, Keystone-Mast Collection, 1870-1963, Online Archive of California Using the criterion for evaluation Criterionreferenced evaluation Normreferenced evaluation Ladies Hockey Team, ca. 1910, Collecting Toronto: Through the Eyes of Larry Becker, City Archives of Toronto Data collection Watching the users Asking the users questions Nazaret ‘Chick’ Cherkezian and NET camera, National Public Broadcasting Archives, Univ. of Maryland Methods for watching the users Direct observation during use Think-aloud protocols Library of Congress Manuscript Division Transaction logs Archdiocese of Milwaukee Archives Methods for asking questions Questionnaires Surveys Measures Interviews Focus groups Campaign poster for UNC Daily Tar Heel editor, 1954 Univ. of North Carolina Southern Historical Collection Sample satisfaction measure McKinney, Yoon, & Zahedi, 2002 After using this Web site, I am… Very dissatisfied 1 2 3 4 5 6 7 8 9 10 11 Very satisfied After using this Web site, I am… Very displeased 1 2 3 4 5 6 7 8 9 10 11 Very pleased Using this Web site made me… Frustrated 1 2 3 4 5 6 7 8 9 10 11 Contented After using this Web site, I am Terrible 1 2 3 4 5 6 7 8 9 10 11 Delighted Will never use it again 1 2 3 4 5 6 7 8 9 10 11 Will definitely use it again After using this Web site, I… Study sample Define the population of interest Consider sample size Develop a sampling plan Probabilistic sampling Nonprobabilistic sampling Caroline Virginia Still Anderson (1848-1919), Presbyterian Historical Society Research design Controlling extraneous variables Achieving internal validity Achieving external validity Alexander’s Ragtime Band, Historic American sheet music, Rare Book, Manuscript & Special Collections Library, Duke University Data analysis: quantitative Descriptive statistics Inferential statistics Quarantine Station, D. A. McLaughlin, National Archives of Canada, Canadian Genealogy Centre From results to conclusions Check for statistical significance Check for meaningfulness of results Draw conclusions University College, University of Toronto Archives & Records Mgmt. Services Data analysis: qualitative Qualitative content analysis Specifying a unit of analysis Recognizing themes Looking for negative evidence Boy’s hat, Documenting the Southeast Asian Refugee Experience, Univ. of California, Irvine, Special Collections and Archives From results to conclusions Identify themes for further analysis Eleanor Roosevelt, 1937 Farm & Home Week, Cornell University Archives Weigh positive against negative Draw conclusions Richard Nixon & Elvis Presley, 1970, National Archives & Records Admin. Review The research/ evaluation question Evaluation criteria Data collection Study sample Research design Data analysis Drawing conclusions Bank, bottle, bumper sticker, Coca-Cola Company Archives Photo/ image credits (1) Archdiocese of Milwaukee Archives, www.archmil.org/aboutus/dept.asp?ID=469 Bentley Historical Library, University of Michigan, www.umich.edu/~bhl/ Canadian Genealogy Centre, www.genealogy.gc.ca City Archives of Toronto, www.city.toronto.on.ca/archives/ Coca-Cola Company Archives, www2.cocacola.com/heritage/ourheritage.html Cornell University Archives, rmc.library.cornell.edu/collections/subjects/ archives.html Duke University, Rare Book, Manuscript, and Special Collections Library, scriptorium.lib.duke.edu/ Cunard Passenger’s log book, Mystic Seaport Library Photo/ image credits (2) Mystic Seaport Library, www.mysticseaport.org/library/home.cfm Library of Congress Manuscript Division, www.loc.gov/rr/mss/ National Archives and Records Administration, www.archives.gov National Park Service Archives, www.cr.nps.gov/history/hisnps/NPSarchive s.htm National Public Broadcasting Archives, University of Maryland, www.lib.umd.edu/NPBA/ North Carolina State Archives, www.ah.dcr.state.nc.us/sections/archives/ arch/ Online Archive of California, www.oac.cdlib.org Back cover of “the state of Connecticut vs. new haven 9” Yale University Manuscripts & Archives Photo/ image credits (3) Presbyterian Historical Society, www.history.pcusa.org University of California, Irvine, Special Collections and Archives, www.lib.uci.edu/libraries/collections/ special/special.html University of North Carolina Southern Historical Collection, www.lib.unc.edu/mss/shcabout.html University of Toronto Archives & Records Management Services, www.library.utoronto.ca/utarms/arch ives/ Yale University Manuscripts & Archives, www.library.yale.edu/mssa/ Winston Churchill to Lady Randolph Churchill, May 1882 Churchill Archives Centre, Cambridge, UK, Churchill and the Great Republic (exhibit), Library of Congress Manuscript Division