A Study on Search Effectiveness:
How can we design search tools so end users find it and find
it quicker?
Margaret Hentz, B.S., M.L.S, D.S.I.S.
Technical Manager, The Dow Chemical Company, Indianapolis, Indiana
Jenny Emanuel, B.A., M.L.I.S.
Assistant Professor, Digital Services & Reference Librarian, University of Illinois
at Urbana-Champaign
Abstract
Search interfaces are often built without a full understanding of how users search
and conduct research. Factors independent of the search interface can have profound
effects on the search process and can be easily overlooked. It is important to identify
these factors and then develop improvements in search interfaces, processes, and training
activities so end users can conduct information searches more effectively and efficiently,
enabling users to spend more time using information.
A two year collaborative research project with The Dow Chemical Company and
researchers at the University Library at the University of Illinois at Urbana-Champaign is
underway to study end users’ search behavior. The project started with a general assessment of
available Research & Development (R&D) information research tools at Dow, surveying current
Dow end users to determine their searching behavior, and analyzing these results. Following the
general survey, a smaller pool of users will participate in individual user experience testing. This
testing will utilize both remote unmediated task-based tools and remote moderated interview
tools that record all computer use, audio, and video of how individuals search for information
and conduct research. The collected data will be triangulated with data collected from the survey.
Analysis of the collected behavioral information will lead to enhancements to the search
interfaces to existing Dow R&D systems which would be developed, pilot-tested, and deployed,
as appropriate.
Background
Understanding how users search and conduct research is a fundamental part of interface
and interaction design. It is critical to identify the factors that can affect the research process in
order to develop improvements to search interfaces, the search process, and training activities
that can help users conduct research more effectively and efficiently.
There are many elements that can influence how individuals search for information.
Technology experience and skill, academic background, language abilities, and interface
elements, among other factors, all influence how a researcher searches for information. It is
important to identify and take into consideration as many of these elements as possible and
utilize a variety of research methods to obtain as complete picture as possible as to how
individuals perform research and search for information online in order to provide
recommendations to improve the process.
Numerous tests are available that can obtain data about how individuals search for
information and complete their research. Ethnographic research is a process in which a
researcher observes individuals completing activities in their natural environment, and can be
carried out through a variety of methods, including surveys, semi-structured interviews, and
observational studies. For the semi-structured interview process, researchers ask provoking
questions that lead individual to discuss their thoughts about specific tasks, alongside interface
comparison, user expectations, and opinion data to derive a holistic view of how individuals
search and conduct their own research. Observational studies, in which the researcher-observer
watches a subject perform a task, is an additional ethnographic technique that is utilized with
software such as Morae to record the subject’s thoughts and actions as them complete both nonstructured and structured tasks.
Additionally, understanding how scientific researchers conduct research can have
implications for information literacy instruction in academic libraries. The groundwork for
learning how to conduct searches for scholarly information is laid through assignments and lab
work while individuals obtain degrees, especially in Master’s and Doctoral programs that require
a thesis or dissertation. Most information literacy instruction discussion focuses on
undergraduate students, though graduate student needs and skills are an emerging area of
research related to information literacy. Understanding the research needs of scientists working
in industry can influence how librarians approach information literacy instruction among
students in graduate programs, who may one day work as scientists in industry themselves.
Literature Review
Studying the information seeking habits of a user group is a common research practice
that library practitioners have utilized in order to understand how their patrons respond to
interfaces and resources. The data gathered in such studies not only aid librarians in
understanding how patrons find information, but also can have implications on interface design,
taxonomy, ontology, and instruction efforts. User data can also show practitioners the strategies
researchers utilize in order to process, acquire, verify and disseminate information, as well as
examine potential access issues and attitudes.i
Understanding how people find information allows librarians and organizations to
develop taxonomies and data structures that ultimately improve the search experience, through
better internal organization structures or recommendations to third party vendors, to improve
data structures of commercial products to enhance retrieval.ii Most user-based studies have
focused on folksonomies, in which users, and not information professionals, assign keywords to
documents in order to enhance findability.iii
Information seeking habit research can be done in many ways, including asking
individuals to self document their attempts at finding informationiv, interviewingv, surveyingvi,
and task based observation studiesvii. Studies of information seeking behaviors tend to involve
only one method of data collection, though large scale studies such as the OCLC study
“Scholarly Information Practices in the Online Environment: Themes from the Literature and
Implications for Library Service Development” employ multiple methods and triangulate the
data in order to achieve results that are generalizable and reliable.viii Most studies and especially
large scale studies such as the “National Study of Information Seeking Behavior of Academic
Researchers in the United States” employ only survey methods due to the ease in conducting
survey across geographical limitations.ix No evidence of large scale studies utilizing qualitative
techniques including interviews and observation methods have been found.
To date, few studies have focused on the information seeking habits of users in the
corporate research setting. Most research focuses on higher education institutions, as the setting
provides a convenient sample at the most common place where such studies originate. Cecelia
Brown conducted a survey among mathematicians and scientists at the University of Oklahoma
and found that faculty in these areas emphasize preprints to stay current in their field and obtain
other article information primarily from the citation lists in other journal articles. Faculty also
tended to retain their own collection of scholarly articles, obtained through preprint articles over
utilizing library indexes and subscription services to find information.x A large scale (n > 2,000)
survey study of scientists of varying disciplines at five United States research universities
showed that researchers most frequently used electronic journals available through the university
library subscription, followed by open access subscriptions and personal electronic journal
subscriptions. It also showed that users are as likely to start their research from Google as they
are from the library’s website or indexes and that about half of users maintain their own personal
bibliographic database.xi
Several studies focus on science and engineering faculty at universities in the United
States, users with similar education, research experience, and information needs such as those at
The Dow Chemical Company. Several studies looked at search result patterns and article
downloads of American Chemical Society (ACS) journals by looking at search logs. xii A 2004
Cornell University study also looked through ACS transaction logs and found that faculty are
most likely to view an ACS journal article directly from SciFinder Scholar or PubMed, with only
18.2% of referrals coming from the library’s electronic journal finder tool. The study also found
substantial referrals coming from generic Internet searches, from email messages, and
interactions with colleagues.xiii Robbins, Engel, and Kulp of the University of Oklahoma
surveyed engineering faculty at twenty public research institutions found that access to electronic
resources were the most important tools that libraries can make available to faculty, followed by
interlibrary loan and library databases. The Robbins study also noted that information seeking
habits and tools utilized are generalizable from one institution to another.xiv A 2007 study from
the University of North Carolina also surveyed faculty scientists and emphasized the importance
of online tools and stated that researches utilize (in order) electronic journals, web pages,
databases, and personal communication when conducting research. The UNC study also found
that when it comes to searching for information, databases are the most commonly used tool,
followed by web search engines, full text digital libraries, and personal search tools, with the
library collection being the least searched tool.xv
Project Description
This project is a two-year collaborative research agreement between The Dow Chemical
Company (Dow) and the University of Illinois at Urbana-Champaign as part of a company-wide
initiative for research collaborations with selected universities throughout the country. An initial
project proposal included a market/opportunity statement, project description, a look at the
competitive landscape, and major project milestones. A project team consisting of members of
the Technical Information Services (TIS) group at Dow and a principal investigator and assistant
from the University of Illinois was formed. Since members of the project team were spread
across four different states (Indiana, Michigan, Illinois, and Pennsylvania), a Share Point site was
established to share project documents and enhance collaboration. Ongoing biweekly project
meetings are scheduled and participants attended remotely via virtual audio and web
conferencing.
The project team first assessed what research tools are available to end users at Dow and
made sure that all team members, especially the representatives from the University of Illinois,
were familiar with the search interfaces for each of the tools. The team then developed and finetuned a survey for the end users. In addition to gathering demographic data, the purpose of the
survey was to determine how the users’ education background, training, and experience with
information impacted their searching behavior. At any time, the user could exit the survey and
not answer all of the questions. An incentive (a drawing for some prizes) is provided to
encourage users to take the survey. The follow up interview and observational studies will be
covered in a separate, future publication.
Survey Details
The survey questions can be grouped into major themes including: demographics,
comfort level with tools related to information seeking behaviors, use of popular technologies
and devices, use of information resources during one’s education, training and education
experiences, and use of information tools offered by Dow. The survey data will be analyzed
using descriptive statistics to determine the skills and needs of Dow researchers as well as
examine correlations on particular demographics, background, and skills.
Demographics data is being collected in order to determine if language, gender,
education, job classification, business/function, length of professional employment, and
geographical site location correlated with the use of Dow’s information tools. To avoid
ambiguity in responses, the format of some of these questions was the same as an employee-wide
survey that Dow employees take on an annual basis. Other questions were specific to the
academic questions posited by the library researchers at the University of Illinois at UrbanaChampaign.
In addition to specific R&D tools available through Dow, the end user’s comfort level
with an extensive list of general information technologies was determined to see if technology
skills are factors with the effective use of information tools. The information technologies
included various types of mobile devices, social networking sites, operating systems, and
software tools. Understanding what tools users utilize for both professional and personal
information seeking and dissemination can influence instruction, services, and tools that are
offered professionally to users.
There are also survey questions on the extent that library information resources (print
and/or electronic) were used during one’s education, how one learned to use these tools, and how
one’s education prepared the respondent to deal with research within industry. Usage statistics
for the information resources provided by TIS and which features of these tools were used is an
important factor that is analyzed alongside other collected data. To see if Dow’s custom training
on information tools could be improved, the survey included questions on the types of training
that the respondent took (if any), the usefulness of that training, and if other types of training
should be offered. Additionally, the survey inquired about how the process respondents use to
learn about new technologies.
Survey questions also include information on education outside of the offered training by
TIS and how Dow employees learned about the resources and technologies that they use to
conduct research related to their job. These questions were added in order to gain an
understanding of the role that higher education and other methods played within developing
effective information seeking habits. The conclusions from the survey can also be used to
influence how academic libraries provide instruction to students who may go on to research jobs
in industry, including Dow. A copy of the survey is in the appendix.
Remaining Project Activities
Once the survey results are analyzed, then the ethnographic interviews and observation
sessions will be conducted to observe individuals completing information searching activities in
their native work environment. Individuals will be chosen from survey respondents who
responded to a call for a follow up, in-depth study, and sampled by factors such as age,
education, Dow work unit, and geography in order to obtain a diverse set of users. The data will
be collected by several qualitative methods, including the semi-structured interview observation
of sample members performing both uniform example searches and unstructured information
seeking activities based on the subject’s recent research activities. These interviews will be
conducted remotely utilizing web conferencing software and Morae to record and analyze user
interactions with interfaces, speech, and video. The semi-structured interviews and observation
sessions will result in screencasting with both audio and video of how individuals use Dow’s
information tools for searching. The data will focus on specific features and functions that could
be changed to improve an individual’s search effectiveness. The collected data would also help
gain insights on how users search and disseminate information. The audio and video data will
also be compiled with data using the online service Loop11, which tracks user actions through
structured searches. Loop11 allows for task-based user behavior testing that does not involve a
live moderator, but does allow for more real-time data collection at the user's convenience.
Once the data has been coded and analyzed for patterns possible changes in the search
interfaces, selected enhancements to the Dow end user tools will be made, tested and validated
by a pilot group, and deployed as appropriate. The analyzed data will also be shared in various
scholarly sources, including conference proceedings and journal articles. As this is the first
large-scale study of information seeking habits of researchers that uses qualitative measurements,
the results of this study can have implications beyond its initial scope because it will show, on a
large scale, how research scientists working in industry (and not higher education) conduct
research
Conclusion
This project is among the first large-scale information seeking habit research studies
among professional researchers and it has implications in many different areas, both within Dow
and within the broader library and information science scholarship. The results will indicate the
factors that influence research behavior, These factors could include education or job
experience, language, familiarity with other information technologies, the type of training taken
by the end user (if any), previous experience with similar information tools, etc. The results of
this project can be taken into consideration during the development of information tools and
educational curriculum so that both tools and end users can be more efficient and effective in
their research.
Endnotes
Lokman Meho and Helen R. Tibbo. “Modeling the Information-Seeking Behavior of Social
Scientists: Ellis’s Study Revisited,” Journal of the American Society for Information Science
and Technology 54 (2003): 570-587.
i
Emma Marya Coonan. “Navigating the Information Landsacpe,” Serials Librarian 61 (2011):
323-333, accessed April 9, 2011, doi: 10.1080/0361526X.2011.618345.
ii
B. E. Dixon and J. J. MGowan. “Enhancing a Taxonomy for Health Information Technology:
An Exploratory Study of user Input Towards Folksonomy,” Studies in Health Technology and
Informatics 160 (2010): 1055-1059.
iii
Vicky Duncan and Lorraine Holtslander. “Utilizing Grounded Theory to explore the
Information-Seeking Behavior of Senior Nursing Students,” Journal of the Medical Library
Association 100 (2012): 20-27, accessed April 9, 2011, doi: 10.3163/1536-5050.100.1.005
iv
Lokman Meho and Helen R. Tibbo. “Modeling the Information-Seeking Behavior of Social
Scientists: Ellis’s Study Revisited,” Journal of the American Society for Information Science
and Technology 54 (2003): 570-587.
v
Xuemei Ge. “Information-Seeking Behavior in the Digital Age: A Multidisciplinary Study of
Academic Researchers,” College & Research Libraries 71 (2010): 435-455.
vi
Jeonghyun Kim. “Task as a Context of Information Seeking: An Investigation of Daily Life
Tasks on the Web,” Libri 58 (2008): 172-181,
vii
Carole L. Palmer, Lauren C. Teffeau, and Carrie M. Pirmann. “Scholarly Information
Practices in the Online Environment: Themes from the Literature and Implications for Library
Service Development,” (OCLC Research and Programs Report, 2009). Accessed online April 9,
2011. http://www.oclc.org/programs/publications/reports/2009-02.pdf
viii
Xi Nui and Bradley M. Hemminger, et al. “National Study of Information Seeking Behavior of
Academic Researchers in the United States,” Journal of the American Society for Information
Science and Technology, 61 (2010): 869-890.
ix
Cecelia M. Brown. “Information Seeking Behavior of Scientists in the Electronic Information
Age: Astronomers, Chemists, Mathematicians, and Physicists, “ Journal of the American Society
for Information Science 50 (1999): 929-943.
x
Xi Nu et al. “National Study of Information Seeking Behavior of Academic Researchers in the
United States,” Journal of the American Society for Information Science and Technology 61
(2010): 869-890. Accessed online April 9, 2011, doi: 10.1002/asi.21307.
xi
Angel Borrego and Cristobal Urbana. “Analysis of the Behavior of the Users of a Package of
Electronic Journals in the field of Chemistry,” Journal of Documentation 63 (2007): 243-258.
Accessed online April 9, 2011, doi: 10.1108/00220410710737204.
xii
Philip M. Davis. “Information-Seeking Behavior of Chemists: A Transaction Log Analysis of
Referral URLs,” Journal of the American Society for Information Science and Technology 65
(2004): 326-332. Accessed online April 9, 2011, doi: 10.1002/asi.10384.
xiii
Sarah Robbins, Debra Engel, and Christina Kulp. “How Unique are our Users? Comparing
Responses Regarding the Information-Seeking Habits of Engineering Faculty,” College and
Research Libraries 72 (2011): 515-532.
xiv
Bradley M. Hemminger, Dihui Lu, K. T. L. Vaughan, and Stephane J. Adams. “Information
Behavior of Academic Scientists,” Journal of the American Society for Information Science and
Technology 58 (2007): 2205-2225.
xv
Appendix
Informed Consent T
This is a research study by Jenny Emanuel, a librarian in the University Library of the University
of Illinois, in collaboration with The Dow Chemical Company Technical Information Services
(for the TIS website, click here) and the support of Theresa Kotanchek, to examine how
researchers at The Dow Chemical Company use library and information research resources and
conduct research. You have been invited to participate in this study because you are an employee
of The Dow Chemical Company. Participation is voluntary. If you consent to participate in this
study, you will complete a survey about the library and information research strategies,
technologies, and resources you use to conduct research at Dow. There are 36 questions, and it
should take 1015 minutes to complete. You are free to skip any question for any reason and to
quit the survey at any time. All surveys data is anonymous to the researchers and identifying
information collected at the end will be used for purposes of a follow up study and incentive, and
will be separated from survey answers There are no anticipated risks to your participation
beyond those that exist in daily life. You must be at least 18 years of age to complete the survey.
If you complete this survey and enter your email address, you will be entered to receive one of
ten $50 credits to Amazon.com. It is expected that 1,000 individuals will participate in this study,
giving the odds of winning a credit 1 in 100.
Data collected in this survey will be analyzed by researchers at the University of Illinois. An
aggregate of the collected data will be shared in the form of a report with project stakeholders at
Dow.
If you have any questions or concerns about participating in this study, you may contact the
principal investigator, Jenny Emanuel, at emanuelj@illinois.edu or 2172443770.
If you have any questions about your rights as a participant in this study please contact the
University of Illinois Institutional Review Board at 2173332670 or via email at irb@illinois.edu.
By continuing to the next page of the survey, you thereby grant your consent to participate in the
survey.
1. SIGNATURES
I have read this consent form and my questions have been answered. By clicking through this
agreement I confirm that I am at least 18 years of age and I do want to participate in the study. I
know that I can remove myself from the study at any time without any consequences.
Yes
No
Information Seeking at Dow
2. How many years have you searched electronically for information resources in school and on
the job?
Less than 5 years
5 to 9 years
10 to 19 years
20 to 29 years
30 or more years
Other
3. Why do you search for information resources for your job? Please check all that apply and
indicate frequency.
Never
Yearly
Monthly
Weekly
Daily
Regulatory/Environmental/Litigation
issue
Explore new idea or technology
(overview)
External funding/funding opportunities support
M&A, JV, Licensing Support
Patent process or support (Novelty,
FTO, Validity, Opposition)
Product development, commercialize, manufacture, or solve problem
Current awareness
Other (including citation, author,
assignee searches)
If other, please specify:
4. Please indicate how frequently the following statements occur at work:
Never
Yearly
Monthly
I prefer to look for information resources myself
I prefer someone to help me look for
information only when there are significant consequences to missing
information resources (e.g., for business or patent processes)
I prefer someone else to help me
look for information resources
5. I ask TIS for help when... (Please check all that apply.)
I do not know what tool to use.
A tool is too difficult or time consuming to use.
I do not have time.
I cannot access a tool from my virtual office (or similar reason).
Too much information is needed for me to process it myself.
Weekly
Daily
I do not ask TIS for help, I perform searches myself without the aid of someone else.
What is TIS?
Other (please specify)
6. Do you search for information resources when you work remotely?
Yes
No
If yes, do you sometimes have any issues while searching? Please describe.
7. How do you access information resources provided by TIS? Please check all that apply.
Public/shared computer
Personal work station
Home computer
Work mobile device
Personal mobile device
Other (please specify)
8. How often do you use the following information resources?
Never
Yearly
Monthly
Weekly
Daily
9. How often do you use the following features within TIS information resources?
Never
Yearly
Monthly
Weekly
Daily
Dow Registry
Online Card Catalog
Lab Notebook Tracking System
TIS AZ
Journal List (Link)
Google to search Dow Intranet
Google for external use
Google Scholar
ACS Journals
Knovel
SciFinder
Thomson Innovation
TIS Website
Web of Science
Wiley EJournals
Other EJournals
Search
Read documents
Submit documents
Work on projects
Help features
Sharing to social media
Links to citation management software
Tagging
Downloading full text articles
Emailing documents
Emailing results list
Creating alerts
Printing results
10. What might prevent you from using information resources provided by TIS? Please check all
that apply.
No need to use it
Unable to access
Forgot password
Trouble logging in
System response time
Too difficult to use
Cost
Other (please specify)
Training
The following questions are intended to assess training so that TIS can improve the training that
is offered to you as an employee of The Dow Chemical Company. Please answer to the best of
your ability.
11. What type of training have you received for the information resources provided by TIS?
How useful was it?
Not use- Neutral
Very use- N/A
ful
ful
Self-taught
Colleague(s)
Virtual Dow instructor led class
In-person
Dow instructor led class
Recorded videos
Webinars
Vendor conducted training
Online training materials/user documentation (provided by vendor)
Online help/user documentation
provided by Dow or TIS
Help available within the tool
12. Additional training on TIS research and information tools that would be helpful are...
(check all that apply.)
In an offsite classroom with a lecturer
As a webinar (at my desk) at a scheduled time
As a webinar (at my desk) archived for access on demand
As a guided computer tutorial
As a written guide with text and screenshots
Regularly scheduled walk-in coaching sessions
Other (please specify)
13. What can be done to improve existing training?
Demographic Information
14. What is your gender?
Female
Male
Do not wish to state
15. How many years have you worked professionally (at Dow and elsewhere)?
Less than 5 years
5 to 9 years
10 to 19 years
20 to 29 years
30 or more years
16. What is your current business/function at Dow?
Advanced Materials
Basics (Hydrocarbons, Chemicals & Energy)
Core R&D
Dow AgroSciences
Performance Plastics
Performance Products & Systems
Other (please specify)
17. What is your job level?
Administrative Family
Technician/Technologist Family
Functional Specialist/Functional Leader Family
18. Where are you located?
Global Leadership Family
Asia Pacific
Europe
Latin America
Middle East and Africa
North America
19. Is English your native language?
Yes
No
Education Information
Please tell us about your education. This will help us to learn more about your background doing
research as well as allow the researcher to examine the role higher education played in the
research practices of Dow employees.
20. What is your highest level of education?
High school diploma
Some college
Associate's Degree
Bachelor's Degree
Master's Degree
Doctoral Degree
Postdoctoral
Degree
Professional Degree (MD, JD, MBA)
Other (please specify)
21. How long ago did you graduate from your last degree program?
Less than 5 years
5 to 9 years
10 to 19 years
20 to 29 years
30 or more years
22. How did you learn to use search engines and other research tools (including print) in your
education?
I had instruction from a librarian or professor on library resources and search engines.
I taught myself how to use library resources and search engines.
I learned from a faculty member (non-librarian).
Other (please describe)
23. To what extent do you feel your education prepared you to deal with the research and
information seeking aspects of your job?
My education did not prepare me at all.
My education prepared me a little.
My education prepared me a lot.
Comments
Language Information
24. What is your first language?
25. Are you comfortable searching in English?
Yes
No
26. Are you comfortable receiving training in English?
Yes
No
Technology Experience
This section will help us to determine your comfort level with tools that are related to
information-seeking behaviors. The results will help us to select and develop new research tools
and technologies.
27. Please rate your comfort level with the following technologies at the present time:
Technology in General
Not Comfortable
Neutral
Very ComI do not
fortable
know what
this is.
Blogging software
EndNote
Mac OS X
Microsoft Office
Non MS Office Productivity Software
Mobile Devices Android
Mobile Devices iOS
Mobile Devices Blackberry
Mobile Devices Other
RSS Feeds
Social Networking Sites
Tablets Android
Tablets iOS
Tagging
Video Conferencing
Wiki software
28. Are there any other library and information research technologies not mentioned above that
you are currently familiar with? What are they?
29. Are there any other library and information research technologies not mentioned above that
you currently want to learn? What are they?
30. How do you learn about a new technology? Choose all that apply.
Help within a tool or supplied by its vendor
Other online training material or user documentation
Association or academic sponsored workshop
Recorded videos or webinars
Training through Dow
From friends and coworkers
I teach myself
Other (please specify)
Personal Technology Use
We'd like to get an idea as to what technologies you use in your private (non-Dow) life.
31. Do you have current accounts or use the following:
Blogger
CiteULike
Delicious
Facebook
Google+
Google Docs
Google Reader
Instant Messaging (Any Service)
LinkedIn
Mendeley
Pinterest
Skype
SlideShare
Twitter
TypePad
Vimeo
Wikipedia
WordPress
Yahoo Pipes
YouTube
Zotero
Any other related tools?
Follow Up
32. Would you be interested in participating in a follow-up interview designed to more closely
examine the information seeking behaviors of Dow employees? If so, please indicate your Dow
email address so I can follow up with you at a later date.
33. Additionally, if you would like to be entered in the drawing for $50 in Amazon credit, please
list your email below. Entering your Dow email does NOT put you on a list for a follow-up
interview.