Research Paper in Usability
FREE-ACCESS ACADEMIC RESEARCH
ENGINES:
www.scholar.google.com
www.academicindex.net
www.eric.ed.gov
By
Gregg Orr
Designs and Strategies for New Media
Dr. Min Liu
April 18, 2007
Table of Contents
Introduction ............................................................................................................. 3
Selection of Web Sites ............................................................................................ 4
Selection of Users ................................................................................................... 5
Plan for Testing ....................................................................................................... 6
Test Description .................................................................................................. 7
Pre-Test Questioning ............................................................................................ 8
Test Procedure ........................................................................................................ 9
Test Data ............................................................................................................... 10
Raw Data from Post-Test..................................................................................... 10
Other User Feedback .......................................................................................... 12
General Observations.......................................................................................... 17
Specific Observations ......................................................................................... 17
Interpretation of Results ........................................................................................ 18
General Analysis ................................................Error! Bookmark not defined.
Suggestions for Improvement ............................................................................... 19
Appendix A: PRE TEST QUESTIONAIRE......................................................... 21
Appendix B: Usability Test for Free Access Academic Research Engines.......... 22
Areas of research: .............................................................................................. 22
Sites ................................................................................................................ 22
Appendix C: POST TEST QUESTIONAIRE ...................................................... 23
References ............................................................................................................. 24
2
Introduction
As graduate students, most of us are accustomed to doing subject research
online. During previous generations, we would visit the university library and
look through the card catalog or periodical indexes. From there, we could follow
leads that we found to volumes that are stacked on the library shelves.
Today’s searches are quite different. Still through our university libraries,
we use the Internet to access hundreds or thousands of databases. We set search
criteria, and have to narrow the settings to limit our searches. Otherwise, the data
available in returns would be overwhelming. Through the university library
system or through workplace subscriptions, researchers have access to databases
containing papers, subscription journals, and some books. Most of these are
available in hard copy through a subscription, but people just needing a specific
article can take advantage of getting to just the information they need.
Not everyone doing research is enrolled at a university. They may not
have access through the workplace either, yet they may need access to
information. Outside of subscription, what are the best chances for accessing
scholarly materials? How easy is it to find these documents? Once located, can
regular people access the information for free? What is the quality of the data?
(Krug, 133)
These and other questions prompted me to investigate academic research
engines that are offered to the public for free. The following paper compares
three different free access academic research engines. Three users agreed to test
all the sites by searching for information in different and unrelated areas.
3
Selection of Web Sites
Several sites were explored and considered for research. Limiting
ourselves to academic research engines which did not require a subscription made
the work a little easier. There seem to be hundreds of “for-a-fee” research
engines, as well as hundreds of general search engines. The crowd grows a lot
smaller when we are only looking for sites where we can do academic research
without paying a subscription fee.
Since Google Scholar (www.scholar.google.com) is widely known and
fairly new, I wanted to include that engine, knowing that all of our testers have
had some experience with regular Google.
Intute (www.intute.ac.uk) is another site I tired which claimed to rival
Google. I found that it only turned up other search engines, no actual data, so I
ruled it out. For a while, Microsoft had a beta project out called “Windows Live
Academic.” It had good reviews and could be a competitor of Google, but was
not available at the time of research and writing.
AcademicIndex (www.academicindex.net) was interesting because it used
the criteria of an academic research engine, though it did not have the same depth.
I decided to test AcademicIndex because it works somewhere between a broad
research engine like Google Scholar and a premium fee-based service.
4
Lastly, I chose ERIC (www.ERIC.ed.gov) for our study. ERIC stands for
Education Resources Information Center, and is run by the federal government.
Because it has its own source of funding, ERIC can take a few extra steps in
setting up search engines, and in making data available. As will be explored in
more depth later, ERIC takes a little more ramp up time than the other research
engines, because of the detailed parameters available for search criteria (Krug,
133).
Selection of Users
Candidates for user testing were chosen among a mix of who would give a
broad representation and who was available. Since we could not have three
people fairly closely aligned, I chose three users who were evenly spaced apart
and a broad representation of who might want to make use of free access research
engines (Preece et al, 345).
First was Cherie, who finished her bachelor’s degree over 20 years ago.
She often has to do research of FDA issues, and normally Googles a term of what
5
she needs to look for. She finds it difficult to narrow down what materials are
useful and what can be tossed, and her workplace does not have a subscription to
a premium service.
Our second tester was Chad. He is in high school, and rarely opens a book
when researching a topic for a school project or term paper. Chad often starts off
by Googling a term, then digs into sites that he finds helpful. He is more
interested in presentation of the data than the quality of the data itself. Chad
catches on to web-based applications fairly quickly, so he seemed like a good
candidate for searching inside and outside of his areas of study.
Les is a Ph.D. student of Educational Psychology. He often uses academic
research engines, and is very comfortable in that environment. I thought his
opinions and reactions would inform my understanding of the processes using
these sites.
Plan for Testing
I asked each tester to review three sites, and search three academic terms
each for a total of nine searches for each tester. Each topic was chosen so that
each tester would be a subject matter expert in one area and not in two areas. For
instance, Chad is currently researching the “Yellow River Culture” of China for a
history project, so one of the search terms is the “Yellow River Culture” of China.
Cherie often researches “e coli” and other infectious diseases, so they were
included. Les deals with motor skills among preschoolers, so that was an item of
research. What is easy for the subject matter expert is new territory for the others,
so it balanced out over the project.
6
FDA issues
and diseases
Educational
Psychology
Chad’s
Area
Specialization
Cherie’s
Area
Specialization
Les’s
Area
Specialization
Les
Cherie
Chad
Yellow River
Civilization
Table 1. Users and Areas of Specialization
Keep in mind that I asked each tester to test three different issues, but that
they had to test them in three different research engines for a total of nine
searches. I tested it myself, and found that it could be done easily in one hour.
Test Description
After agreeing to the usability testing, the testers came to my house and
looked over three documents, which can be viewed in the Appendix section. I
explained the Pre and Post Tests, and then read through the explanation given in
the Usability Test for Free Access Academic Research Engines (Preece et al,
342).
After that, the testers were given a maximum of one hour to type in the
URL’s for the various sites and get going. They were instructed to start off by
browsing the site, then going to work on the three different areas of research. I sat
nearby and made general observations, asked questions along the way, and found
7
out that it helped to suggest that they think out loud while using the research
engines.
Figure 1. Testing Room Layout
More details of the testing description are located in Appendix B.
Pre-Test Questioning
The Pre-Test Questionnaire (See Appendix A) asks general questions
about the user’s preferences and experiences for online research. Interestingly,
their ages were different, but they were not too far apart in terms of how they get
their information through tools available on the web. Les is, by far, the more
advanced in doing research, but Chad and Cherie were not too far behind, having
at least some familiarity and experience with it.
8
Previous Academic Research
Finding Information Rate
Les
Technical Rate
Cherie
Chad
Length of Internet Use
Frequency of Internet Use
0
1
2
3
4
Table 2. Self-Rating on Level of Internet Usage for Research
Additional data gathering asked users about age and their primary method
of gathering information from the web. The oldest and youngest in the testing
seemed to alternate in being more proficient than the other in web-based research.
Les, landing in the middle of our age years, was consistently on top in this area.
When asked to write their favorite academic resources for research, both
Cherie and Chad stated that they go straight to regular Google first. Les does
most of his research from UT NetCat.
Test Procedure
Users accessed the Internet from my personal computer at my home.
Once they went through the orientation, they were pretty much on their own to
type in the URL’s and the search criteria.
Once the testing started, I realized the directions would be more useful if
they included some explanation of procedure. Each user asked whether they
should do all the searches at one research engine before moving to the next
engine, or researching one topic at all three engines before moving on to the next
9
topic. This is not a major issue, but for control, I advised them to start by
researching one topic at all three engines before moving on to the next topic.
While the users were researching, I was sitting behind them noting their
usage of the sites, occasionally asking why they did something, or what they look
for if they wanted to narrow or broaden a search. Sometimes, I would ask the
latter of these questions to get them to look for more specific criteria if they were
not trying that already.
At the end of the test, I asked each user to fill out the Post-Test
Questionnaire (Found in Appendix C).
Test Data
Test data was mainly gathered from two sources: user feedback and my
observations. User feedback came in the form of filling out the Post-Test
Questionnaire, and through their comments as they were taking the test. My
observations were made during the time of testing.
Raw Data from Post-Test
The Post-Test asks users to evaluate how easy it was to use each site to do
what they needed to do. They were asked to rate each research engine in terms of
ease of use for finding information within and outside of their area of subject
matter expertise. They also rated the sites in terms of satisfaction with the quality
and amount of data they found. They also answered questions regarding user
interaction.
After rating the research engines on a scale of 1-5, they ranked the three
sites on a scale of 1-3. These questions asked about the number and quality of the
results, but also asked a more general question that informs our research. “If you
10
were researching a topic, where would you start?” At first, I thought this was a
good question, because it helps to summarize their overall impression. But my
thinking changed when Les mentioned that he would start at Google to get an
overall picture, then spend most of his research time in ERIC.
Eric.ed.gov
46.2%
Google Scholar
92.4%
Academicindex
63.6%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Table 3. Overall score of Research Engines by three users
Overall, the data pointed to Google as the favorite site, but the preferences
broke apart among the individual users. Without a lot of academic research
experience with premium services, Cherie and Chad were unhappy with the
layout and interaction in ERIC, but Les thought it was great.
CHAD
Eric.ed.gov
CHERIE
Google Scholar
Academicindex
LES
0.0%
20.0%
40.0%
60.0%
80.0% 100.0%
11
Table 4. Individual Preferences for Research Engines
The rankings break down even further. Cherie and Chad seemed to rank
ERIC with a low score, but Les ranked it close to Google. Additionally, Les rated
Google lower than Cherie and Chad. Les had no problems with the interactivity
of ERIC, but both Cherie and Chad got lost. This may be because the user
interface is not familiar and all three testers know the Google interface.
There was a large discrepancy in terms of quality of data that turned up.
Both Cherie and Chad ranked ERIC on the bottom and Les ranked it on top.
Though we could write this off as, “They don’t know good data when they find
it,” I want to be careful to note a few items. For their purposes, the data they
found through Google Scholar was sufficient in terms of quality. Perhaps their
criteria will be different when Chad reaches higher requirements for resources in
his term papers and Cherie needs substantial authority to meet FDA requirements.
Other User Feedback
The following information was gathered from users as they were taking
the test:
Though all users would start with Google, they found that several of the
results were gathered from ERIC. Because it is a meta-search engine, Google
scholar would often turn up data that was in ERIC, or similar sites.
None of the users tried using Google’s “Advanced Scholar Search.”
Though it is more like regular Google “Advanced Search” than a premium
academic research engine, it offered some more criteria, most helpful in searched
that would turn up more information than needed.
12
Google also offered a suggested spelling replacing “e coli” for “ecoli.”
This may be an important feature as some terminology may be different or have
other options for spelling. As it was, all the search engines could turn up different
results depending on how you spell the word, but Google was the only research
engine that suggested a different spelling.
Google advanced allows users to narrow the search in terms of area of
study, but Les would like to see this limit the kind of publication in which the data
is posted. It’s possible that this would simply limit it to ERIC documents. Les’s
overall comment about Google Scholar is that it turns up everything that has any
relation to what you’re searching for, but does not have the capability to narrow
searches to what is usable for academic purposes.
Cherie liked the subdivision of topics offered by AcademicIndex, but did
not like the research engine overall. Chad and Les thought that the relation of the
data to search criteria was slightly more on target than Google Scholar. Les found
that their results were hard to read. For one of his searches, Les got thousands of
responses from Google and three targeted responses from AcademicIndex.
Les also complained about their use of a “Results Grouped by Phrase”
box. This box “attempts” to guess at how to sort your search results. The
algorithms are not very smart, and seem to organize it into unusable divisions.
13
Sometimes, all the contain all of the articles, making it a waste of page space.
With the image below, AcademicIndex was searching for the terms “FDA” and “e
coli.” The divisions offer nothing useful in the search.
Grouping Results in AcademicIndex was little or no help.
The only way to narrow the search in AcademicIndex was to choose an
area of study. What if you want to check in all areas? You could choose that
criteria from the opening page, but not once you started your search. As he used
it, Les figured out that users would likely start from a broad search, and would
narrow it later according to subject matter.
Les liked ERIC and thought that the descriptors worked well for what he
wanted to do. Though he found that he would sometimes turn up nothing because
the engine was too specific, he could try different arrangements of criteria to
produce some results in all of the searches. His favorite feature on ERIC was that
it produced related topics for research, which he could simply click on to pursue
another branch in the same field, or a term that is relevant that he had not thought
of earlier.
14
ERIC provided clickable links to similar topics
Google Scholar produced similar keywords, but they were not clickable.
The user would need to add the terms in their Google search.
Google Scholar had “Keywords” but were not clickable.
In addition to this feature, Les also liked the ERIC option to save the
results to a file that he would use later. “When doing a lot of research,” he
remarked, “I often want to save the reference to look up later.” This would save a
lot of time and effort, while not disrupting the search effort.
15
ERIC users can click the “Add” feature to build a list of references.
One more feature that stood out for ERIC, is that the users could turn up
articles directly from the search, or limit the search entirely to journal articles or
pdf files. From the search results, users can click directly to a pdf file or journa
article. Google also has a feature to limit searches to file types, but not the ability
to do both at once.
What one user loved, the others hated! Chad found it hard to navigate
during his first use of ERIC. When he couldn’t produce any results, he thought
there were no results to be found. I coached him to try setting different criteria,
which he was able to do, but only under my direction. He thought the options
were too complicated.
Search criteria can be intimidating for first time visitors to ERIC.
16
At first, Cherie felt like it was so difficult to use and produced so little
results that ERIC was useless. Similar to Chad, with guidance, she was able to
turn up some quality results. The effort in getting there, she felt, was not worth it.
General Observations
Average researchers want to get to their information quickly. They would
rather trust the search engine to give them the best information out of thousands
of returns in the first page. More advanced researchers are looking for limited
results of data that carries some weight. Some of the articles in ERIC were
flagged as “peer reviewed” depending on the publication. This is important to
people researching for a Ph.D., but not for someone writing a high school paper.
Cherie was visibly frustrated at several occasions. She didn’t like trying
to search outside her subject matter expertise, or using a format she wasn’t
familiar with.
Specific Observations
Google Scholar could be summarized as everyone’s starting point. They
may stay at Google, or move on, but all of the users would prefer to start there.
The search capacity was broad and users can get to results in a matter of seconds.
This is both an advantage and disadvantage, because though the results are
available, they are not always usable.
AcademicIndex narrowed the search somewhat, and added some search
criteria. Users could get directly into quality documents, but the results varied. It
especially dropped the ball in terms of having useful information on the page.
The layout and criteria of ERIC lends itself to people who do a lot of
academic research. The engine can produce high quality results and has the only
17
system of storing the findings without downloading everything the first pass
through. Though it was difficult for first-time users, our testers didn’t have the
time to explore the “search tips” that is offered to the right of the opening screen.
These search tips take them to another page which opens and gives all
kinds of tips and tutorials for using the search engine. Given more time, and
having heavier requirements on their resources, I think both Cherie and Chad
would find ERIC very useful.
Interpretation of Results
Google Scholar is everyone’s favorite place to start. This is no surprise, as
each of these users often goes to regular Google for information, sometimes on a
daily basis. The frustration with ERIC is understandable for first time users only
given an hour to do the whole testing. If there were a second round, I would ask
for less searches, and allow them ramp-up time for exploring how to use ERIC. It
seemed that AcademicIndex needed some revision of the additional items. It was
18
an attempt at something between a general metasearch engine like Google and a
specific academic research engine like ERIC. Either way, the broad observations
were helpful, and the users eventually got to their information.
Suggestions for Improvement
Google Scholar could improve by moving into more specific options for
criteria set by the researcher. Though the results were different from running the
same criteria through regular Google, the usefulness of the results was about the
same—thousands of documents, a few of which are useful. I would suggest and
option allowing choice between journal articles and web pages. This would
eliminate a lot of redundancy and clutter. They could also add two items from
ERIC, including suggested search criteria and description of the type of source
they are citing. If Google Scholar had an “add to clipboard” feature, it would be a
killer research machine!
It might be that Google and AcademicIndex are so automated that there is
little human involvement. It makes me want to find out to what degree people
have to staff ERIC, or if they have an automated system for logging data.
AcademicIndex could be improved by adding more intelligence to the
features that attempt to help the researcher. In most cases, these were useless, but
they could be in the areas they were set up for.
19
Lastly, I would ask ERIC to include tips and a tutorial, but they already
exist. Our users were not asked to ramp up on the engine, so we had to try it cold.
ERIC may be improved with a “quick search” feature. Their opening one line
search criteria, though, does not produce much results, so they may improve
usability with some boarder criteria.
20
Appendix A: PRE TEST QUESTIONAIRE
1. How old are you?
<18
18-25
25-32
33-40
41-47
2. How often do you use the
Internet for work or school related
research?
 Never
 A few times a year
 Once or twice a month
 More than once a week
3. How many years have you used
the Internet for research?
<1 year
1-2 years
2-5 years
>5 years
4. How would you rate your
technical ability in general?
 Not very technical
 Somewhat technical
 Technical
 Extremely technical
5. How would you rate your
abilities at finding the information
you need?
 Need assistance finding any
information
 Need assistance finding most
information
 Need assistance finding some
information
 Find almost all information
independently
6. Circle the option that best
describes your previous experience
with academic research:
 I don’t know what academic
research is
 I have used academic
research applications in the
past
 I frequently use academic
research applications
 I subscribe to academic
research applications
7. Name one of your favorite academic
resources or web sites:
21
Appendix B: Usability Test for Free Access Academic Research Engines
You are about to participate in a research project, which tests the usability of free academic
research engines. You will research three different topics at three different sites each for a total
of nine searches. One topic will be in an area of your subject matter expertise, and two will be
outside of your expertise.
Areas of research:
Psychology of Education Topic: Motor Skills for Preschoolers
Chinese History Topic: River Civilization
FDA Regulations Topic: Listeria and Ecoli
Sites
www.scholar.google.com
www.academicindex.net
www.eric.ed.gov
Your Checklist (make notes as needed)
River Civilization
Listeria and Ecoli
ERIC.gov
AcademicIndex
Google Scholar
Motor skills for
preschoolers
You will need to pay attention to the following:
 How easy was it to find your information?
 Once you find the information, could you access entire articles? Or did they just offer a
summary or abstract?
 Does the information you found seem useful?
 Which search engine is easier for you to use? Which one would you go back to the next
time you are doing research?
Thanks for your help in this project!
22
Appendix C: POST TEST
QUESTIONAIRE
Which of the following have you
used before today:
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
____ None
On a scale of 1-5 (1=lowest and
5=highest)…
Easy to know how to look for
information
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
Easy to find information within
your subject matter area
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
____Scholar.google.com
____ Eric.ed.gov
How would you rate the quality of
information you found?
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
How would you rate the overall use
of the following sites?
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
Rank the sites 1-3 (1=highest –
3=lowest)
Number of results
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
Easy to find information outside
your subject matter area
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
Quality of results
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
Were you satisfied with the amount
of information available for free?
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
If you were researching a topic,
where would you start? (Order the
sites from first to last)
____ Academicindex.net
____Scholar.google.com
____ Eric.ed.gov
Was it easy to get back to the
search area once you viewed the
results?
____ Academicindex.net
23
References
Blossom, John, “Evaluating The Academic Search Engine From Microsoft: Windows Live
Academic.”
http://www.masternewmedia.org/news/2006/04/17/the_academic_search_engine_from.htm
Krug, S. (2006). Don't Make Me Think: A Common Sense Approach to Web Usability.
Berkeley, CA, New Riders Publishing.
Norman, D. (2002). The Design of Everyday Things. New York, Basic Books.
Preece, J., Rogers, Y. & Sharp, H. (2002). Interface Design: Beyond Human-computer
Interaction. New York, John Wiley &Sons, Inc.
Shedroff, N. (1994). Information Interaction Design: A Unified Field Theory of Design.
1–15.
24
This paper was written by Gregg Orr for the course EDC385G Designs & Strategies for New Media at the
University of Texas – Austin.
25