Design and Strategies for New Media, Fall 2011
Usability Evaluation,
Research Component
Primary Source Educational Collections
Rob Scordino
11/20/2011
Contents
Introduction
3
Sites Evaluated
Presidential Timeline of the 20th Century
Presidential Recordings Program
Digital Classroom Initiative
Library of Congress Teachers’ Website
Target Users
3
4
5
6
Methodology
Participants
Questions
Tasks
6
7
7
Design
Outline
Logistics
Data Collection
8
9
9
Findings
Library of Congress
Presidential Recordings Project
Presidential Timeline
10
13
15
Discussion
Library of Congress
Presidential Recordings Project
Presidential Timeline
17
22
25
Conclusion
27
Appendices
A- Consent Form
B- Background Questionnaire
C- Exploratory Task Protocol
D- Specific and Open-ended Task Protocol
E- Post-evaluation Survey
30
31
32
33
34
2|Page
Introduction
When used in the history classroom, primary sources provide first-hand accounts of the
specific time period being studied. Using primary sources in the classroom does more
than provide mere supplement to textbook curriculum. The use of such sources can
promote historical thinking—a way of learning history through inquiry and investigation.
Historical thinking through analysis of primary sources engages students in their
learning and helps them develop critical thinking and research skills.
Access to primary source collections (from archives, libraries, and museums) has
greatly increased due to recent digitization efforts and improved Internet speed.
However, access alone does not guarantee teachers will be able to find and use the
sources they need. Large archive may not offer acceptable organization or user-friendly
interfaces for teachers to find content in their limited free time. To address this issue,
many organizations have made efforts to create themed primary source collections and
lesson plans for teachers to access online. This paper outlines a usability study of three
such websites.
Sites Evaluated
Presidential Timeline of the 20th Century
The first site decided upon was the Presidential Timeline Project
(www.presidentialtimeline.org) (Figure 1). This project (funded by the National
Endowment of the Humanities and the LBJ Presidential Library) is a collaborative
project between the Presidential Libraries. The project contains collections of digitized
documents from each of the 13 libraries. These collections are organized into learning
activities for teachers to use. The project is affiliated with the National Archive and
Records Administration (of which the presidential libraries are part), and links to
additional resources from the NARA.
3|Page
Figure 1. Presidential Timeline of the 20th Century
Using the Presidential Timeline as a model, criteria were established for the other two
sites to be chosen. These sites:
 Must contain collections of primary sources organized into educational objects
 Must be targeted toward teachers and students
 Must be free
 Must have not be connected to a political organization (nor have any obvious
bias)
A Google Search of “Primary Source Collections” brought up many hits. There were
many sites that contained good collections, but were not organized for easy use in the
classroom, and therefore did not fit my criteria. After some searching, two sites were
decided upon: The Miller Center’s Presidential Recordings Project
(http://www.whitehousetapes.net) and the Library of Congress Teacher Collections
(www.loc.gov/teachers).
Presidential Recordings Project, Digital Classroom Initiative
The Presidential Recordings Project (PRP) is part of the Miller Center- a non-partisan
Institute with the purpose of expanding the understanding of the president and policy.
The PRP contains links to presidential tapes and transcripts from those tapes from the
4|Page
FDR through the Nixon administrations. The Digital Classroom Initiative (DCI) was
designed to help teachers use these presidential recordings. The DCI grants educators
easy access to the PRP’s unique flash audio-transcript clips by grouping these, as well
as the more extensive “virtual exhibits,” along topical lines. It also provides links to
course syllabi and classroom activities featuring the tapes themselves.
Figure 2. Presidential Recordings Program
The Library of Congress Teacher’s Resource
The Library of Congress website provides access to digitized copies of many
documents, pictures. And other objects housed by the Library of Congress in
Washington, D.C. The Library of Congress Teachers’ site contains thematic collections
for use by educators. These collections are able to be searched by date, topic, or state
standard. In addition to the resources, the LOC Teachers site provides lesson plans for
teachers as well as links to other government resources for teaching through primary
sources.
5|Page
Figure 3. Library of Congress Teacher Website
Target Users
These sites may be used by anyone who is interested in history. However, the target
audience for these sites is classroom teachers. Specifically, teachers who will use
these sites are likely to be history teachers of secondary grades. Therefore,
assumptions can be made about the likely end user of these sites. These users will:




Have a college education
Have basic Internet and research skills
Be interested in the content
Be motivated to successfully use a site like the ones being tested
Methodology
Participants
Two participants were selected to perform a usability test of the three websites. The
participants were selected by the moderator through convenient sampling. Although
chosen from a convenient sample, the participants were adequate representations of
the probable end user of these websites. As a secondary history teacher, one
participant (Matthew) was—in fact-- an exact representation of an end user. The other
6|Page
participant (Kristin) is a science teacher. Although she is not a history teacher, she is
familiar with teaching secondary students, and was therefore acceptable as a
participant.
The participants’ interests and background information was collected through pre-test
questionnaire. The information is as follows:
Kristin W:
- early 30s, female
- Employment: Middle School Teacher (Science), 4th year
- Education: B.S. Biology, Some Grad School
- Computer experience: Medium. Uses Internet frequently for browsing, email, finding
sources for lesson plans. She has never used sites for primary sources, but has used
teacher websites for science education.
- Content interest: She expressed interest in using more technology in her teaching,
but admits that she hasn’t been able to do it to the extent to which she would like. She
is not an expert in history or in history education, and is not likely to visit these sites on
her own.
Matthew S.
- late 20s, male
- Employment: High School Teacher (Social Studies), 5th year
- Education: B.A. History, Master’s Degree
- Computer experience: Medium to High. Uses Internet frequently for browsing, email,
lesson planning. He has used the Library of Congress site and similar sites to find
primary sources for lesson plans.
- Content Interest: As a history teacher with a content background, he has a high
interest in both history and teaching with primary sources. He would (and does) visit
websites like those being tested.
Questions
The purpose of this usability evaluation is to investigate the websites’ ability to address
the needs of the likely end user and to determine design attributes that contribute to the
usability of these sites. Therefore, two questions guided this study:


To what extent do these three websites meet the usability needs of teachers?
What characteristics of information, interface, and interaction design contribute to
the usability of these sites?
Tasks
The participants of this study were given a series of tasks, which are described in the
7|Page
Design section of this paper. Each of these tasks will fit in one of three types of tasks
as defined by Nielsen, et al. (2000) (exploratory, specific, or open ended). The purpose
of each task is to mimic the probable use of an end user. The likely end user of these
sites are classroom teachers, thus, the tasks will mimic the likely actions of teachers:



Explore the site and give impressions (exploratory)
Find a document (specific)
Plan a lesson (open-ended)
Design
A single usability evaluation of all three websites was run in two individual participant
sessions. The design of the usability evaluations were greatly influenced by Rubin and
Chisnell’s (2008) Handbook of Usability Testing. In each session, the participants
evaluated all three websites. The evaluation session consisted of (1) participant
background questionnaire and orientation (2) performance evaluation of each website,
(3) post-evaluation questionnaire, and (4) participant debriefing.
Background questionnaire and orientation
The participants received a short, verbal introduction and orientation to the evaluation.
During this time the moderator explained to them the purpose and objective of the
evaluation, and additional information about what is expected of them. Participants
were assured that the websites were the center of the evaluation and not themselves,
and that they should perform in whatever manner is typical and comfortable for them.
The participants were told that they will be assigned several tasks, and the moderator
will observe and take notes as they complete those tasks. They were also informed that
screencasts may be taken during the evaluation. After this was explained, the
participants filled out a consent form (Appendix A) as well as a short questionnaire
(Appendix B) about relevant aspects of their background.
Performance evaluation
After the orientation, the participants sat at the evaluation computer. Each website that
was to be evaluated was preloaded on separate browser windows. Each site was
evaluated using the same tasks.
The evaluation tasks consisted of the following:
1.
2.
3.
4.
Browse site and give initial thoughts
Find information for teachers about teaching with primary sources
Find an image of any president
Search for a specific file (see Figure 4)
8|Page
5. Using the available tools, investigate a topic for use for a lesson plan.
Task
#4
Library of Congress
Find Ben Franklin’s
“Join or Die” political
cartoon from the
Revolutionary Era
Presidential Recordings
Find the recording of the
phone conversation
between Lyndon
Johnson and Martin
Luther King, Jr.
Presidential Timeline
Find Ronald Reagan’s
copy of the speech
following the Challenger
disaster
Post-evaluation questionnaire
After each website is evaluated, Participants were asked to fill out a post-evaluation
survey. The survey was designed to make the participants rate aspects of information
design, interface design, and interaction design of the site, as well as the participant’s
overall satisfaction with each site. These questions will ask the participant to evaluate
aspect of these four elements using a Lykert scale of 1 to 5. Participants will also be
given the chance to make suggestions to improve the usability of each website.
Participant debriefing
A debriefing session took place after the participant was finished evaluating all three
sites. Any further questions the administrator had following the evaluation were asked
during this time. After the debriefing session, the participants were thanked for their
efforts, and released.
Logistics
Each participant was tested separately and evaluated all three sites in the same
individual session. Each participant performed their test in the in the environment
where they most often planned lessons. Evaluating the usability of a product at a user’s
normal place of work is advantageous in that it evaluates the product being used in
realistic settings. Evaluating in context is also easier on the user, because they have to
schedule less time than if they were to travel to a testing site, and they may be more
comfortable working in a familiar environment (Rubin and Chisnell, 2008, pp. 100). In
each evaluation, the testing environment most closely resembled the “simple singleroom setup” (see Figure 4) from Rubin and Chisnel’s Handbook.
9|Page
Figure 4. Simple single-room setup.
The evaluation was done using a 17 inch Macintosh Powerbook with a dual-core 2.66
Ghz processor, 4 GB of RAM and connected to the Internet via high speed wi-fi. This
configuration exceeds the requirements needed to use each of the sites
tested. Participants were given the choice of using OSX Snow Leopard or Microsoft
Windows 7, 64-bit. The participants were also given the choice of using their preferred
browser: Mozilla Firefox, Google Chrome, Apple Safari, or Mircrosoft Internet Explorer.
Both participants used OSX with the Firefox browser, as this was the configuration each
was most familiar.
Neither of the participants required additional supports to meet accessibility needs. In a
larger-scale usability test, it would be important to involve participants with varying
abilities to address accessibility design issues as well. In such cases, modifications
would have to be done to the current configuration (such as implementation of screen
readers, screen magnification, modified keyboards, etc.) to properly evaluate the
usability of these websites.
Data Collection
Data sources:
 Background questionnaire
 Moderator notes from observation
 Participant evaluation
 Post evaluation questionnaire
Quantifiable data collected from the evaluation included:




The amount of time to complete each task
The number of attempts to complete each task
The number of errors from which the user could not recover
Participants ratings of certain aspects of the website
10 | P a g e
Qualitative data to be collected during observation included:




Subjective opinions of the usability and aesthetics of the product expressed by
the participants
Indications of frustration or joy from the participant
Participants’ answers to open-ended questions
Participants’ suggestions for improvement
Findings
Findings from the participant evaluation of the three sites are discussed in the following
section. Descriptions are given of participants’ experiences completing each of the five
tasks. Measurements of time and mouse clicks needed for completion and number of
critical errors were taken for tasks two, three, and four. These measurements are also
given in this section.
Library of Congress
Task One
Both participants reported using the Library of Congress website in the past. Matthew
reported using content from the Library of Congress (“mostly pictures”) in his classroom.
Both participants noted the large amount of information on the LOC Website. Because
both participants were at least moderately familiar with the Library of Congress site,
they were also familiar with the purpose and target audience. While browsing, both
participants seemed pleased with the content that was available.
Kristin, who had never visited the Teachers’ site, first clicked on the “More Classroom
Materials” button. When I asked why she clicked there, she said “because it was what I
saw first.” This brought her to a page of lesson plan topics. She scrolled down through
the Lesson Plans, which were organized “By Topic.” She then stopped at the heading
“Science and Technology.” She said “Maybe I could use this,” referring to the links to
the science-related lessons. After checking a couple of lessons topics (“Natural
Disasters,” “Thomas Edison”), she seem less interested (The lessons were very history
heavy). Although she may not have been able to find something she could use for her
class, she did say it was “cool that you can search through all these lesson plans
already made for you.”
Although Matthew was familiar with the Teachers’ LOC Website, he admitted that it had
“been a while” since he last used it. (He said He used the lesson plans from the site
often in his first year teaching.) Like Kristin, he immediately interacted with the
11 | P a g e
navigation tool featured on the first page. He scrolled through the featured links for a
short while before abandoning it and moving on to the rest of the page. He used the
side navigation to go to the Classroom Materials page. Like Kristin, Matthew also
browsed the available lesson plans before returning back to the homepage. When
asked about his impressions of the site, he said “There is a lot of stuff on there for
teachers—you could probably plan a whole year using [the Library of Congress]. But, if
I was just browsing the site without some sort of goal, I would probably just waste a
bunch of time looking at pictures.”
Task Two
Measure
Time
Clicks
Errors
Score
(KW)
< 10 sec
1
0
Score
(MS)
< 10 sec
1
0
The LOC Teachers’ site has extensive information available for teachers who want to
use primary sources in their classrooms. Much of the homepage is devoted to such
information, including links to professional development modules on using the Library of
Congress and Teaching with primary sources. Both participants quickly found the
“Using Primary Sources” option on the navigation menu on the left side of the page.
Task Three
Measure
Time
Clicks
Errors
Score
(KW)
15 sec
3
NA
Score
(MS)
12 sec
3
NA
It did not take long for either participant to find a picture of a president on the LOC site.
Matthew-- who had looked at a collection of Abraham Lincoln-related items while he
was browsing—simply retraced his steps to find a portrait of Lincoln. Kristin again used
the search function to search for “Kennedy.” From the results, she was able to find the
“President’s Portraits collection and link to a picture of John F. Kennedy.
Task Four
Measure
Time
Clicks
Errors
Score
(KW)
11 sec
2
0
Score
(MS)
14 sec
2
0
12 | P a g e
When assigned the task of finding the “Join or Die” political cartoon, both Kristin and
Matthew used the search box to search for “join or die.” The search resulted in multiple
links. Kristin chose the first link, which brought her to a PDF image of the cartoon.
Unlike Kristin, Matthew did not choose the first link. Instead, he scrolled through a
couple of choices until finding a link to an image of the newspaper on which the cartoon
was first published.
Task Five
One of the main purposes of the Library of Congress Teachers’ is to provide lesson
plans using documents from the Library of Congress. These lessons can be accessed
through multiple means, making them easy to find. Also, participants became familiar
with the site while browsing, and both had already browsed the collection of lessons.
Because of this, neither participant struggled with finding the lesson plans. After
choosing a lesson plan topic, both participants were able to link to all necessary
materials.
Presidential Recordings Program
Task One
This was the first time either participant visited the Presidential Recordings Program
Website. Both participants recognized that the purpose of this site was to provide
teachers with materials for their classroom. Kristin also thought that “people
researching history” might also be a target audience of this site. Both participants noted
the simplicity of the layout—Matthew referred to it as “simple,” and Kristin referred to it
as “plain” and “kind of dull.”
Kristin used the left navigation menu to check the content of the site. She clicked on
each menu item to scan what was available on each page. After scanning the site, she
clicked on the menu item Topics and said “I guess this is where you would go for
information for lessons.” After continuing in this direction, however, Krisitin was brought
out of the teacher section into various Virtual Exhibits. I asked her if she noticed that
she was no longer in the teacher section and she said she didn’t. I asked her to try to
get back to the teacher section, and doing so made her realize that the left navigation
had changed, so she had to use the browser’s back button.
Matthew did not scan the entire site while browsing. Instead, he immediately clicked on
the Classroom item from the left navigation menu. This took him to a page with some
lesson plan ideas. After looking at this page for a while, he said “These topics are only
from the 60s. I thought there would be more.” After browsing a bit more, he decided, “I
13 | P a g e
guess I could get some stuff for lessons here. I would probably use the recordings [of
the phone conversations]. I think it’s good to make [historic figures] seem more
human.”
Task Two
Measure
Time
Clicks
Errors
Score
(KW)
N/A
NA
NA
Score
(MS)
NA
NA
NA
It is difficult to find information about teaching with primary sources by using the
Presidential Recordings Project website. A link to this information is not embedded in
an obvious section of the website. Instead, such links are only available on lesson plan
pages. Because of this, both participants struggled with the second task. Kristin spent
enough time on the task to conclude “it isn’t there, is it?” and gave up. Matthew was
more determined to find the information, but was eventually stopped by the moderator
to move on to the next task.
Task Three
Measure
Time
Clicks
Errors
Score
(KW)
15 sec
3
0
Score
(MS)
28 sec
6
0
Although the presidency is the sole topic of the website, there are not many images of
presidents available on the PRP teachers’ site. Despite this, both participants were able
to find a picture of a president with relative ease. Kristin immediate went to the search
bar in the top right corner of the page, where she searched for “Kennedy.” The search
resulted with a page of links to parts of the site in which the word “Kennedy” appeared.
She was then able to click on a link labeled Resources which brought her to links of
pictures of JFK. Matthew also used the search function, however, he only used it (he
searched for “LBJ”) after he was unable to find a picture by browsing the site.
Task Four
Measure
Time
Clicks
Errors
Score
(KW)
27 sec
3
0
Score
(MS)
25 sec
3
0
14 | P a g e
When asked to find the recording of the phone conversation between Lyndon Johnson
and Martin Luther King, Jr., both participants once again made use of the search
function in the upper right hand corner. Kristin searched for “martin luther king phone”
and Matthew searched for “MLK.” Both searches came up with similar results, from
which both participants could find the recorded conversation.
Task Five
The final task required participants to find enough content to form a lesson plan. By
then, the participants seemed familiar with the website, and neither had any difficulty
finding actual lesson plans on the site. However, both encountered issues with finding
the materials needed to implement those lessons. Kristin was visibly frustrated during
her attempts to find the materials. Ultimately, she was stopped by the moderator and
reassured that her inability to complete the task was due to poor design of the website.
Matthew also became frustrated while trying to find materials. He stopped his attempt
on his own and said, “Realistically, if I wanted to use this lesson, I would probably just
Google the materials.”
Time, clicks, and errors were not tallied for task five. Due to the inability or refusal of
the participants to complete the task, both instances were considered failed attempts.
Presidential Timeline
This was the first time either participant visited the Presidential Timeline website. Both
participants identified the PTL as a site for teachers, and Matthew also identified
students as a potential target audience. Both participants expressed their opinion of the
visual appeal of the PTL over the other two sites.
After spending a few moment looking over the homepage, Kristin clicked a picture link
to open the Interactive Timeline,” which opening in another browser window. While
exploring the Timeline, she remarked, “this is neat, but what’s the point, exactly?”
Eventually, she clicked on the Educators link (which opened in a new tab of the original
browser window). “Oh,” she said, “this is the stuff for teachers.”
Unlike Kristin, Matthew immediately found the Educators page. From here, he linked to
the Classroom Activities page. “These look like lessons—pretty neat.” He said as he
scrolled through the activities page. He then investigated a couple of lesson topics.
“Are these WebQuests?” he asked. After realizing that the lessons were WebQuests,
he stated that he could “very easily” use the materials on the Presidential Timeline in his
class.
15 | P a g e
Task Two
Measure
Time
Clicks
Errors
Score
(KW)
13 sec
3
0
Score
(MS)
22 sec
5
0
Kristin came across the Teacher Resources page during her browsing, so she easily
found the information on teaching with primary sources. Matthew took a bit longer to
find the information. He initially went to the homepage, expecting there to be a link to
the information he needed. After realizing it wasn’t there, he went back to the
Educators page. He then read the description of the Teacher Resources link, which led
him to the information he needed.
Task Three
Measure
Time
Clicks
Errors
Score
(KW)
<5sec.
1
0
Score
(MS)
<5sec.
1
0
The majority of the pictures on The Presidential Timeline are of presidents themselves.
Both participants immediately found such an image to complete this task.
Task Four
Measure
Time
Clicks
Errors
Score
(KW)
23 sec
8
0
Score
(MS)
32 sec
13
0
Participants encountered more trouble when trying to find the copy of Ronald Reagan’s
speech. Kristin used the Interactive Timeline’s Exhibit feature to find the speech. She
had to browse around a short while before finding the section containing the document.
Matthew, who started this task from the Educators page, instead searched for the image
by browsing the Educational Activities. He scrolled down until he came to the lesson
regarding Reagan’s speech. He then navigated through the lesson page until finding
the scanned copy of the speech. Although this was not as efficient as Kristin’s method,
it was equally effective.
Task Five
16 | P a g e
Like the LOC page, a major purpose of the Presidential Timeline is to provide teachers
with lesson plans and materials. By the time the participants attempted this task, they
were not only familiar with the site, but familiar with the evaluation process as well.
Therefore, they had no trouble finding lesson plans on the PTL site. As with the Library
of Congress Page, the Presidential Timeline provided all necessary materials for
executing each lesson plan. These materials are part of the PTL and were easily found
by both participants.
Discussion
In this section, results of the post-evaluation survey are given. Using the information
gathered from participant observation, the survey scores are contextualized. Each site
is analyzed in terms of participant satisfaction, information design, interface design, and
interaction design.
Library of Congress
Design
Element
Information
Interface
Interaction
Satisfaction
TOTAL
Score
(KW)
19
19
18
14
70
Score
(MS)
22
19
19
19
79
Total
Possible
25
25
20
20
90
Overall Satisfaction
Both participants were generally satisfied with the Library of Congress Teachers’
website. Matthew-- who had more interest in the content of the LOC-- also reported
enjoying the website. As might be predicted Kristin (a science teacher) reported that it
was not likely that she would use this site for her teaching. She did, however, report
that she would likely recommend this site to a colleague. Matthew had already
mentioned that he has used the LOC site in his teaching, and also reported that he
would recommend the site to colleagues.
Information design
Overall, the information design of the Library of Congress Teachers’ site was rated fairly
high by both participants. Both reported finding information easily, which is likely due to
good information design. A search box and navigation menu is displayed in the same
place on every page of the LOC Teachers’ site. The consistency of layout allows the
user to easily access these search tools. Additionally, important information was often
able to be found through a variety of ways. For example, when searching for
17 | P a g e
information on teaching with primary sources participants could see several sections of
the homepage devoted to that information (Figure 5).
Figure 5. Multiple ways to access information
Both participants agreed that there was a lot of information available on the website.
Matthew, a history teacher, found this abundance to be beneficial. Kristin, however,
thought that the information was overwhelming and considered the large amount of
information a design flaw.
Kristin also rated the organization of the information lower than did Matthew. While
completing the performance evaluation, Kristin mentioned that the choice of topics of
the available collections and lessons seemed “a little random.” The lesson plans
available from the site come from teachers, who have varied interests. Therefore,
Kristin’s critique of the seemingly arbitrary choices does have merit. Matthew, on the
other hand, gave a high rating to the organization of information on the site. During his
evaluation, he mentioned the benefit of primary sources being organized by theme and
18 | P a g e
by time period. He also said that he liked the organization of lesson plans, in that he
was able to search through them by topic, by era, or by alphabetical order.
Figure 6. Multiple sorting options
Interface design
Like the information design, the interface design of the LOC website received good
ratings. Almost all of the elements rated were given 4 out of 5 by both participants.
This suggests that although the interface was nothing spectacular, it highly usable and
moderately appealing. However, both participants noted the text-heavy appearance of
many pages.
Despite the large amount of content, both participants finished each task very quickly.
Navigation elements of the interface allow multiple ways to find information. Users can
choose from the navigation menu on the left, the search box on the left, the
breadcrumbs on top, or various links throughout the page. The well-designed navigation
also allows the user to freely explore the site while maintaining an escape hatch to find
their way back.
19 | P a g e
Figure 7. Navigation options
Two search boxes appear on the pages of this site. The box in the upper right corner is
for searching the entire Library of Congress website. The box over the navigation menu
on the right is for searching the Teachers’ site (Figure 8). Although neither participant
struggled with this, it may be helpful for designers to make the distinction between the
search boxes more clear. If the user searches with the box in the upper right corner, he
or she will find relevant content, but they will have left the teachers’ site, and might have
trouble returning.
Figure 8. Multiple search boxes
20 | P a g e
Interaction design
The interaction design of the LOC was rated very highly by both participants. The LOC
does not have extensive interactive elements. However, this site is in many ways offers
more interaction than the other two. The search functions of this site work very well and
provide a number of options to meet the user’s needs. As explained earlier, a teacher
can search for lessons or collections by theme, by era, or by alphabetical order.
Teachers may also search for content by state standard and grade level.
The Library of Congress also offers the ability to manipulate some content. High
resolution images can be viewed with tools that allow users to zoom in and move the
objects. Users are also given the option of different sizes and resolution of images to
download.
Finally, although not brought up in the evaluation, the Library of Congress Teachers’
page is the only website of the three evaluated to incorporate social media elements.
There is an available RSS feed for the site and a blog. Visitors can sign up to make
comments on parts of the site. Most of the pages also have a “Share” icon that allows
users to post links to the social networking tool of their choice. (Figure 9) This option
also allows users to save links to return to later. This is especially helpful for a contentrich website such as the LOC.
Figure 9. Use of social media.
21 | P a g e
Presidential Recordings Program
Design
Element
Score (KW)
Score (MS)
Total
Possible
Information
14
15
25
Interface
Interaction
Satisfaction
TOTAL
19
10
6
52
18
13
12
60
25
20
20
90
Overall Satisfaction
The Presidential Recording Project’s Teachers’ site was the lowest rated of the three
evaluated. Responses from both surveys showed that the PRP was perceived to be the
least desirable in each of the four elements evaluated. Satisfaction with the site was
low, especially from Kristin. Neither participant said it was likely that they would
recommend this site. Matthew said he might use some content from the site because
he found value in the actual recordings. However, he said that he would create his own
lessons instead of using those available on the PRP site.
Information design
Of the five criteria for information design evaluated by the participants, predictability and
uniformity of pages was rated the lowest. Although the pages within the teacher site are
generally uniform, many links lead to other parts of the site (such as “Virtual Exhibits”)
or external links. When linked away from the teacher site, the user encounters a
completely different page design, with different color schemes and different information
layouts. This can confuse the user, who may not notice that they have navigated to a
different site. This problem was exemplified when Kristin was initially exploring the site.
Within a few clicks from the homepage, she found herself in a Virtual Exhibit (a nonworking one, at that) (Figure 10). Although this exhibit was part of the main site
(whitehousetapes.net), it was outside of the teachers’ section and looked much
different. If this site were to be redesigned, special attention should be given to making
the site more consistent.
22 | P a g e
Figure 10. Non-working and inconsistently designed page
Availability and organization of information were also given a poor rating by both
participants. Neither was able to find general information for teaching with primary
sources. This is because the website has no such information readily available. The
only way to access this information from the website is if a user happens to stumble
upon a link embedded in several lesson plan pages.
This site also lacks in availability of content. The PRP itself contains recordings from
Presidents Roosevelt to Nixon, spanning over 30 years. The lesson plans, however,
are from a much more limited timeframe. While Matthew was browsing he remarked: “I
thought there would be more stuff. It looks like they only have lessons about LBJ.”
When the lessons plans were explored for task five, both participants encountered
trouble finding the materials needed to complete the lessons. This was due to poor
organization of information. There are links to materials on the lesson plan pages.
Instead of linking directly to the file or page, these links often lead to the PRP’s main
page, requiring the user to search through the site to find the required materials.
Furthermore, many links simply don’t work. Both participants encountered broken links
when browsing for files. Matthew also encountered at least one link that was
inaccessible without a password (Figure 11). The content of this site needs to be
reorganized to minimize external links and store all needed materials in a single area
that is more easily accessible.
23 | P a g e
Figure 11 . Red paper clip icon signifying a password-protected file
Interface design
Participants were slightly more generous when evaluating the interface design of the
website. Three of the five elements evaluated involved visual aspects of the pages.
Both participants rated these elements fairly high. The PRP site is predominantly text.
Although this is not visually stimulating, the text on this site is clear and easily read.
Most pages are not cluttered with text, and there is good balance between text and
images.
The interface navigation was not rated as highly. While on the teachers’ site, users can
navigate through several means. First, the navigation menu on the left… The user may
also navigate by using the breadcrumbs on all pages. Finally, the user may use the
search bar that is always in the top right hand corner (Figure 12). Consistency such as
this helps the users “know where they are and how to find things” (Tidwell, 2010, p.
142). Where this site falls short is the high likelihood of the user wandering off the
teacher website. When a user browses out of the teachers’ area, pages are no longer
consistent and the three navigation options are no longer available. The user must rely
on the browser’s back button when faced with this situation.
Figure 12. Predictable navigation options
24 | P a g e
Interaction design
Interaction design was also rated poorly by participants. Search functions worked;
however, the links found were often incorrect and would link participants to places they
did not expect. There are little opportunities for interaction with the Presidential
Recording Program website. Unlike the Library of Congress, objects from the PRP
cannot be manipulated. There are also no options for customizability, communication, or
user input.
Presidential Timeline
Design
Element
Information
Interface
Interaction
Satisfaction
TOTAL
Score
(KW)
21
23
14
14
72
Score
(MS)
23
24
17
20
84
Total
Possible
25
25
20
20
90
Overall satisfaction
The Presidential Timeline received the highest rating for overall satisfaction out of the
three sites. Both were highly satisfied with the site and reported enjoying using the
site. As with the other two sites, Kristin said she was unlikely to use this in her
classroom. However, she rated the PTL as the site she would most likely recommend
to a colleague. Matthew (who gave a perfect rating for overall satisfaction) said he
would use this site in his classroom and recommend it.
Information design
Information design also received a very high rating. Both participants reported the
ability to find information quickly while completing their tasks. Most of the information
comes from internal sources except for Teacher Resources (such as information about
using primary sources) are linked externally. However, these mainly link to the National
Archives website (http://www.archives.gov) and are very reliable.
The only search function available is on the Interactive Timeline, which was not used
for most of this evaluation. That participants were able to quickly find information
without the use of a search function speaks a great deal to the organization of
information on the PTL. In fact, neither participant seemed to notice the lack of search
box, and instead relied on browsing to find all information.
The PTL is the only site that received a high rating from both participants for the
amount of content available. This shows that the Presidential Timeline has plentiful
25 | P a g e
content, but not so much that it is overwhelming. The content of the PTL is from the
presidential libraries, so content is limited to the 20th Century. This focus makes the
content more manageable to users like Kristin, who though the content from the Library
of Congress could be overwhelming.
Interface design
Out of the three sites, the Presidential Timeline was given the highest rating for
interface design by both participants. The contrast and text make the site easy to read,
and there is very little clutter, as text is used sparingly. Out of the three sites, the PTL
has the least amount of content on any given page. Colors schemes and fonts are also
consistent throughout the site.
Figure 7. Minimal text
The Flash-based Interactive Timeline is much different than the interfaces of the other
sites. This interface was appreciated by the participants, who both mentioned its visual
appeal. Although this interface may be no easier navigated than a HTML webpage,
effects such as roll-overs and animations make it more inviting for users to explore.
26 | P a g e
Figure 7. Flash-based interface
Interaction design
Unlike the other elements rated, the interaction design of the PTL received only midlevel ratings. This may be somewhat surprising, considering the site has a major
feature called the “Interactive Timeline.” However, this evaluation focused mostly on
the Educator section of the site. This section offers little in terms of user interaction,
such as customizability, input, or object manipulation. The pages in this section are
essentially set up in lists. As mentioned earlier, there is no search function for the
content available on these pages, so a user only has the option of scrolling through the
list of available content. A search box would be a helpful addition to the site, especially if
more content is added
Conclusion
For any website to be successful, it needs to meet the needs of its users. These
websites are designed for a specific user-type who has a specific set of needs. This
evaluation investigated the extent to which the websites met these needs. By having
teachers evaluate teacher resource sites, some characteristics of good design were
determined.
27 | P a g e
Information design
 Provide ample content without overwhelming
 Categorize content so it is easy to find
 Maintain consistent organization patterns
 Provide multiple ways of sorting content
 Minimize reliance on external links
Interface design
 Use clear text and avoid clutter
 Balance images and text
 Provide simple navigation
 Keep page layouts consistent
Interaction design
 Design website to allow multiple ways of use
 Put high priority on search functions
 Allow for easy downloading
28 | P a g e
Summary
This paper outlines the design and results of a usability test of three websites that
provide teachers with content needed for teaching with primary sources. The websites
evaluated were The Library of Congress Teachers’ website, the Miller Center’s
Presidential Recording Project Digital Classroom Initiative, and the Presidential Timeline
of the 20th Century. The usability evaluation consisted of performance evaluations and
post-evaluation surveys using two participants representative of probable end-users of
the websites. Results are discussed in the perspective of information, interaction and
interface design. Some recommendations based on the usability test results are also
made for each website. In conclusion, critical considerations for designing primary
source educational websites are proposed.
Related Links
http://www.presidentialtimeline.org/
http://www.loc.gov/teachers/
http://www.whitehousetapes.net/content/classroom
References
Nielsen, J., Snyder, C., Molich, R., and Farrell, S. (2000). E-Commerce User
Experience: Methodology. Nielsen Norman Group. [Online] Available
http://www.nngroup.com/reports/ecommerce
Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct
Effective Tests. Indianapolis, IN: John and Wiley & Sons.
Tidwell, J. (2011). Designing Interfaces. Sebastopol, CA: O’ Reilly Media.
Author information
This paper was written by Rob Scordino, a doctoral student in the Instructional
Technology Program at The University of Texas, Austin. This paper was written as part
of the final project for the course Design and Strategies for New Media, Fall, 2011
taught by Professor M. Liu
29 | P a g e
APPENDEX A- Consent Form
Participant name:
Understanding Your Participation
You have been asked to participate in a usability study about how websites meet your
needs. Our purpose in conducting this study is to understand what makes some
websites easier to use than others. In the session, we’ll ask you to visit some websites
and look for various things. Please keep in mind that this is a test of websites; we are
not testing you. We may videotape all or some of the test for evaluation purposes but
we will not release the recording to anybody. All information we collect concerning your
participation in the session is confidential.
Participation is voluntary. There are no physical or psychological risks associated with
participating in this study.
During the session, the study administrator will assist you and answer any questions.
You may take short breaks as needed and may withdraw from this evaluation at any
time.
If you have any questions, you may ask now or at any time during the test.
Statement of Informed Consent
I have read the description of the study and I am aware of my rights as a participant.
The conductor of the research study has assured me that my identity will remain
anonymous and confidential. I agree to participate in the study.
Signature _______________________________
Date ___________________________________
30 | P a g e
APPENDEX B - Pre-evaluation Questionnaire
1. Have you ever visited sites similar to or including the Presidential Timeline, Library of
Congress, or Miller Center Websites?
[ Yes ] [ No ] [ Unsure ]
2. If so, which sites?
3. If you have visited any of these, what was your purpose for visiting?
[ ] Primary sources (pictures, documents, videos, etc.) for lesson planning
[ ] Educational resources (teacher aids, lesson plans, study guides)
[ ] Personal research interests
[ ] Online Exhibits
[ ] Other Information (please specify) : __________________________________
4. How would you rate yourself as an Internet user?
[Beginner] 1
2
3
4
5
[Expert]
5. What do you use the Internet for?
6. How often do you use the Internet for teaching or planning?
[Never]
1
2
3
4
5
[Daily]
7. In a sentence or two, describe your use of the Internet for teaching.
In which which age group are you?
[ under 26] [ 26-30] [31-35] [36-40] [41-45] [46-50] [51-55] [55+]
8. Please check all that apply to you:
[ ] K-4 Educator
[ ] 5-8 Educator
[ ] 9-12 Educator
[ ] Post-secondary Educator
[ ] Teacher Educator
[ ] Other ____________________________________
9. If you are a teacher, how many years have you been teaching?
What subject do you currently teach?
31 | P a g e
APPENDIX C - Exploratory Task Protocol
Moderator:
This is an exploratory exercise. Take a few minutes to browse the site.
If you feel comfortable doing so, please verbalize your thought process as you browse
and tell me your impressions of the site. Please be honest, your objective opinion will
only support the purpose of this study.
Prompts 1. What do you think is the purpose of this site?
2. Who do you think is its target audience?
3. If this is the first visit of this web site, what would you say about this site?
4. At a first glance, do you think this web site could provide you with any
information of your interest?
32 | P a g e
APPENDIX D - Specific and open-ended task protocol
(The scenario will be the same for each site. The tasks will be similar, but the specific
items to be looked for will be different based on the content of each site)
Scenario: You are a secondary history teacher creating a lesson. For your
lesson, you want to use primary sources from the site you are about to evaluate.
From the main page, find information about using primary sources in the
classroom.
You need an image of a president for a presentation. Find any image of any
president well as information about that image and its context.
You are preparing a lesson on (topic). Find (specific file related to that topic).
Also, find another artifact (document, picture, phone conversation) related
(topic).
You need to plan a lesson. Choose a topic related to the information available
from (website). Find some documents, videos, transcripts or other files and
some contextual information about that topic that you could use to make a
lesson.
33 | P a g e
APPENDIX E Post-evaluation questionnaire
(given after each site evaluation)
To what extent do you agree with the following?
1.It was easy to find the information I needed.
2.There was adequate material available on the site.
(Please note if you thought there was too little or too much)
1
1
2
2
3
3
4
4
5
5
3.The material was neatly organized.
4.All labels and captions were accurate.
5.Pages were organized uniformly and predictably
6.While browsing, I could easily return to the home page.
7.The icons and navigational tools were clear and worked well.
8.The text on the website was easy to read.
9.The pages on the site were tidy (not cluttered).
10.Text and graphics were well-balanced.
11.Search functions and other forms worked well.
12.This site offers customizability.
13.A user could use this site in multiple ways.
14.I could manipulate (zoom, change, download) files from the site.
15.I liked browsing this site.
16.I am satisfied with this site.
17.I would use this site in my teaching.
[ ] If you do not teach social studies, check here.
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
18.I would recommend this site to a colleague or friend interested in
history.
1
2
3
4
5
19.If you are not satisfied, please describe the reasons for your dissatisfaction below.
20. What changes would you like to be seen to make the website easier to use?
34 | P a g e