: F i
“Software Development
Company”
“Product Name”
Usability Test
: :
F i i n a l l R e p o r t
Randolph G. Bias, Ph.D.
Associate Professor
University of Texas at Austin
School of Information
1 University Station, D7000
Austin, TX 78712-1276 rbias@ischool.utexas.edu
huangsc@mail.utexas.edu
Last updated:
December 13, 2007
© Copyright 2007, “Software Development Company”, Inc.
“Software Development Company” Confidential
© Copyright 2007, “Software Development Company”, Inc 2
1. Executive Summary
The “Software Development Company” User Experience team contracted with the
University of Texas at Austin School of Information to conduct a usability evaluation of the “product” (code name “Code Name”). A total of 16 people (15 current “Software
Development Company” customers one webmaster from the University of Texas) each carried out 11 tasks with the new “CODE NAME” user interface. The participants were asked to “think aloud” as they carried out their tasks, and they were observed and timed as they did. Performance data (e.g., time on task), satisfaction data (via a questionnaire), and particular usability problem areas were gathered and are summarized here.
The user satisfaction was in the “good” range, with 10 out of 11 tasks yielding a median score of “Satisfied” or “Very Satisfied” and the entire product yielding a System Usability
Scale satisfaction index of 68.3 (on a scale of 0 to 100).
The performance data revealed median times-on-task to range from 1 min 45 sec (“Create an article”) to 3 min 35 sec (“Approve and publish content”).
Sixteen potential usability problems were identified and communicated to the development team in late August. These included:
0 critical issues,
2 major issues,
8 moderate issues, and
6 minor issues.
Video clips are included to illustrate some of the identified problems.
The development team made some changes to the product before the product was shipped, as a result of this testing. The next step is to conduct a comparison test with the old product, to quantify the presumed increase in usability.
© Copyright 2007, “Software Development Company”, Inc 3
2. Context
The week of August 20, 2007, “Software Development Company” and the University of
Texas at Austin School of Information continued their usability collaboration with a usability test of the “product”. The “product” user interface (UI) had undergone other phases of formative (i.e., “find-and-fix”) usability testing within “Software Development
Company”, the results of which have helped steer the current design. The current test was an end-of-the-cycle, summative, “benchmark” test, entailing the collection of performance data (e.g., error rates, and time-on-task) and satisfaction data (i.e., survey data) against which future emerging releases could be compared.
As of today we have completed nine individual test sessions with test participants who are representative of the “product” target audience. A final report, complete with summary statistics of the quantitative data, will have to await the planned testing of six additional test participants. However, given that “Software Development Company” is very near “code freeze” for this product, we wished to share these results in hopes of their having maximal impact on the usability of the ultimate shipped product.
3. The Product
The product tested was the “product” for “Software Development Company” as part of the
“Code Name” release. The “product” was designed to reduce some of the known usability challenges working in the “Software Development Company” “Product Component” for content contribution and not to completely replace the current “Product Component”. The
“Product Component” was designed as a one-stop shop for all content management needs and as a result it does not meet the needs of any one user type particularly well. The
“product” was designed to assist the Amy and Kristen personas who use the strategy of working in projects rather than on the page.
4. User Audience
This test was to be explicitly of users who create content for the web as a part of their job.
We’ve enjoyed a mix of test participants, diverse in respect to their previous experience with “Software Development Company” products, and with respect to the web management roles in which they serve play. These preliminary results represent the findings from test sessions with employees of Customer #1, Customer #12, Customer #3, and Customer #4.
5. Method
We tested one “live” participant, and 15 remote participants (using Webex). Each was tested individually. Each was given NO training on the product. Each was given a brief scenario (they were to pretend to be sports writers for an online newspaper, writing a series
© Copyright 2007, “Software Development Company”, Inc 4
of articles about David Beckham), and then given a series of prescribed tasks to carry out while we observed them. They were asked to “think aloud” as they carried out their task, affording us further insight into their thought processes. After each task we ask them to rate their experience on a 1 to 5 scale of “Not satisfied” to “Very satisfied.” At the end of the entire session (which lasted about an hour) they were asked additional questions about their experience, both open-ended questions and scaled (1 to 5) questions. The entire test package, including test scenario and tasks performed, is in Appendix D.
6. Findings
As in all good usability tests, we collected three basic types of data: performance data, satisfaction data, and specific usability problems identified.
6.1. Performance Data
Owing to our willingness to discuss the tasks with the test participants (who were, for the most part, also customers) the time on task data are messy. In anticipation of the comparison test with v. 7.3.05 of the product, we went back and reviewed the video tape for the five tasks that were shared between this test and the 7.3.05 test. The time on task data for these five tasks are shown in Appendix A. Median times-on-task ranged from 1 min 45 sec (“Create an article”) to 3 min 35 sec (“Approve and publish content”).
6.2. Satisfaction Data
The satisfaction data came in two forms: a single satisfaction question at the end of each task, and a 10-question System Usability Scale (SUS) questionnaire at the end of the study.
The SUS was developed at Digital Equipment Corporation in the mid-1990s and has been often used as a “quick and dirty” measure of usability.
Appendix A shows the detailed results of the per-task satisfaction question. All of the questions were “On a scale of 1-5, how satisfied were you with the task you just performed”? The median score for eight of the 11 tasks was 4. Only one task (“Use attributes from previous article to create a fast way to write articles”) yielded a lower median score, and still this one was 3. Two tasks (“Create an article” and “Remove image shortcut”) yielded a median score of 5.
The overall SUS score, as seen in Appendix C, was 68.3, with a standard deviation of 19.7.
This would be a decent, if not great, user satisfaction score.
© Copyright 2007, “Software Development Company”, Inc 5
6.3. Findings
The following are the potential usability problems that we’ve identified so far. We offer them with the following caveats and considerations:
ALL these findings are based on behaviors and comments of real users representative of the “product” user audience – these are not simply the opinions of usability professionals.
Changes made early are cheaper (both in terms of development costs, and customer satisfaction) than changes made in subsequent releases.
We realize that corporate tailoring and training in the client company may influence users’ perception of the product.
We further realize that interaction with a product in the first hour does not necessarily represent practiced interaction with the product.
That said, the following issues are presented because we believe they may inhibit ease of use of, and customer satisfaction with, “Software Development Company” “product”.
6.3.1. Severity of the problem created by the issue
For now, we will present only the most salient usability problems identified. We will assign each a severity according to the following table:
SEVERITY
RATING
SEVERITY
DESCRIPTION
SEVERITY DEFINITION
1
2
3
Critical
Major
Moderate
The identified issue is so severe that:
Critical data may be lost
The user may not be able to complete the task
The user may not want to continue using the application
Users can accomplish the task but only with considerable frustration and/or performance of unnecessary steps
Non-critical data may be lost
The user will have great difficulty in circumventing the problem
Users can overcome the issue only after they have been shown how to perform the task
The user will be able to complete the task in most cases, but will undertake some moderate effort in getting around the problem
The user may need to investigate several links or pathways through the system to
© Copyright 2007, “Software Development Company”, Inc 6
4 Minor determine which option will allow them to accomplish the intended task
Users will most likely remember how to perform the task on subsequent encounters with the system
An irritant
A cosmetic problem
A typographical error
In all, below are listed:
5 “good things,”
0 critical issues,
2 major issues,
8 moderate issues, and
6 minor issues.
6.3.2. Good Things
Issue Good1: Users commented on liking the different widths for modules, and the automatic reflow of the module depending on which column it was placed in.
Issue Good2: The instructions within the modules are clear and helpful.
Issue Good3: Though people almost universally did not like the pencil-plus-downarrow icon, they liked the presence of the drop down menu of actions.
-
Issue Good4: Users liked the “Quick action” functionality.
Issue Good5: The progress indicator bar (“Loading”) is great.
6.3.3. Critical Concerns
None were identified.
6.3.4. Major Concerns
Issue Major1: The tree navigation only displays only 10 items, requiring the user to “view more.” This could be a “moderate” problem – all users WERE able to work around the problem. However, the fact that users universally hated this elevates it to a major issue; such an obvious usability problem will erode users’ confidence in the product. Here are three quotes from usability testing:
“Now I see why you need a shortcut.”
“I’m glad I’m not looking for Zimbabwe.”
“That’s onerous.” o Severity: 2
© Copyright 2007, “Software Development Company”, Inc 7
o Recommendation: Change the navigation, allowing the whole list to display, requiring users to simply scroll. o Video clip 1 ; Video clip 2
Issue Major2: There are various points where the feedback to the user is subpar, so that the user doesn’t know either the status of the system or what actions are expected of him/her. Here are some examples:
1.
When the “Find Content” module is toward the bottom of the screen, on the righthand side of the “product”, the results of the search appear below the fold, and the test participant didn’t realize the search query had returned some results.
2.
The refresh on a schedule led to users not realizing they had successfully added content or a quick action. Test participants didn't get any confirmation for actions because the refresh wasn't occurring. NOT A SINGLE TEST PARTICIPANT
EVER REFRESHED ANYTHING. Users need confirmation and reassurance that their actions worked. Because of refresh on a schedule, several test participants performed the actions multiple times, hoping for a response.
3.
Users were consistent in their LACK of understanding if some bit of content had been published. Subjects were hungry to get confirmation/indication about approval and publishing status. Everywhere they saw content. They were very worried about these actions and needed reassurance. o Severity: 2 o Recommendation: When you save a search a line appears at the top of the module with a message “The search has been saved.” Offer similar feedback for every positive action. o Video clip 1 ; Video clip 2 ; Video clip 3
6.3.5. Moderate Concerns
Issue Moderate1: Web-based application problems. The application seems to aspire to emulate desktop (Windows-based) applications. The reason for this is, of course, positive transfer of learning – every user of this application will have had experience with Windows-based applications, and will benefit from those similarities. However, where the product behavior is DIFFERENT from actual desktop application behavior, those inconsistencies will be costly, both in terms of the users’ time, and in the users’ confidence that they know what to expect in the way of future behavior from the product. Here are three examples:
Users wanted to right click to see actions they could perform on product objects. Right click offers only browser-related tasks.
SOME “pop-up” windows look like Windows pop-ups, but users cannot interact with them the same way (e.g., move them by grabbing the title bar), so they violate some users’ expectations.
When the user launches a saved query, it “bops” (in the words of one user) the user back into the “Search” tab. Now, this is the behavior we expect, i.e., to run the search. But to change tabs is not consistent with a Windows-like application. o Severity: 3
© Copyright 2007, “Software Development Company”, Inc 8
o Recommendation: ALMOST consistent is worse than not-at-all consistent.
That is, if interface widget A looks like interface widget B, and works only
SOMEWHAT like it, that’s worse than working not at all like it. Make sure all pop-ups behave as pop-ups. Also, we could have the search function within the Saved Searches tab. o Video clip 1 ; Video clip 2 ; Video clip 3
Issue Moderate2: Small icons/widgets. Many things.
1.
In the navigation tree, the check box is too small. At least one user thought she had clicked it (she clicked the word, which works SOME of the time) but had not.
2.
Shouldn’t have to mouse over icons (e.g., edit a quick action) to see that it is there.
This is an example of a violation of the usability tenet “make the user aware of system status.” If a function is available (i.e., those functions represented by the icons) the user should NOT have to take some action (i.e., rolling over them) to discover this fact.
3.
The icon for images isn't viewed as an image icon, instead it seems to be a "movie" icon. As a result, subjects had a hard time figuring out they had preview ability.
4.
One user said the right arrow (show details) looks like a “play” button on a CD player (or online music/video player). We don’t think we’re worried about this one, as we’ve seen this widget in lots of Internet apps.
5.
The pencil icon seems to mean “edit” to most users. It’s becoming an Internet standard. But the pencil-plus-down-arrow implied to perhaps NO test participants that an action menu was available.
6.
In the New Shortcut pop-up, the (standard!) information indicator looks too much like one of the radio buttons. Separate it more from the selections, and/or indent it.
7.
Other small icons don’t “carry the day.” No one knew that the small icon to the left of the shortcut was, that allowed the user to preview an image without clicking on it. o Severity: 3 o Recommendation: For this release, changing the pencil-plus-down-arrow icon plus indenting the information indicator (items 5 and 6) are the only two we’d request. o Video clip 1 ; Video clip 2
Issue Moderate3: Some pop-up windows are displayed such that the user has to scroll down to see the action buttons at the bottom. o Severity: 3 o Recommendation: Don’t require this. o Video clip
Issue Moderate4: Multiple test participants were confused by the “Copy of” in the name field of the “New article” when going there from a Quick Action. o Severity: 3 o Recommendation: Would it work to have the “Copy of <old article name>” selected, so that the first keystrokes would replace it? o Video clip 1 ; Video clip 2
© Copyright 2007, “Software Development Company”, Inc 9
Issue Moderate5: It looks to confuse users, that the "publish" sends you into the old
"job" terminology. It was confusing to many. One subject said, “I like the job I currently have.” o Severity: 3 o Recommendation: Rework the job display text so that it uses the more friendly wording consistent with the rest of the console. o Video clip
Issue Moderate6: The "copy attributes" tab doesn’t copy the path and channel of the copied item. Subjects were repeatedly confused as to why they had to select a channel and path. In fact, most ended up adding the path during the “create an article” process. It would be nice if users could edit the attributes of the item after copying them. o Severity: 3 o Recommendation: Copy all attributes including the project and channel.
Allow users to edit attributes of a copied item during the “copy attributes” process.
Issue Moderate7: Subjects had a hard time discerning the difference between a quick action and an article and a task. This could be a function of our test scenarios and our limited workflow since the article and the task had the same name. o Severity: 3 o Recommendation: In the descriptive text we tell them how to "edit the quick action" but we don't say how to "create a piece of content". That might help.
Issue Moderate8: The red "x" next to the channel in the quick action page is a potential problem. Two test participants had a hard time with that. They both thought that the red “x” meant something was wrong, rather than it being a way to remove the channel association. o Severity: 3 o Recommendation: Use a different way to allow users to remove a channel association, perhaps a clearly worded check-box.
6.3.6. Minor Concerns
Issue Minor1: One user noted that she wanted the whole project path displayed, on the New Quick Action screen. o Severity: 4 o Recommendation: Display entire project path. As the subpath could be in any of a number of higher-level paths.
Issue Minor2: Terminology items.
1.
“Quick action” vs. “Template.” Users loved the Quick Action functionality, but they tended to call it “Template.” We realize that predecessor “Software
Development Company” products make this change hard.
© Copyright 2007, “Software Development Company”, Inc 10
2.
“OK” and “Apply” vs. “Save” and “Save as.” One user wanted to “Save,” and was confused by the “OK” and “Apply” terminology. (This is related to the Windowsbased application model issue, Issue Moderate1.)
3.
Multiple test participants wanted to see “Author” or “Editor” instead of “Creator.”
(Is this an issue about only the terms that a company’s IT team would insinuate into the product, then we needn’t worry about this one.)
4.
“Not selected” seems ambiguous (could be the item was not selected, rather than there not having been an exemplar of that item selected).
5.
What's the difference between “clone” and “copy attributes”? o Severity: 4 o Recommendation: Save the “template” item for consideration in a later release. We THINK the OK terminology will work. Consider changing the
“Not selected” to “None selected”.
Issue Minor3: Inconsistency in the way “required field” is indicated. In the “New
Article” page, required field captions are in bold. On most screens, required fields are indicated via an asterisk. o Severity: 4 o Recommendation: Be consistent. Go with the asterisk.
Issue Minor4: Need article/content name on the New Job page, so the user will not have to remember what he/she is publishing. o Severity: 4 o Recommendation: Add it.
Issue Minor5: One test participant wanted the completed tasks not to just disappear, but to offer “completed tasks” as a type to view. o Severity: 4 o Recommendation: Add this feature.
Issue Minor6: When I go to create a new Quick Action, I need to click on the name field to get started. o Severity: 4 o Recommendation: Put the cursor in the name field when the screen comes up.
7. Next Steps
For this evaluation, the next steps will be to conduct a comparison test with the old product user interface. We welcome any requests for clarification, to Randolph
( rbias@ischool.utexas.edu
, cell 512-657-3924) or “User experience professional from the
Software Development Company.
© Copyright 2007, “Software Development Company”, Inc 11
Randolph G. Bias, Ph.D.
University of Texas at Austin
School of Information
1 University Station, D7000
Austin, TX 78712-1276
(512) 657-3924 rbias@ischool.utexas.edu
8. Contact Information
© Copyright 2007, “Software Development Company”, Inc 12
Per Task Time-on-Task (min:sec) Data
9. Appendix A: Time on Task
Subj.
Ques. 2 4 5 6 7
1.5 2:24 4:26 1:21 1:50 7:10 x
8 9
2:00 x
10 11
1:45
12 14
2:26 1:20 x
15 16 18 19 Median
0:58 1:40 3:30
1.9 3:31 1:29 0:58 0:58 2:25 x
1.11 3:13 1:48 1:04 1:31 4:24 x
1:15 3:36 2:35 2:08 2:20 x
0:56 3:25 1:48 1:20 1:07 x
0:56 1:05 1:45
1:38 1:55 2:27
1.13 3:02 2:56 1:50 1:32 6:55 x 1:34 1:03 2:55 2:05 1:20 x 6:00 2:15 4:30
1.17 4:45 3:35 2:33 6:27 5:35 4:10 2:10 1:28 3:07 13:25 1:23 5:00 2:05 6:27 3:10
1:53
1:45
1:48
2:15
3:35
© Copyright 2007, “Software Development Company”, Inc 13
10. Appendix B: Per Task Satisfaction Data
Per Task Satisfaction Data
Subject
Question 4 5 6 7 8 9 10 11 12 14 15 15 16 17 18 19 Median
1.2.1
1.4.1
4
3
5
5 5
2
2
5
4
5
4
5 3
4
4
5
4
5
3
3
4
3
5
5
5
4
5
3
4
4
4
4
1.6.1
1.8.1
1.10.1
1.12.1
1.14.1
1.16.1
1.18.1
1.22.1
1.24.1
2 5 5 1 4 4 3 3 3 5 5 5 5 5 4 3
2 3 4 1 4 3 4 1 4 3 2 2 2 3 3 4
4 5 5 2 5 4 1 5 5 5 5 4 4 5
4 5 5 1 3 4 3 4 2 3 5 5 5 3 4 4
3
3
5 2 5 4 3 4 2 4 3 3 3 4 4 5
4 4 5 5 5 3 4 3.5 3 5 4 4 5
3 4 2 5 5 2 1 5 5 5 4 3 2 4
3 4 5 5 3 4 5 2 4 5 5 5 5 3 4
4 5 5 5 5 5 5 5 5 5 5 5 5 3 5
4
3
5
4
4
4
4
4
5
© Copyright 2007, “Software Development Company”, Inc 14
11. Appendix C: SUS Data
SUS Score Calculator
Subject
Ques. 4 5 6 7 8 8 9
1 3 5 4 3 4 4 4
2 2 1 1 4 3 3 2
3 3 4 4 2 4 3 4
4 2 1 1 1 2 2 3
5 4 4 3 3 3 3 5
6 2 1 1 3 2 2 2
7 2 4 4 3 2 2 4
8 4 1 1 4 4 4 2
9 4 4 4 2 3 2 4
10 2 1 3 4 3 3 2
10
5
2
4
3
5
1
5
2
4
3
11 12
3
5
3
4
3
5
2
4
3
5
13 14 15 15 16 17 18 19
5 5 5
2 2 1
4 4 4
1 1 1
4 3 4
2 1 1
4 3 3
2 1 1
4 3 4
3 1 1
5 5 4 3
1 2 3 4
5 5 3 4
1 2 2 3
5 5 4 4
1 2 3 3
5 3 4 4
1 1 2 3
5 4 4 4
1 3 4 3
SUS
SCORE 60 90 80 43 55 50 75 28 -- 78 80 88 100 80 63 58
SUS
STATISTICS
AVG
Min Max
68.3 19.7 0 0
© Copyright 2007, “Software Development Company”, Inc 15
12. Appendix D: Test Scenario
CONTENT CONTRIBUTOR WORKSPACE TEST
Usability Study of “Software Development Company” Product ......
Ошибка! Закладка не определена.
© Copyright 2007, “Software Development Company”, Inc 16
© Copyright 2007, “Software Development Company”, Inc 17
13. Instructions and disclaimers
These mockups are just concept sketches. They are not complete and may or may not represent functionality that will be seen in a future product. Depending on the results we obtain from users the actual product could vary significantly from what you see today.
You will be asked to carry out some specific tasks with our web content management software. As you go we’d like to ask you to think aloud, telling us what you’re looking at and trying to do. After each task we ’ll ask you some survey questions, to judge your attitude about the product.
The test is testing the product and not you. Any problems you have a result of the product being unclear.
If you have any questions feel free to ask us, but we may not be able to answer them. If at any time you would call the “Software Development Company” help desk, just use the phone to do so, at 512-657-3924 (or tell us that you would).
You can quit the test at any time for any reason.
We MAY interrupt you at some point an d ask you to move on to the next task, as we’ve already learned what we think we need to learn about a particular task.
The entire test will take about an hour, and you may take a break at any time if you wish.
Thanks for your willingness to participate! Your input will help us create a better product!
© Copyright 2007, “Software Development Company”, Inc 18
14.
Who you are
You are a writer for an on-line newspaper. You write a lot of sports articles for the on-line newspaper. You have just been assigned to cover David Beckham (Becks). You will be writing stories about his life on and off the field -- his great soccer moves for the Galaxy as well as his move to Los Angeles with his famous wife Victoria Beckham (Posh) and their four children.
This is the first time you have used the new and improved workspace for
“Software Development Company”. You are going to set it for making your work as fast and easy as possible for you.
© Copyright 2007, “Software Development Company”, Inc 19
If you have problems that you cannot solve on your own, you can call the
“Software Development Company” support desk at (512) 657-3924, if you have access to another phone. Otherwise, just tell the test administrator that you would call the support desk, if you had the ability.
© Copyright 2007, “Software Development Company”, Inc 20
1. First Impressions of the “product”
This is the first time you have used the new workspace for “Software Development
Company”.
You are interested in learning how to make your content creation, approval and publishing process more efficient.
To see how you could use the new workspace, you play around with the modules on the workspace by moving them on the page. You also try turning them off and on.
Task
What to do, if the system is able:
1.1.1. Move the Tasks module over above the Quick Actions Menu.
1.1.2. Move the Shortcuts module over below the Find Content Menu.
1.1.3. Remove the Recent Content module from the workspace.
1.1.4. OK, that was cool! (Or maybe it wasn’t.) But you decide that you want to put it back how it was until you use it a little more.
1.1.5. When you are finished with the task, say “I’m done.”
1.2. Questions
1.2.1. Is this something you would use?
1.2.2. Can you tell us your best bet as to how you would organize your page?
© Copyright 2007, “Software Development Company”, Inc 21
1.2.3. On a scale of 1-5, how satisfied were you with how easy it was to perform that task?
(Check one box.)
NOT
SATISFIED
1.1. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.2. ATT
RA
5 CTI
VE
1.2.4. Comments:
© Copyright 2007, “Software Development Company”, Inc 22
2. Create a fast way to write articles
You know you will be writing a lot of stories about David Beckham’s personal life since many people are interested in his move to Los Angeles with his famous wife
Victoria. Since these stories are all about his personal life off the field, you know that most of the stories will have the same descriptions (attributes).
You decide the first thing you are going to do is set up your workspace so that you can more quickly and efficiently create stories about David Beckham off the field over and over again.
2.1. Task
What to do, if the system is able:
2.1.1. Create the ability to quickly create a new David Beckham article with the following information:
2.1.2. Name: Beckham off the field
2.1.3. Content type: Sports Article
2.1.4. Project Path: Atlas/America
2.1.5. Channel: MyMyMy:/Home/channel_A/channel_B /channel_C. (Your company doesn’t use classifications).
2.1.6. Check to make sure you created “Beckham off the field” with all the correct attributes.
2.1.7. When you are finished with the task, say “I’m done.”
2.2. Questions
2.2.1. On a scale of 1-5, how satisfied were you with how easy it was to perform that task?
(Check one box.)
NOT
SATISFIED
1.3. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.4. ATT
RA
5 CTI
VE
2.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 23
3. Create an article about becks’ life
You need to write a spin on the story just released about how David and Victoria Beckham are buying a six-bedroom home in Los A ngeles near Tom Cruise and Katie Holmes’ house.
Luckily, you just created a way to add new articles about David Beckham’s life off the field quickly to the system.
3.1. task
What to do, if the system is able:
3.1.1. Create a new article the fastest way you can about David Beckham’s life outside of soccer.
3.1.2. Designate the “sport” as “NFL” (National Football league, of course ;-)
3.1.3. Title the article: “<your initials>Big and Pricey”
3.1.4. Make the author: your initials
3.1.5. Have the article say, “Fancy”.
3.1.6. When you are finished with the task, say “I am done.”
3.2. Questions
3.2.1. On a scale of 1-5, how satisfied were you with how easy it was to perform that task?
(Check one box.)
NOT
SATISFIED
1.5. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.6. ATT
RA
5 CTI
VE
3.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 24
4. Use attributes from previous article to create a fast way to write articles
Who would ha ve guessed! Your article about David and Victoria Beckham’s new house has spawned a media frenzy about the house and their new neighbors. You are writing several articles a day about that house!
Since you are writing basically the same article over and over again with small changes, you decide to create a quick way to write stories about Posh and Becks’ house. The fastest way is use the system to set up a template that copies the attributes from article you wrote previously about the house.
Then, you can use that template over and over again.
4.1. Tasks
4.1.1. Create the ability to quickly create new articles about Posh and Becks’ house based on an article that you wrote previously about the same topic. Use the following information:
4.1.2. Name: Posh and Becks house
4.1.3. Content type: Sports Article
4.1.4. Article whose attributes to use: “<your initials>Big and Pricey”
4.1.5. When you are finished with the task, say “I’m done.”
4.2. Questions
4.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.7. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.8. ATT
RA
5 CTI
VE
4.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 25
5. Create A HOUSE ARTICLE
You get new information about David and Victoria Beckham’s house. Victoria has been quoted as saying it is decorated in simple, muted tones. As this decorating style may start a national trend, you know you have to quickly write an article about it.
You write a new article, using the fast way you created earlier, about David and Victoria
Beckham’s new house.
5.1. tasks
5.1.1. Create content about Posh and Beck’s house the fastest way you can.
5.1.2. Title the article <your initials> Big and subtle?”
5.1.3. Have the article say: “muted tones”.
5.1.4. When you are finished with the task, say “I’m done.”
5.2. Questions
5.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.9. ATT
1
RA
CTI
5.2.2. Comments:
VE
2 3
VERY
SATISFIED
4
1.10. ATT
RA
5 CTI
VE
© Copyright 2007, “Software Development Company”, Inc 26
6. Create a fast way to get to a folder
A lot of the articles you are writing have been going to the France website since your French readers are very interested in what David Beckham is doing in the United States. So, you create a fast way to get to the France/home channel.
6.1. Task
What to do, if the system is able:
6.1.1. Create a fast way to get to the channel: “MyAtlas://France” .
6.1.2. When you are finished with the task, say “I’m done.”
6.2. Questions
6.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.11. ATT
1
RA
CTI
6.2.2. Comments:
VE
2 3
VERY
SATISFIED
4
1.12. ATT
RA
5 CTI
VE
© Copyright 2007, “Software Development Company”, Inc 27
7. Find a picture of a penguin
You think it would be funny to have a fancy cartoon image of a penguin in one of your articles about David and Victoria Beckham’s house. You know you’ve seen it in the system before and you know it was a file type “png”. When you find it, you decide to create a shortcut to it so that you can find it again really easily.
7.1. Task
What to do, if the system is able:
7.1.1. Search “*.png” files in the system.
7.1.2. Look for one called “babytux.png” in the list.
7.1.3. Create a shortcut to “babytux.png”.
7.1.4. Look at the image from the shortcuts menu using no clicks.
7.1.5. When you are finished with the task, say “I’m done.”
7.2. Questions
7.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.13. ATT
1
RA
CTI
7.2.2. Comments:
VE
2 3
VERY
SATISFIED
4
1.14. ATT
RA
5 CTI
VE
© Copyright 2007, “Software Development Company”, Inc 28
8. All content I created
At this point, you can’t really keep track of all of the articles you’ve written about sports. So you decide to create and save a search that looks for all the sports articles written by you.
You save the search, double check it’s details and then run the search again.
8.1. task
What to do, if the system is able:
8.1.1. Find all the sports articles written by you (use your initials like you did previously).
8.1.2. Save the query and name it “<your initials> sports articles”.
8.1.3. Look at the query’s details to verify that it has the parameters in it that you wanted.
8.1.4. Run the search again to see if you are happy with the results.
8.1.5. When you are finished with the task, say “I’m done.”
8.2. Questions
8.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.15. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.16. ATT
RA
5 CTI
VE
8.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 29
9. Approve and publish content
You are thrilled with your lovely articles about David and Victoria Beckham’s house. You know that every newspaper in the country is going to try to write similar articles, so you want to get these published out on the website fast!
In the fastest way possible, approve and publish the articles you wrote.
9.1. task
What to do, if the system is able:
9.1.1. Approve and then publish the article “<your initials>Big and Pricey” you wrote.
9.1.2. Do it in the fastest way possible (using the “product”).
9.1.3. When you are finished with the task, say “I’m done.”
9.2. Questions
9.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.17. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.18. ATT
RA
5 CTI
VE
9.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 30
10. Workflow 1
At your on-line newspaper, the workflow you use assigns you the task of reviewing and approving the articles you wrote.
For the purposes of this test, imagine that your task inbox contains many dozens of tasks.
You’re trying to see a list of just the tasks that have the word “Editors” in their Description column. How would you do that?
10.1. task
What to do, if the system is able:
10.1.1. Find a way to see just the tasks that contain the word “Editors” in their Description column.
10.1.2. When you are finished with the task, say “I’m done.”
10.2. Questions
10.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.19. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.20. ATT
RA
5 CTI
VE
10.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 31
11. Workflow 2
At your on-line newspaper, the workflow you use assigns you the task of reviewing and approving the articles you wrote.
Since you have just created two articles, you have two articles to review and approve. You review, add a note and finish the first article in the list.
11.1. task
What to do, if the system is able:
11.1.1. Review the first article in the list.
11.1.2. Add the note, “great”.
11.1.3. Finish the task.
11.1.4. When you a re finished with the task, say “I’m done.”
11.2. Questions
11.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.21. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.22. ATT
RA
5 CTI
VE
11.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 32
12. Remove images shortcut
You received website traffic data that says the France site’s Beckham articles aren’t receiving any traffic. You know you won’t be publishing content to the French site anymore so you delete the “France” shortcut.
12.1. task
What to do, if the system is able:
12.1.1. Remove the shortcut to “France”.
12.1.2. Minimize the shortcuts module.
12.1.3. When you are finished with the task, say “I’m done.”
12.2. Questions
12.2.1. On a scale of 1-5, how satisfied were you with the task you just performed?
NOT
SATISFIED
1.23. ATT
RA
1 CTI
VE
2 3
VERY
SATISFIED
4
1.24. ATT
RA
5 CTI
VE
12.2.2. Comments:
© Copyright 2007, “Software Development Company”, Inc 33
13. Post-Test Questions
What other features would you expect in a product like this?
Would using a user interface like this improve your current process? How?
Was the terminology and wording clear?
What was your favorite part of the user interface?
What was your least-favorite part?
© Copyright 2007, “Software Development Company”, Inc 34
14. System Usability Scale
Please check the box tha t reflects your immediate response to each statement. Don’t think too long about each statement. Make sure you respond to every statement. If you don’t know how to respond, simply check box “3”.
Created Tuesday, September 15, 2009Page 35 of 39
Development Company”, Inc.
© Copyright 2006, “Software
1. I think that I would like to use this product frequently.
Strongly Strongly
Disagree Agree
1 2 3 4
2. I found the product unnecessarily complex.
5
Strongly Strongly
Disagree Agree
1 2 3
3. I though the product was easy to use.
4
Strongly Strongly
5
Disagree Agree
1 2 3 4 5
4. I think that I would need the support of a technical person to be able to use this product.
Strongly Strongly
Disagree
1 2 3
Agree
4 5
Created Tuesday, September 15, 2009Page 36 of 39© Copyright 2006, “Software Development Company”, Inc.
Content Workspace Tasks
5. I found the various functions in the product were well integrated.
Strongly Strongly
Disagree Agree
1 2 3 4 5
6. I thought there was too much inconsistency in this product.
Strongly Strongly
Disagree Agree
1 2 3 4 5
7. I imagine that most people would learn to use this product very quickly.
Strongly Strongly
Disagree Agree
1 2 3 4
8. I found the product very awkward to use.
Strongly Strongly
5
Disagree Agree
1 2 3
9. I felt very confident using the product.
4 5
Strongly
Disagree
Strongly
Agree
“Software Development Company” Confidential 37
Content Workspace Tasks
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this product.
Strongly Strongly
Disagree
1 2 3
Agree
4 5
Thank you very much for your time and feedback.
“Software Development Company” Confidential 38
Content Workspace Tasks
“Software Development Company” Confidential 39
