This report summarizes the initial findings from a series of usability testing sessions conducted using the scientific authoring tool add-in developed for use with Microsoft Word
2007. These test sessions were held as part of the Sheridan Libraries project, Digital
Archiving of 
 by IMLS and Microsoft Corporation.
In general, users were optimistic about the potential of the tool and its use for their authoring needs. When asked what they liked best about the tool, three of the five users made positive comments about the importance of metadata, linking to data and linking tables to data during the authoring process. However, one user stated the tool was difficult to use. Another stated the tool needs more work to be easier to use. Labels on interface elements were not what the user expected them to be. The language was described as being too technical or having too much jargon.
Most of the users had no trouble finding and opening the provided Sample File With
Keywords document, however each user expressed some degree of frustration with finding their way around the interface. Two of the users could not complete the tasks involving the
Resource Map after viewing a video explaining how to use it. Their comments are listed in the following User Feedback Matrix. A heuristic evaluation, based on Jakob Nielsen’s 10
Heuristics for User Interface Design , is also provided to further summarize the observations.
Five task-based evaluation sessions were held during October 2009 at the Johns Hopkins
University campus. Participants from the Johns Hopkins University Department of Physics and Astronomy Department, Computer Science and Biomedical Engineering were recruited to evaluate the tool, as well as science librarians at the Sheridan Libraries. All of the participants were Windows users and indicated they used Word 2007 regularly on the pretest survey. Three of the users were familiar with or have used other template-based tools to submit articles to journals. The equipment used was a Windows laptop, with Version 2
Beta 2 of the authoring tool loaded into Microsoft Office.
To assist with learnability and understanding of the tool’s functionality, each user viewed
Connecting External Data with Scientific Articles , a video that was produced by the tool developer.
The video stated the purpose of the authoring tool and demonstrated the features of the Resource Map. User s were then given a series of tasks to complete using the tool that progressed from discovering the interface to linking resources to external data. Users were asked to think aloud as they interacted with the tool and their comments and actions were recorded by note taking. A post-test survey was given to document the participant comments about and reaction to using the authoring tool.

1.
From Microsoft Word 2007, start a new file using the Sample Article with Keywords from the templates directory. This document will be your sample article. Using the authoring tool interface, add information about you as the author. Add information about the article.
2.
There are several collaborators on your paper. Use the authoring tool interface to add information for two additional authors.
3.
One of your collaborators has multiple affiliations. Add these affiliations to his or her name.
4.
More content needs to be included in this article to support the arguments presented. Add a data table, a figure and an equation to the report section.
5.
Use the authoring tool to associate the table, figure and equation you just added to urls of external datasets.
6.
Link the table with an external resource by attaching a file from your local disk and link it to the figure in the document.

Question
What features were vague or confusing to you, if any?
What is your impression about navigating the interface?
What else should be included in the tool?
What did you like best about the tool?
User1

Adding author details

Pre-defined relations

Should scan document to start with

Resource map GUI needs work.

New way (for me) to add a figure.
Generally, it seems easy and straightforward once you’ve played with it a bit.
Displaying metadata in the document, e.g., authors and affiliations.
I understand the importance of metadata.
User2

Didn’t recognize the term
‘figure’ as it relates to publishing.

The word ‘link’ does not appear in the resource map.
No comment.
Nothing specific
User3

Equations, but then I hardly ever use them.

Was not any more confusing than any other new program
I’ve used.
It’s nice that it fits in with
Word. Makes it easier to navigate.
Nothing comes to mind.
No big error messages when I goofed. It followed a pattern.
Once I got the pattern, it worked pretty well.
User4
I need an overview of the logic of how the whole tool works. It seems that the workflow is that you create the data tables first
(as urls or files), and then associate them with tales, figures, and equations in the document.
I found it very difficult to use. It seems like there were may steps required to accomplish tasks. It might have been easier if I had started with the data and worked from there.
The ‘Insert Sections’ or
‘Insert Special’ menus should appear by default when the tool is in use.
The ability to autogenerate formatted tables and figures from data.
The repetitiveness of the interface.
User 5

Jargon in the Resource
Map is confusing

I hope/feel the tool interface needs to be more transparent, as the concept of linking components in a document is very interesting /useful
The session titles can be better; e.g., I don’t know the difference between
‘Insert’ and ‘Insert Special’
No comment
 Cross linking and referencing the original data
 Getting things from the cloud
 Keeps you from telling lies. Cannot tweak this like LaTeX
Too many jargons in the language
What did you like least?
Do you have any other comments?
Authors not showing up in the document. Look of the figure is a showstopper.

Document has no signs of associated metadata, i.e., little anchors to click on.

Figure is not publication quality; needs to be customizable
Nothing specific

Was unsure of its integration into MS Word.
Will it now be part of Word or will it be a separate piece of software?

How to locate it once the
Resource Panel is in the background.
I don’t know.

I think it will be useful. Will publishers need to have the same tool when they edit the submitted papers?

Without the authoring tool, can Word open a document created by Word with the authoring tool? There are implications for authors and editors.
No

I use mostly LaTeX and templates, which are not as powerful as this tool, but has better ergonomics, i.e., no buttons to search or understand, but still needs manual.

This is very ambitious

Will take a lot for people to get used to this

Assumes dependencies

Astronomy journals don’t require role or bibliography

Visibility of system status
Rationale:
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
Match between system and real world
Rationale:
The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural
and logical order.
User control and freedom
Rationale:
Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Supports undo and redo.
Consistency and standards
Rationale:
Users should not have to wonder whether different words, situations, or actions mean the same thing.
Follow platform conventions.
Flexibility and efficiency of use
Rationale:
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
Observations:
1.
The interface to the authoring tool is hard to distinguish from the default ribbon.
2.
Author names are not immediately displayed in underline, per instructions. User must click in another area of the page for these to appear.
1.
Change display or label of custom tabs so that user can identify the location of the tool.
2.
Re-write help text to be more specific.
3.
User must assume that author names are added without the semi-colon after each name, to be considered a group of authors.
Recommendation:
Observations:
Users seemed to not understand the function of ‘Scan Document’ or
‘Duplicate’ in the Resource Panel
Recommendation:
Change the language on these labels to be less technical
Observations:
Cannot delete a row in the Resource Map if information is not correctly entered.
Recommendation:
Provide instructions in the help documentation.
Observations:
The label ‘Self’ as used in the resource map to indicate the current document can be confusing.
Key combinations are not working as expected. Ctrl-O does not open the authoring tool.
Recommendation:
Refer to current document with a more specific label or mention this in the help documentation.
Observations:
Dialog box for author details such as address, affiliation, and comments requires that properties be declared before adding.
Recommendation:
Provide contextual help
Sheridan Libraries, Johns Hopkins University
Digital Research and Curation Center
October 28, 2009

Error prevention
Rationale:
Even better than good error messages is a careful design, which prevents a problem from occurring in the first place. Either eliminates error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
Recognition rather than recall
Rationale:
Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Aesthetic and minimalist design
Rationale:
Dialogues should not contain information that is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Help users recognize, diagnose, and recover from errors
Rationale:
Error messages should be expressed in plain language
(no codes), precisely indicate the problem, and constructively suggest a solution.
Help and documentation
Rationale:
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.
Observations:
When adding multiple affiliation information for authors, the last entry is hidden to the user.
Recommendation:
Display affiliations separately
Observations:
Users tend to go back to the article to select document elements to make associations, rather than the Resource Map itself.
Recommendation:
Provide contextual help
Observations:
The authoring tool is hard to distinguish in the ribbon.
Recommendation:
Make the tool location in the ribbon obvious to the user, e.g., change the color of the tab ‘Insert Sections’ when the tool is loaded.
Observations:
When user makes an error, e.g., typing in the dialog box to add author affiliation information, a beep occurs. User continues to click back and forth until he or she figures out they have to use the drop down arrow.
Recommendation:
Provide contextual help
Observations:
1.
Instructions for adding author information are not visible until user hovers over the author panel area.
2.
F1 does not display help content for the authoring tool
Recommendation:
1.
Make the help section persist while using Author and Resource panels.
2.
Provide a visual cue for existence of Help.
Sheridan Libraries, Johns Hopkins University
Digital Research and Curation Center
October 28, 2009