Using Usability Experts to Improve Software
Quality
Ilari Kajaste1, Deepa Mathew1, Suvi Peltomäki1, Timo Poranen1
1Department of Computer Sciences
University of Tampere
Kanslerinrinne 1
FIN-33014 University of Tampere
Finland
Abstract
Software project courses are an essential part of university studies of Bachelor's
and Master's degrees all around the world. When outcomes of projects are
estimated, one of the key quality factors is usability of the software. In this paper
we introduce how usability issues are taken into account in a software project
course at the University of Tampere. Our experiences show that using student
usability experts in software project courses, gives promising improvements in
software quality, but also in project learning in a more general level.
1.0 Introduction
Software project courses belong to body of knowledge in all computer science
related disciplines all universities. In many undergraduate degree programs,
software project course is studied on the third year just before receiving BSc
degree. ACM and IEEE describe (SE2004) the capstone project as follows:
“Provides students working in groups, with a significant project experience In
which they can integrate much of the material they have learned in their program,
including matters relating to requirements, design, human factors, professionalism,
and project management.”
Department of Computer Sciences at the University of Tampere has two major
subjects: Computer Science and Interactive Technology. Students who major in
computer science study traditional subjects in their Master’s programmes:
Algorithmics, User Interface Development, Software Development, Data
Management and Information Retrieval, and Information Systems.
Interactive Technology students concentrate in their studies on issues related to
human-computer interaction. The content of this major subject is described as
follows in the Curricula Guide (ECTS guide):
“Interactive technology aims at training all-round IT professionals who have
friendly approach to their work. Making the quality of interaction between man
and technology is a core element in teaching of interactive technology. Due to a
multidisciplinary basis, students can utilise their backgrounds and interests and
specialising in making software and hardware usability evaluations, or
concentrate on developing new and better ways of interaction from a human
perspective. Students can find jobs in a wide variety of different fields; they can
become product development professionals in the software and telecommunication
industry, usability experts in internet and multimedia companies, and researchers
in the field”
About one fourth of all our students are majoring in Interactive Technology. These
students really need experience on working in software projects and they should be
familiar with the different phases of software development lifecycles. Before year
2005, Interactive Technology students studied the software project course normally
with the all other students. They were expected to do similar tasks as other
students, although in reality they usually concentrated on user interface –related
issues.
In our university, only basic programming courses are required as prerequisites
from the interactive technology students. Instead of software development studies,
these students have strong skills in user interface design and evaluation.
Two years ago we decided to develop our software project teaching concept, and
we started so called usability team experiment. In our software project courses
there are roughly 100 third year students and some number of older students who
work as a project managers for our projects. We selected seven students to study
the software project course in the usability team. The role of the usability team
student differs from the other project course students: usability team members
concentrate only on usability related issues in projects.
In this paper, we describe our usability team concept and our experiences from
academic years 2005 -2006 and 2006 – 2007. During these years we have gained
experience in total from 37 projects and two slightly different usability team
experiments.
The rest of this paper is organised as follows. Next, we introduce how usability
issues are related on software projects and what kind of deliverables might be
useful for software projects. In Section 3 we describe our experiences from the
usability team experiments. Section 4 concludes our work.
Deepa's text about usability team models here?
1.1. Advantages of the Usability Team -arrangement
There are three main advantages for using the usability team model.
Firstly, the team allows students majoring in interactive technology to concentrate
the project work studies in a way that is meaningful for their studies. They get
practical experience in collaborating with software projects, a situation in which
they may very well encounter in their work as well.
The team also makes sure every project's usability matters are being looked into.
Since there are not enough usability students for each project - and not each project
needs the full attendance of a usability person - the team provides a way to divide
the work in the best way.
Finally, the team provides peer support for the members of the usability team
working in different projects. Many usability tasks (e.g. usability testing) require
collaboration with multiple usability aware people, and the team arrangement
makes that collaboration frictionless.
Further, as an indirect result, the more complex organisational structure of the
project work course provides experience in varied collaboration skills for all the
participants of the course, not just the usability team.
2.0 Usability in software projects
It is important to first notice that usability in software project is not a one-off issue
you can cover with a couple of documents. It is like good code structure - you
cannot simply create your software, and then make it usable. To achieve software
with good usability, it needs to throughout the entire software project.
2.1. Why usability is needed
Something here?
2.2. Usability documents
2.2.1. Waterfall development model
Usability in essence concerns the same thing as software development itself: it is
about the users and what they do. So, as with requirement specification, it is
important to first learn about the users and their context of use. The required
information can be gathered at the same time the project team starts learning to
know about the context of the project. From that information, the usability
analysis document is produced. It contains definition of general user types of the
final product, context of the product, environment of use, and based on these the
usability requirements for the software. There is also an indirect advantage in
considering usability at this point: thinking about the project from the usability
perspective may also provide valuable insights to what the software should be
about in the first place. In some cases, this may change even some fundamental
aspects of the project, and these insights are valuable to notice before the project
plan is written.
Since the requirement specification defines what the software should be, it cannot
contain definitions that are contradictory to the usability requirements of the
software. The requirement specification of the software is, thus, based in part on
the usability analysis document. This also provides a good foundation for
considering usability later on in the project.
Some aspects of the requirement specification may depend on the user interface of
the software, so preliminary draft of the user interface plan should be created
together with requirement specification. This should contain the general user
interface model and how work will be done in the project. Another reason for
drafting the basic UI design already is more indirect: If UI is not designed at this
phase, some user interface will be assumed by the people writing the requirement
specification. Even though requirement specification should be abstract idea of the
software when defining what software should be able to do, people still assume
some interface for how the software will be doing it. And that interface will more
likely be designed from the technical, not the usability point of view. Because of
this the people designing the user interface plan also need to work in collaboration
with the people writing the requirement specification.
The most visible aspect of software usability is, of course, the user interface itself
which is defined in the user interface plan. Since the plan ties together with the
implementation, it should be done in cooperation with the people writing the
implementation plan. This way the user interface designers get valuable feedback
on what designs are actually possible to implement. This way the people
implementing the software will not discard the user interface as impossible. It is
worthy to note that when compromises must be made for the sake of
implementation, the better user interface design concepts should nevertheless be
written down to the user interface plan to help further development.
These documents are the basic building blocks of including usability strongly in
the software project. Of course, the documents should also contain multiple
references to each other for better integration, so the usability documents are not
seen as an issue separate from the "actual" project. In addition to these basic
documents, there can be other documents concerning usability issues, such as
usability test reports, heuristic analysis reports and the like.
2.2.2. Other development models
In the incremental development model [SOURCE] there are some differences. The
definition of general user types, context of the product and environment of use can
be defined for the entire software in the usability analysis. However, since each
increment may concern a different part of the software, the usability requirements
should be written independently for each increment. These can of course be
separate documents, or just revisions for the original usability analysis. Naturally,
also the user interface plan needs to be written independently for each increment.
In agile development models the same idea applies but of course with a more
flexible approach to documentation, with most of the documents affecting one
another in a more dynamic and open nature which is the whole point of agile
models. The basic development ideas conveyed by the waterfall model document
relations should still apply.
2.2.3. Usability team deliverables (?)
The graphs show how different basic software project documents relate to each
other and usability documents. Strong lines represent document being based on
information from another. Dashed lines represent the same relation, but
significantly weaker. Lines with arrows at both ends mean the documents are done
in collaboration (thus are based on information from each other).
Figure 1: Usability related deliverables in projects applying the waterfall software
development model.
Figure 2: Usability related deliverables in projects applying incremental software
development model.
2.3. Usability design and analysis methods
Some usability design and analysis methods that can be used to improve the
usability of the software being designed. The most common analysis method is
probably usability testing, and paper prototype testing which provide valuable
sight into how actual users interpret the system though the user interface. This is
important because in usability one key issue is that people think within their own
context. While user interface designers and IT professionals may understand some
feature naturally, the users might not. Especially paper prototypes may help
discover significant unexpected user behaviours in very early stage of design.
A closely related issue is that user interface designers easily become accustomed to
their own design and blind to their own mistakes. They may understand how
something works simply because they themselves designed it, and it is often hard
to notice when this happens. (Similar effect happens with coders too: it often takes
a long time for someone other than the coder to read how certain code works.) To
overcome this, the easiest method is probably to hold peer reviews of the user
interface. Peer review means holding a meeting with other user interface designers
unrelated to the project. The interface is informally discussed together, with the
designer presenting the interface to the others. The meeting needs to have an
informal brainstorming atmosphere so that it does not focus on what the designer
has done wrong but rather what can be improved.
Heavier method is heuristic evaluation where the user interface is analysed by
usability professionals taking a set of usability heuristics (most often Nielsen's 10
usability heuristics) and assess whether the user interface follows these guidelines.
Cognitive walk-thoughs?
....... more methods? Or do we need these at all?
3.0 Usability team experiments
3.1 Usability team 2005-2006
The Usability Team (UT) pilot experiment took place in the semester 2005-2006.
The need for the usability team had risen from having a quarter of all students at
the department studying Interactive Technology as their major subject and
previously there had not been a good chance to give these students a meaningful
role in the project applying their major subject’s skills. Previously the students
with extensive knowledge about usability and user interface design had simply
been regular members of the project work groups. Now there was a need to have
outside consultants helping the UI design and evaluation. The design process often
blinds the designer so that they do not see the flaws in their own design. This is a
reason for bringing in outside consultants.
In the beginning of the course, each project’s manager was told that they could
freely employ an outside usability expert for their project for an estimated 50
hours. The managers could then decide for which phases the expert was needed.
Each student chose two or three projects to consult but most students also
consulted other projects when this was needed. In the end of the course, the
projects reported how many hours the experts in fact had been employed in the
project and it was found that the time was anything between 20 to 161 hours,
which in most cases was far more hours than expected. A few projects did not
employ a usability specialist.
The projects ranged from ones where the customer was a researcher from the same
department to projects where the customer was a large international corporation.
The platform also varied from mobile to web, so the consultants got a good chance
to work on multiple different kinds of projects.
The nature of the project determined how the expert worked and what role they
played. In some cases, the usability expert worked as a complete outsider to the
project, only consulting from a “distance”, and sometimes the expert became a
solid member of the group. The members of the project being consulted did not
always see the role of the expert consulting from the outside as positive, which
could lead to design solutions not in line with the advice given by the consultant.
Projects demanding very good usability, though, could have the expert become a
member of the group all the way from the beginning. This was the case in the
Safety at Work project as described in 3.1.1. Case study: Safety at Work project.
This first UT had seven students, all in their third or later year in their studies and
each had Interactive Technology as their major subject. The team did not have a
project manager but rather everyone in the group managed it equally. This was
seen as a serious shortcoming as a consensus was harder to reach without someone
setting and managing the agenda, distributing tasks and keeping the larger picture
in mind. The weekly group meetings were used to keep in touch and make sure
each project was being consulted in almost the same manner. The meetings also
offered a chance to discuss design decisions and many also took part in doing
heuristic evaluations or cognitive walkthroughs for the others’ designs. These were
early forms of the next year's peer reviews.
The estimated number of hours to spend on the course for the course credits was
______ and the average time the consultants spent on the course was 250 hours. In
general students not involved in UT spend _____ hours on the course. UT
members’ performance was evaluated also in the number of pages they produced
for the projects’ documentation. The average amount produced was 138 pages. It
was considered that the amount of documentation was too much in comparison to
the value derived from the documents so it was made a goal for the next year’s
course to decrease the page amount but increase the value.
3.1.1 Case study: Safety at Work project
In the autumn of 2005 Salpaus Further Education [x] from Lahti, Finland,
approached the Department of Computer Sciences and offered a student project for
the project work course. The idea was to develop a web based learning
environment for studying and getting a work safety card.
Salpaus Further Education had decided to move from normal one-day (8h) course
arranged by a certified trainer to a web based training [old 4,5] and aimed to do
this by taking into account the recommendations for adult education given by the
Ministry of Education [old 1]. The project’s goal was to move the course material
to the web to allow the student an opportunity to study the material and do
exercises independently so that the training would become easy and motivating.
The project’s biggest challenge became ensuring the learning environment’s good
usability as it would potentially have tens of thousands of users whose level of
technical skills varies from a novice to an IT professional. The project group saw it
an easy way to go about the development to use the already existing Moodle
platform [SOURCE] and then developing the needed extra modules for it. Moodle
as such was insufficient in its usability so the usability expert worked together with
the project group to plan enhancements to the interface. These enhancements were
needed to make the course material easily navigable, exercises pleasing and
educative to do with good feedback and making it easy to keep on track of the
learning process.
The usability expert had coincidentally been involved in making the preliminary
requirements specifications for the project. Because of this, it was natural to
involve her in the project meetings from the very beginning. The project group had
a student also keen on usability so the expert in worked together with her to ensure
a good user interface design. The usability documents produced for the project
included usability plan, user interface plan, usability test plan and usability test
report. Additionally, another expert from the UT was consulted for an expert
usability evaluation after the user interface plan had been created.
The project’s culmination point was when the nearly finished product was tested
with several end users in a usability laboratory. The testing was done to see how
real users would react to the concept of studying the material online and to spot all
remaining usability flaws in the user interface. Normally the users are expected to
study the material and work on the exercises for multiple hours before going to an
exam so in the test it was only possible to try briefly see how the users felt about
the learning environment and whether they could navigate around it and find all
necessary items.
Paragraph about findings! For many immigrants taking the course, it may be easier
to study the material independently as they have time to study at their own pace
and they can go back easily if they do not understand the material.
Most problems found in the testing were fixed before releasing the product for
further development at Salpaus. Many other projects decided to skip the formal
usability testing in a laboratory but in the Safety at Work project it was seen a
meaningful phase in delivering a good quality product.
3.2 Usability team 2006-2007
The first usability team worked on the course in the year 2005-2006, during which
the different practices for the team were tested. The current (year 2006-2007)
team's practices are based on the experiences from the last year's pilot team.
The number of usability team members was increased to 12 members, and 2
projects managers were assigned to the team. The team was split into English and
Finnish group. The English group deals with the usability issues of 8 English
projects and the Finnish group deals with 10 Finnish projects. Each usability
member is assigned to one or two projects, but they may also do smaller projecttype work to help other projects. The amount of involvement depends on the
projects: some projects may not even have a user interface, and some project
groups already have usability experts working in their own group.
3.2.1 Usability Team Structure
Fundamentally, there are two basic alternatives for structuring a usability group
within an organization:

Centralized organizational structure: Usability experts belong to one
organizational unit such as the Usability team, and have their own
usability manager (See Figure 3). When project teams request usability
assistance, one or more usability members work closely with the team
during the project.

Distributed organizational structure: Usability experts are assigned to
work on separate project teams and report to project manager (See Figure
4). On many projects, this means only one usability expert per project. [6]
We used the Distributed organizational structure for structuring the usability team
since it allows the usability expert to work together and closely with the project
development team members. Staying with the project team from the beginning to
end of the project may increase the chances of usability recommendations being
implemented.
3.2.1 Case Study: SysMLL Project (2006-2007)
Background: SysMLL is a web based multiple languages learning software. The
“Extension of SysMLL” project is further development of existing working version
of SysMLL. Original SysMLL project was carried out in year 2005-2006 and it is
now operational. Extension of SysMLL is further development project of SysMLL
and project will continue until May 2007. The client of the project is “School of
Modern Languages and Translation Studies” of University of Tampere.
One of the English usability experts was assigned to work closely with this project
team and report to the usability team project manager.
Project Goals: Main goal of this project is to add more functionality to SysMLL.
Incremental method is used for this project, so each phase has a new goal. First
goal is achieved by adding multimedia to system and in future phases by adding
exercises and games.
Usability Goals: Since this project uses the Incremental model, in each phase the
usability expert has to look into the usability issues, conduct usability tests, and
create usability documents and reports.
Time Line: As incremental model is used for the project, each phase is planned to
last only for a short period. Group will continue adding increments to SysMLL
until the end of the course period (May 2007). As the final increment is ready in
April-May, final documentation and presentations are done, the project will be
ended.
Usability test
In the beginning of the project, a usability test was arranged in the Computer
Science Department's usability laboratory in February.
The test consisted of three phases. Each participant started by filling a brief
questionnaire. During the main phase of the test, each participant worked on the
given test tasks on the computer. This section took around 50 minutes. The
interview in the end took up about 10 minutes.
Before beginning the usability test, the usability expert explained the participants
how to use the think-aloud protocol when working on the tasks. Each participant
was asked for a permission to record the test on a video. Then test tasks were given
one by one to the participants.
Mozilla web browser was used in the test because it was also used in the heuristic
evaluation. There was a moderator and observer throughout the tests to monitor
and record the findings. In addition, a technical support assistant was available
during the test period to take care of the technical problems of the lab and the test
application.
Participants
There were four participants, two females and two males. Three of them had over
ten years of computer experience while the other had less than 10 years of
experience. Two of them never used the SYSMLL before the testing while the
other two had used the system for less than one year. They all had some kind of
experience with other web-based language learning applications.
Test tasks
Focused was on evaluating two major functions in the SYSMLL application. These
two functions are related to the exercise taking the data editing. However, test tasks
indirectly touched other functions of the system such as settings, user management
and help. The test was targeted at students and teachers of the system.
Test tasks were selected according to the usability expert’s focus. The moderator
gave one task one at a time on a slip of paper to the participants. Tasks were
regarded as completed when participants got the result required by tasks and
usually said, “I think I have completed the task.” or, when they said, “I do not
know how to continue, I want to give up”.
Findings from the usability tests
The user tests found out that the design of navigation menu of SYSMLL is good
and clear except that a few terms such as “Edit Data” and “Words and Equivalent
2” are uninformative.
Most usability problems found are related to the system feedback, interface layout
and button designs. In addition, there are some confusing concepts used in the
system such as “Meaning”, “Topic” and “Name”, which appear quite often here
and there.
Findings from the interviews
All participants expressed in interviews that SYSMLL might be a helpful exercise
application for learning languages if it is easy to use.
Most of the participants felt that the most frustrating part with SYSMLL is that it
does not seem to provide any feedback to users during any operation. They are not
sure if some action is correct or complete. Secondly, in many situations, the
instructions given by the interface do not support effective operations at all.
They all expressed that if there are dictionary like function in the system, it will be
more helpful in learning new languages.
3.2.2.4 Future Hopes
Lessons Learned: From the usability evaluations and tests it was found that the
application needed many improvements in usability related issues. These usability
problems could not have noticed without the usability test. Therefore, it is very
beneficial to have a usability expert in the team to work closely with the project
team to improve the products usability.
Future Plans: Now the project is stepping into the final incremental phrase and
more usability planning, evaluations and tests are to be planned and done.
4.0 Conclusions
Here we introduce usability team -concept and how it is related on course and
project's phases. We should use references like [1], but since these do not change
dynamically in word -documents, we can fix them just before submission...
5.0 References
1.
Abran, A., Moore, J. , Bourque, P., Dupuis, R., and Tripp, L.: Guide to
the Software Engineering Body of Knowledge – SWEBOK, trial version,
IEEE-Computer Society Press, 2004. http://www.swebok.org
2.
Dalcher, D. and Woodman, M.: Working together: software engineering
in a group context, in Proceedings of INSPIRE VIII, 75-87, 2003.
3.
ECTS Study Guide Curricula 2006-2007, University of Tampere, Faculty
of
Information
Sciences,
2006.
http://www.uta.fi/studies/ects/information.pdf
4.
Nielsen J, Designing Web Usability: The Practice of Simplicity. New
Riders Publishing, 2004.
5.
Nielsen, J, http://www.useit.com/ (visited April 2007)
6.
Mahmood, Z. and Rashid, K.: Group working: an essential component of
software engineering education, in Proceedings of INSPIRE VIII, 89-99,
2003.
7.
Pressman, RS. Software Engineering – A Practitioner's Approach.
McGraw-Hill 2005.
8.
Poranen, T. (editor). Report D-2006-6: Software Projects 2006 (partially
in Finnish), http://www.cs.uta.fi/reports/dsarja/D-2006-6.pdf
9.
Hakola J, Hautaniemi J, Parviainen V, Peltomäki S and Poranen T,
Käytettävyys ja oppiminen: SAFETY-projektissa toteutettiin Moodlepohjainen oppimisympäristö työturvallisuuskortin suorittamiseksi
verkossa. Pedaforum, 2006.
10. Salpaus Further Education. http://www.salpaus.fi/ (visited March 2007)
11. SE 2004 - Curriculum Guidelines for Undergraduate Degree Programs in
Software Engineering. http://sites.computer.org/ccse/
12. Work
safety
card
training
brochure.
(in
Finnish)
http://www.lamk.fi/material/tyoturvallisuuskortti_2_4_07_esite.doc
(visited March 2007)
13. Moodle course management system. <www.moodle.org>
Appendix
Table 1: Test tasks and the rationale for each task.
Task 1: I want to learn Finnish language from SYSMLL using English as a
reference.
The purpose of this task is to find out how the users will make the setting of the
system in order to learn between pair of languages.
Task 2: I want to add a new title “Animal” about vocabulary in the system.
The purpose of this task is to find out how the users will work with the function
of editing new meaning/topic into the system.
Task 3: I want to create two photos about tiger into the system.
The purpose of this task is to find out how the users will work with the function
of adding new images about some meaning into the system in the database
editor.
Task 4: I want to take away the first photo about” tiger” from the system.
The purpose of this task is to find out how the users will work with the function
of deleting images about some meaning from the system in the database editor.
Task 5: I want to create Chinese language into the system.
The purpose of this task is to find out how easily and the way the users can add
a new kind of language into the system.
Task 6: I want to check if I know how to write “Monday” in Finnish.
The purpose of this task is to find out how easily and the way the users can take
a quiz about vocabulary between pair of languages.
Task 7: I want to learn the vocabulary from English language to Finnish
language in the system.
The purpose of this task is to find out how easily and the way the users can learn
and browse vocabulary between pair of language in the system.
Task 8: I want to know how much I understand the meanings of some Finnish
words.
The purpose of this task is to find out how easily and the way the users can take
quiz in order to check their understanding about Finnish definitions.
Task 9: I want to use English words as reference to read through the available
Finnish words in the system.
The purpose of this task is to find out how easily and the way the users can
browse vocabulary of pair languages.
Questionnaires and interviews
Questionnaires were used to collect background information about the participants
(e.g., the level of experience in computer and web use, previous experiences with
SYSMLL or other language e-learning applications etc.). The completed
questionnaire will be attached to the final report of user test for future reference.
Below is the list of questionnaires prepared for this user tests.
Choose one of the following answers to each question:
1. My gender:
o Female
o Male.
2. My computer experience:
o Over 10 years
o Between 5 to 10 years
o Less than 5 years.
3. I use computer:
o Over 40 hours a week
o Between 25 and 40 hours a week
o Less than 25 hours.
4. I have been using SYSMLL for:
o Over one year
o Less than one year
o Never use.
5. I use SYSMLL for:
o More than three hours a week
o Between one and three hours a week
o Sometimes not at all.
6. I have experience with other language programs over the web:
o Very much
o A little bit
o Nothing.
Interviews
The following open-ended and close-ended interview questions were used to
collect users’ subjective opinion about the user interface usability of SYSMLL.
1. Which task is easiest for you during the test?
2. Which task is the most difficult for you?
3. What is your general opinion about the interfaces presented in the test?
4. Suggestion for improvement? (Show interface of “Data editor”, “Exercises”
during conversation)
3.2.2.3 Results
Task time and completion rates
As mentioned earlier, the participant’s test were recorded and video taped. It was
recorded that participants used around 50-60 minutes to go through all the nine
tasks.
The completion rates of the four participants are displayed below according to the
notes taken by the moderator and the observer.
Usability Test Results
Task Completion rates
120 %
100 %
80 %
60 %
Series1
40 %
20 %
0%
1
2
3
4
5
Test tasks
6
7
8
9
TO DO -LIST:
- The SysMLL case study needs shortening, and the test sheets should be moved
into appendixes
- Description about what is usability? In chapter 2?
- How usability is taken into account in other Finnish universities, and other
countries?
- Conclusions