Report for:
DIA
alpha.newzealand.govt.nz User Testing
Date
Version
Prepared for
Prepared by
Email
Phone
Mobile
7 February 2016
1.0
Department of Internal Affairs
Kristina Nink, Martin Bulmer,
Michael Dutton
Kristina@optimalusability.com
04 498 5743
021 025 32402
Optimal Usability Limited
Level 2
126 Cuba Street
PO Box 24-032
Wellington 6011
New Zealand
www.optimalusability.com
Optimal Usability
DIA – alpha.newzealand.govt.nz
2
Summary alpha.newzealand.govt.nz User Testing
Overview
The objectives of this project were to discover usability and user experience problems with the early
version of the alpha.newzealand.govt.nz website and to make recommendations for improvement.
Summary
The review revealed 28 usability findings. The following issues were identified:
•
•
•
4 major or high priority issues requiring urgent attention,
11 minor issues which are less pressing,
10 observations or cosmetic issues.
Additionally, 3 Positives were identified.
Participants
16 participants were recruited for this user test (including one additional participant who had
mistaken his time and attended a week earlier. He tested the site in between other users):
•
•
•
•
•
three from each of the following age brackets, 18-29, 30-39, 40-49 and 50-59 and 60+ (and
one additional in the 18-29 age bracket)
nine female and seven male
all had used some government websites before
a range of experience using the internet and spread of high and low frequent internet users
a range of income levels.
User Testing Process
The user tests were carried out at BIZDojo on the 31 January with six participants, on the 12
February with four scheduled and one additional user and on the 19 February 2013 with five
participants.
Kristina Nink, Martin Bulmer and Michael Dutton from Optimal Usability facilitated the 60 minute
sessions.
Each test session started with a basic introduction and the facilitator collected some background
information about each participant.
Participants were then given a series of tasks to complete on the alpha.newzealand.govt.nz website.
In the first round of testing three participants completed the tasks on version A and three on version
B of the site. In round two, all participants tested version A of the site, which was modified based on
findings of the first round. And in the last round, all participants tested version B of the site featuring
the same modifications that were created for version A after the first round of testing.
The session concluded with several review questions about the site and users were asked to
comment on some design mock-ups.
Optimal Usability
DIA – alpha.newzealand.govt.nz
3
Key Findings and Recommendations
•
Few participants could identify easily where they were and who owned the site. Several
thought initially they were on the civil defence site. Confusion about the site ownership and
purpose might compromise the credibility of the site.
R Consider making the owner and purpose of the site more apparent by visually
separating announcements from the top of the page to help people understand where
they are.
•
Few participants opted to use the site categories, preferring to use the search. This may be
partially due to the fact that the categories are mostly not visible, being below the fold.
•
There were mixed responses to the search filters, with some participants not understanding at
all what they were about and others suggesting they knew what they were, but would not use
them on this site. Many of the participants were overwhelmed by the filter options.
R Consider decreasing the number of different filters to make these less overwhelming to
users. Make the filtering mechanism much clearer and consistent. Some filters appear
to be links others appear to be checkboxes. Make it clear that these are filter options.
•
Most participants initially overlooked the action button, searching through the page for a link
to a form and often clicking the in-body links instead.
R Consider reformatting the button to make it more visible.
R Consider placing the action button closer to the 'call to action' point, i.e. where the page
has explained to the user his or her next course of action (e.g. right after the
explanatory text).
•
Participants did not mind leaving the site, as long as they were directed to a government site
and to a page they felt was answering their questions. Some participants noticed they were on
another site, some did not.
•
Where there was a step by step list for users to complete, several participants expected links to
further content or necessary forms.
R Ensure content is written to be scanable.
•
Only a few participants noticed the ‘More like this’ section underneath the main content. One
thought it was other services provided by the department the page he was on related to.
Another commented that the links didn’t relate well enough to what he was looking at.
R Consider the placement of the ‘More like this’ links.
R Consider increasing the ‘More like this’ header to increase its visibility.
•
Several participants remarked that the consistency of the tabs made the content pages easy to
use.
•
Several participants hesitated to click contact, missed it or didn’t think it was applicable to
them.
R Consider rephrasing the tab. ‘Contacts’ or ‘Contact us’ is often used for a site-wide
contact, which could contribute to the users’ confusion.
Optimal Usability
DIA – alpha.newzealand.govt.nz
4
R One user suggested placing the contacts in the body content.
•
Some participants commented that the contact details for some pages were too general others
praised the contacts and some participants were confused about the contacts provided.
R Carefully tailor the content under contacts to the specific page and consider the impact
of the introductory sentence on users. Distinguish between an address people need to
drop off a form to, a contact they can use to get more information and general contact
details for the department responsible for this specific action.
•
Several participants commented positively on the ‘Help with rent and accommodation costs’
quick answer page, because it provided them with links to all options and a short summary of
what each was about. They repeatedly referred back to this page to switch between options.
Participants found this page easy to scan and found identifying their next suitable action easy
from amongst the suggested links.
R Consider using the approach demonstrated here across the site.
R Consider alternatives that are in common use by users as opposed to the often more
difficult to comprehend government-speak.
Optimal Usability
DIA – alpha.newzealand.govt.nz
5
User Testing General Findings
Pre Questionnaire
How much time do you spend online?
<1 hour per day
~1 hour per day
2-3 hours per day
One participant said she was looking for jobs and spending more time online than normally.
What government websites have you used before?
Participants mentioned the following websites: IRD, Studylink, city council websites, DIA, NZTA, MFAT,
school websites, government job search site, MetLink, Work and Income and ACC.
What did you think of your experience with those sites?



How do you usually go about finding information online?
I google it
Optimal Usability
DIA – alpha.newzealand.govt.nz
6
Tasks - General Observations
•
Participants all gravitated to using search, no matter where they started the task. Almost all
participants used the ‘suggested search term’ dropdown every time they searched for something.
They used the search recommendations to ‘sanity check’ their searches before committing them,
“21 results for Police Clearance, yes that’s good” or because it “saves you typing”. All mentioned
they like it.
•
Those participants who noticed the footer or the keyword link at the top and successfully used
these to find what they were looking for, tended to use this method again in the later tasks.
•
After completing a few tasks, most participants realised that the site was a gateway/portal to
government information and commented that it is a good idea. “This is an everything website.
That's good. This is a good idea this site, knowing where to go to is half the problem.”
•
Those participants who noticed when links from the test site directed them to other sites were not
fazed, provided the destination site was a government site that answered their question. Many of the
participants did not notice they were on a different site until it was pointed out to them.
•
Participants commented that the amount of white space around the content meant that the site
was easier to use. “This is better than a lot of government websites that I've seen, because it’s
cleaner.” “I think it's, this is quite good, nice and clear, straight to the point.”
•
Some participant tended to start thinking about what department would offer the service they
were looking for before actually searching for the task/action. Several users checked and noticed
the department once they had found their answers – “Ah, the Ministry of Justice does that” – others
did not.
•
Participants tended to scan content for something to “jump out at them”.
•
The official wording of some concepts seemed to be confusing, e.g. police clearance vs. criminal
record, help with accommodation vs. supplement. Providing two links for a concept, e.g.
'Supplement vs. Help with accommodation' did nothing to reduce confusion. Many participants
agonised over which of the two options to select and would have preferred the site to disambiguate
the situation for them.
•
Many participants stated they prefer to complete forms online whenever possible rather than
printing them and sending them in/scanning and emailing.
Optimal Usability
DIA – alpha.newzealand.govt.nz
Post Questionnaire
What did you like about the site?
“The search was really good.”
“It was pretty good, the next thing you needed was most often right there.”
“The consistent layout on each page was good.” “The tabs are quite good; it’s got similar things in
them.”
“It’s professional, it’s pretty straight forward.”
“The info that was needed was right in there, so there was no need to search further.”
“I like the cleaner layout compared to some of the websites it links through to.”
What would you change about the site?
“Maybe change the colour of the button to make it easier to see?”
“I’m still not clear on what site I am looking at.”
“No I think it is quite good, but it does make me nervous when it says this organisation has its own
site, which one is up to date?”
“It seems very plain, not really aesthetically pleasing.”
“I wouldn’t change a whole lot. But there’s work to be done to make it a bit more presentable.”
“I would make the filters a bit clearer on how to use them.”
In what category would you put information about NZ Daylight Savings?
10
8
2
2
–
–
–
–
Key Dates
New Zealand Calendar
Weather and Climate
About New Zealand
Some participants picked more than one category.
7
Optimal Usability
DIA – alpha.newzealand.govt.nz
8
Homepage
1
3
4
2
1
Few participants could identify easily where they were and who owned the site. Several
thought initially they were on the civil defence site. Confusion about the site ownership and
purpose might compromise the credibility of the site.
“Have an introduction so that we know what it’s about, this page [home] is giving off mixed
messages. More territorial thing saying welcome to this site.”
“There’s no explanation what this [civil defence] is, a title or so, like ‘important information’,
would be good.”
R Consider making the owner and purpose of the site more apparent by visually
separating announcements from the top of the page to help people understand where
they are.
2
Few participants opted to use the site categories, preferring to use the search. This may be
partially due to the fact that the categories are mostly not visible, being below the fold.
Almost all participants used the ‘suggested search term’ dropdown every time they searched
for something. They used the search recommendations to ‘sanity check’ their searches before
committing them “21 results for Police Clearance, yes that’s good” or mentioned that it “saves
you typing”. All suggested they expected and liked it.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
Some participants commented that the suggested search results text was too small and too
cramped.
“That's what you'd expect these days. If nothing similar appears underneath you think you
might be in the wrong place, it's reassuring to have that”
R There might be opportunities
to improve this by adding
categorisations, e.g. apple.com's search.
R Make the text bigger and better spaced as
participants sometimes struggled to read it.
R
3
One participant started typing immediately, expecting
the search field to be in focus.
R Consider making the search field active on loading the page, so that people can start
typing without having to click in the text field first, an action users might be used to
from using Google.
4
Several participants used the links in the “Featured today” section. One suggested to rename
the section “Featured” and populate it dynamically based on pages most frequently visited.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
9
Optimal Usability
DIA – alpha.newzealand.govt.nz
10
Search – Version B
5
8
6
7
5
There were mixed responses to the search filters, with some participants not understanding at
all what they were about and others suggesting they knew what they were, but would not use
them on this site. Many of the participants were overwhelmed by the filter options.
No preference or difference in use was observed between filters placed on the right or on left
of the search results.
“I wouldn't usually use that, I would change my search term usually to narrow things down”
“My experience is that if isn't in the first few results, filtering is not going to help finding it”
R Consider decreasing the number of different filters, e.g. only to ‘Find by keyword’ and
‘Find by service provider’ (these were the most frequently used options in the test) to
make these less overwhelming to users. Make the filtering mechanism much clearer
and consistent. Some filters appear to be links others appear to be checkboxes. Make it
clear that these are filter options.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
6
11
Several users commented about the quality of the search results, with some users
commending it and other suggesting too many unnecessary results were returned.
“Again a lot of words, lots of places to go. Was it related to my question? It says health and all
that here and I’m not sure that is related to my question.”
R Improve/narrow the search.
7
Some users struggled with the size of the search field when trying to refine their existing
search, e.g. adding another word.
R Consider making the search field wider.
8
None of the participants noticed the Google search at the top of the page.
R Consider removing the Google search to simplify the page.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
12
Content Page – Layout B
9
9
10
11
Most participants initially overlooked the action button, searching through the page for a link
to a form and often clicking the in-body links instead.
“Not sure why I've not noticed it, maybe because of the dark colour.”
“The button is not quite as clear as the text [under ‘what you need to do’] and link in there.”
(Round 2)
R Consider reformatting the button to make it more visible.
R Consider placing the action button closer to the 'call to action' point, i.e. where the page
has explained to the user his or her next course of action (e.g. right after the
explanatory text).
10
Some participants were a bit startled when the action button linked them through a page
rather than the form directly (especially when in a previous task the button did link directly to
a form).
“Oh I expected a form [form is on page], ah there it is, but I did expect the link should be a
form.”
R Keep functionality of the button consistent or indicate through the button language
what will happen (whether it links to a PDF or page).
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
11
13
Participants did not mind leaving the site, as long as they were directed to a government site
and to a page they felt was answering their questions. Some participants noticed they were on
another site, some did not.
“Didn't bother me, I'd rather be taken straight to the correct page than search for it.”
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
14
12
12
Only three participants overall mentioned the feedback tab. One didn’t think she would be
taken seriously, so she wouldn’t fill it in. Another said he might provide feedback on a new
site, “just to be helpful”, but not otherwise. A third participant was confused about whether
the feedback would be sent to an editor or posted directly on the site.
She also questioned why the rating wasn’t at the bottom of the article.
“Hopefully this is an email, but it looks like you can just post things.”
“Why isn't this down the bottom?”
R Consider whether the feedback functionality is necessary for the site and whether this is
the right placement of the functionality.
R Indicate what happens when users complete the feedback form, i.e. whether it sends an
email to the editor of the site or posts a public comment on the page.
R To relate star rating better to the page content, consider placing it at the bottom of the
main body content and displaying it no matter what tab the users are visiting.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
15
13
13
The participant who entered feedback on the site was confused after hitting ‘Submit’, she did
not notice the feedback at the top of the page.
“Huh, where did it go?”
R Consider making the feedback more visible and displaying it closer to where the original
feedback was located, to help people understand what happened.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
16
14
15
17
16
14
All apart from a few participants scanned a page for content to “jump out at them”.
Where there was a step by step list for users to complete, several participants expected links
to further content or necessary forms.
R Ensure content is written to be scanable.
15
Only a few participants noticed the ‘More like this’ section underneath the main content. One
participant thought it was other services provided by the department the page he was on
related to. Another participant commented that the links didn’t relate well enough to what he
was looking at.
R Consider the placement of the ‘More like this’ links (more participants used these links
to navigate between content when they were placed on the right hand side in round
one).
R Consider increasing the ‘More like this’ header to increase its visibility.
16
Several participants scrolled all the way to the bottom (especially when they were not sure
whether they were in the right place) and ended up using the ‘Explore
alpha.newzealand.co.nz’ footer. A few participants continued to use the footer to browse
content whenever they were stuck.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
17
Several participants used the ‘Keywords’ link to browse for content and continued using the
keywords after they successfully found what they were looking for in a previous task. Some
participants tried ‘Services’, but decided ‘Keywords’ was giving them better results.
More participants noticed the ‘Browse and find’ links in the footer than in the header.
R Consider the value of the ‘Services’, possibly consider removing it to keep site cleaner.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
17
Optimal Usability
DIA – alpha.newzealand.govt.nz
Content Page – Layout B – Contacts
20
19
18
20
20
18
Several participants remarked that the consistency of the tabs made the content pages easy
to use.
“Consistency between topics how they have the ‘what you need to do’ and ‘where to go’ was
good.”
19
Several participants hesitated to click contact, missed it or didn’t think it was applicable to
them.
“Yes I think you could miss that [contacts], some people could be confused by the [specificity]
of the contacts. It should say contact information for this organisation...or something like
that.”
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
18
Optimal Usability
DIA – alpha.newzealand.govt.nz
“Contact is not quite the right word.” (Round one)
R Consider rephrasing the tab. ‘Contacts’ or ‘Contact us’ is often used for a site-wide
contact, which could contribute to the users’ confusion.
R One user suggested placing the contacts in the body content.
“Contact us, feedback etc. why isn't this down the bottom? [points to the bottom of the body
content section]”
20
Some participants commented that the contact details for some pages were too general
others praised the contacts and some participants were confused about the contacts
provided.
“That's very specific, they haven't thrown in the police and all that crap, that's better, it’s very
specific and not bogged down.” [police clearance page]
“The contact should be about paying a fine only, not a general contact.” [pay traffic fine page,
re national police headquarters]
“But this contact doesn’t tell me it’s for jury services. Legal services, well that could be it, but
I’m not sure.” [jury service page]
R Carefully tailor the content under contacts to the specific page and consider the impact
of the introductory sentence on users. Distinguish between an address people need to
drop off a form to, a contact they can use to get more information and general contact
details for the department responsible for this specific action.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
19
Optimal Usability
DIA – alpha.newzealand.govt.nz
20
Keyword Pages
21
21
A few participants did not notice the A-Z links and scrolled past the keywords content to the
larger links in the footer.
“Keywords is more the content, I find that more useful.”
R Consider visually highlighting the A-Z row
R And consider making the footer visually distinctive from the main page content to avoid
confusion.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
21
22
22
One participant commented on the sorting of the keywords results page, saying that
“alphabetically would make more sense”.
R Consider indicating to users how content has been sorted and give them the controls to
select another sort, e.g. 'Results sorted by: Relevance’ (with a drop-down to select e.g.
alphabetical).
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
Section/Topic Pages
23
24
23
Participants commented on the repeated image and mentioned that they were comfortable
using the slider for ‘Name changes’.
24
Two participants used the Google search at the bottom of the page, thinking it was a site
search.
R Consider replacing the Google search with the internal site search. At this point users
were still happy to search this site for the answers to their questions, they were not
expecting to move to other sites.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
22
Optimal Usability
DIA – alpha.newzealand.govt.nz
23
Specific Page Feedback
25
25
Several participants commented that the ‘Police’ page was “messy” and unprofessional
looking.
“This isn't as well laid out as the other pages. Just looks like someone's typed it up in Word. It’s
a bit messy.”
“I don't really like the different colours and different fonts.”
R Consider keeping the font-styles used on the page consistent and minimise the number
of different styles and colours used
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
Optimal Usability
DIA – alpha.newzealand.govt.nz
26
26
Several participants commented positively on the ‘Help with rent and accommodation costs’
quick answer page, because it provided them with links to all options and a short summary of
what each was about. They repeatedly referred back to this page to switch between options.
Participants found this page easy to scan and found identifying their next suitable action easy
from amongst the suggested links.
Several participants selected this Google search result over the Accommodation supplement,
because “it sounds exactly like what I am looking for”.
“It is very clear and user friendly.”
R Consider using the approach demonstrated here across the site.
R Consider alternatives that are in common use by users as opposed to the often more
difficult to comprehend government-speak.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
24
Optimal Usability
DIA – alpha.newzealand.govt.nz
27
27
In general, when participants were looking for “name change” for their drivers licence, they
scanned pages for that expression and discounted information, forms and links that only
stated “replace licence”, which they thought was not applicable.
“That’s only when you lose it”.
R Carefully choose wording of action buttons and headers, using the terminology
expected by the user whenever possible or explain differing terminology.
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
25
Optimal Usability
DIA – alpha.newzealand.govt.nz
28
28
No one noticed the ‘In this section’ drop-down underneath the section headlines.
R Consider removing this option to keep the site cleaner
Positive Finding
Minor Finding
Major Finding
Observation
R Recommendation
26
Optimal Usability
DIA – alpha.newzealand.govt.nz
Search Terms
Google Search Terms
Task 1 – low income, housing support
What did participants search for?
•
•
•
•
•
•
•
•
•
•
•
WINZ (twice)
Work and Income
WINZ accommodation supplement
rental properties
local body rates
Kapiti Council
rent assistance New Zealand
budgeting advice
tenancy financial Hardship
help paying rent
accommodation
Which option did participants select?
•
•
•
Six users clicked the second link ,‘Help with accommodation costs’, “because it was clearer,
seems more personal.”
Two users selected the link directly to Work and Income.
Two users selected the first link, ‘Accommodation supplement’.
Task 8 – Jury duty
What did participants search for?
•
•
•
•
•
•
•
•
Jury Duty (three times)
Jury Service (three times)
Jury Duty New Zealand
jury duty non attendance
jury (selected jury information from drop down)
can’t attend jury duty in New Zealand
Justice Department New Zealand
jury service Wellington region.
27
Optimal Usability
DIA – alpha.newzealand.govt.nz
Which option did participants select?
•
•
•
Five users picked the first link, ‘Jury service – newzealand.govt.nz’ on
alpha.newzealand.govt.nz.
Three users selected the second link, ‘Jury service – Ministry of Justice’.
Three users selected the fourth link, ‘If you can’t attend jury service – newzealand.govt.nz’
Site Search Terms Used
Task 2
What did participants search for?
•
•
•
•
•
•
•
•
Phone scams (twice)
hoax, virus scams
virus
stop unwanted spam
unsolicited calls
computer virus (auto complete)
fraud, internet scams
help with virus calls
Which option did participants select?
•
•
•
Two users clicked ‘Recent phone scams’
Two users clicked ‘Phone scams’
One user used the keywords filter to narrow her results
Task 3
What did participants search for?
•
•
•
•
Redundancy (eight times)
human resources
legal advice
labour department
Which option did participants select?
•
•
Eight users clicked ‘Guide to redundancy’ (first result when searching for redundancy)
One user clicked ‘get help with an employment issue’
28
Optimal Usability
DIA – alpha.newzealand.govt.nz
Task 4
What did participants search for?
•
•
•
•
•
•
•
•
Police clearance (seven times)
clearance (twice)
police clearance New Zealand
police clearance report
police clearance your
police check
vetting and vouching
one user typed her name
Which option did participants select?
•
Eight users clicked ‘Get a copy of your police clearance report and criminal record’ (first
result when searching police clearance)
Task 5
What did participants search for?
•
•
Name change (three times)
change name on your driver’s licence
Which option did participants select?
•
Four users clicked ‘Change name on driver licence’ (first result when searching name
change)
Task 6
What did participants search for?
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Child service(s) (twice)
children services (twice)
child assistance
government child assistance
government child assistance programme
“government child assistance programme”
children government initiative
child support
child support services
help with childcare
immigration
children
child benefits
children agencies
school support
29
Optimal Usability
DIA – alpha.newzealand.govt.nz
Task 7
What did participants search for?
•
•
•
•
Paying traffic fine (twice)
parking fine
AA
A A (with a space in between)
Which option did participants select?
•
•
•
Two users selected ‘Pay traffic fine or infringement’
One user clicked ‘Pay fine’
One user clicked ‘Automobile Association’
30