20 November 2009
Beta Testing
Mike Booth / mxb17u
1
Contents
Introduction ............................................................................................................................................ 3
Where does it fit into the testing cycle? ................................................................................................. 3
What actually is beta testing? ................................................................................................................. 4
History ................................................................................................................................................. 4
Beta Software...................................................................................................................................... 4
Beta Testers ........................................................................................................................................ 4
The steps of beta testing..................................................................................................................... 6
Examples of well known beta tests......................................................................................................... 6
Microsoft ............................................................................................................................................. 6
MMORPGs (Massively Multiplayer Online Role Playing Games) ........................................................ 7
Netscape ............................................................................................................................................. 7
Issues ....................................................................................................................................................... 7
Conclusion ............................................................................................................................................... 8
References .............................................................................................................................................. 9
2
Beta Testing
Introduction
Many people have come across beta testing in real life situations – it could have been for one of
your favourite games or it could have been for your favourite operating systems. Most people
understand beta testing as a pre-release version of software - but what does it actually involve?
As a definition then beta testing can be seen as “allowing the software to undergo usability
testing with users who provide feedback, so that any malfunctions these users find in the software
can be reported to the developers and fixed.”1. This idea has proved highly useful in larger scale
software products as it can identify areas of software improvement while the project is still in its
development stage.
The main point about beta testing is that, in theory, it carries more validity than other forms of
testing as it is real people, in real situations using the software in a real environment. Whether this is
true appears to depend on the different situations that beta testing is used in. According to Joseph S.
Dumas and Janice Redish’s book “A Practical Guide to Usability Testing” then beta testing “seldom
yields useful information about usability. Most companies have found beta testing too little, too
unsystematic, and much too late...”. 2. This however is in contrast with actual company policies such
as Microsoft, who when developing both their Windows 7 and Vista operating systems have used
beta testing to help gain real user feedback.
Where does it fit into the testing cycle?
Beta Testing is performed as the last part of the testing cycle as all software has previously
undergone a full cycle of internal testing, so that the application is sufficiently stable within different
environments.
To aid in illustrating how beta testing fits into the testing cycle:
Unit Testing
Integration Testing
System Testing
System Integration Testing
Regression Testing
Acceptance Testing
Alpha Testing
Beta Testing
3
As you can see then beta testing is performed after Alpha testing, which is where a controlled set of
users test the system. Beta testing is the step on from this where a larger set of users are given the
application to use in a real environment.
What actually is beta testing?
History
The term “Beta” test stems from IBM’s “hardware test convention”, which dates back to their
magnetic tape machines used in the 1970s.3. The hardware was first tested on a smaller scale to
show manufacturing feasibility, known as an “alpha” test. This was then extended to a larger scale
and given to non-developers to test the functions of the hardware. This also meant that the
manufacturing of the types of scale necessary could be tested.
Beta Software
Beta software itself usually contains all of the functionality that will be placed in the release version
but does not guarantee that the software will be error free – this is known as the beta version. This
allows users to use the software and test all areas of the system and help identify any errors that
have been overlooked but it also allows the developer to test how the system will integrate into a
larger, unstructured environment. This is known as the supportability of the software. 4
Since the software is not guaranteed to be error free then the developer will abstain from taking any
responsibility what so ever for any data loss due to using the software and the user will use the
software at their own risk.
In addition to the software then because this will be the first introduction of the software to the user
then the developer will usually contain documentation on how to use the software or a list of
features that the software has. This is not done all of the time and if the developer chooses not to
include this then it is not a necessity.
As well as all of this then the software should have documentation, even if this is not given to the
user. This will include release notes containing information such as:





The exact version number
List of changes since the previous version
Descriptions of any known errors
A full list of system requirements
Hardware (and other software) requirements
Beta Testers
A big problem with beta testing is finding people that are willing to test the software. Gaining
peoples feedback on a piece of software is very difficult because many just want to experience the
software and not provide feedback to the developers.
The job of the user is to do just that – use the software and then ideally provide feedback on their
experience as well as to feedback any bugs or errors that they encounter while using the software.
Feedback that the user should provide in addition to any errors that they find could be:
4


Their current hardware configuration that will run the software – this is needed so that the
developer knows that their software is being tested on a wide range and distribution of
systems.
Any improvements that could be made to the software – this is so that the developer can
make any minor improvements on the software before release.
There are actually 2 different types of Beta Tests:


An Open Beta
A Closed Beta
An open beta is one where any person can access the software and there is no limit on the amount
of people that can use the software. A closed beta is one where the developer controls the number
of people that are allowed to use the software. The latter is considered more useful as open betas
tend to either get too many users that you don’t have the time or the resources to be able to control
the feedback data coming back in, or too few testers and you don’t get enough feedback data. This is
an observation that has extended into real beta tests – using an example of 2 large Beta tests,
Microsoft Windows 7 OS Beta and the World of Warcraft MMORPG Beta then both used closed beta
tests to control the flow of users. 5. 6.
The inherent nature of closed beta tests also gives another advantage in that most people that wish
to try and take part, means that they are more likely to give feedback on the software. Since a user
must apply to use the software then the developer can have the user consent to giving feedback on
the system, which again increases the amount of feedback.
A closed beta still has issues as many beta testers will give feedback at the start using the software
but will tail off towards the end of the testing cycle.
Below is a graph of feedback verses time done for a piece of software called “Softner 1.2”7.
5
As you can see then for the first stage of the beta test then the testers produced a large amount of
feedback and this tails off drastically for the latter stages of the test.
From this same piece of software then it was found that only 55% of the testers actually gave
feedback on the software.7.
How the feedback is given is now much easier thanks to advancements using the Internet, it means
that most software can take advantage of any errors by directly sending feedback reports to the
developers web servers. This is not however a necessity and any other messaging system can be
used such as forums, blogs, instant messaging etc.
The steps of beta testing
1. Define a time frame for your Beta test. Usually a Beta test can’t run too long but needs to be
long enough to gain sufficient feedback. Depending on the size of the piece of software then
the time frame can vary. Having looked at the length of several beta tests, including
Windows 7, then I can say that at least 4 weeks is needed for a moderately sized piece of
software. 8.
2. Decide on whether it should be a closed beta or an open beta. If it is a closed beta then you
will to decide on a way to vet your users so that you get a wide variety of users in different
environments. A good way is to use a subscription method to allow users who are willing to
go out of their way to sign up, as it is these users that are more willing to give feedback.
3. Give a method for allowing feedback for users. An atypical method is a direct messaging
system built into the software (Microsoft Connect) but others are available such as a forum
messaging system. 9.
4. Gather the feedback and analyse for any software changes that are possible, reasonable and
sensible and then adjust the software accordingly. Any errors that have been found should
be fixed.
5. Release the changes and then restart the beta test until the software is up to an acceptable
level of software quality.
6. Once the software has been tested sufficiently then it can be deemed “Feature complete” –
that is no more features to this version of the software should be made and the software
develops into the next development stage – the release candidate.
Examples of well known beta tests
Microsoft
One of the most well known Beta tests in recent memory was the beta for Microsoft’s Windows 7
Operating System and before that Windows Vista. These were closed betas that were performed by
Microsoft to gain feedback on how their operating system worked on different compatible machines
and gain general public opinion on how the operating system worked.5.
One of the main issues that has been prevalent with this beta test was that Microsoft did not take
into account the feedback that their beta testers. 10.
6
MMORPGs (Massively Multiplayer Online Role Playing Games)
Popular online games such as World of Warcraft and City of Heroes performed beta tests of their
games before the actual release. 6. The main reason for this was to test their hardware servers and
make sure that they could handle the amount of players that are likely play at any one time.
However even with beta tests then it is not uncommon for servers to present problems on and after
the release days.
Netscape
Netscape came under intense criticism when they released their Netscape 6 Web Browser in 2000
which was based on the free open source software by Mozilla and was only in its beta stage.11.The
rush in development meant without a beta test then the software was bug ridden.
Issues
As previously stated, one of the main issues that plague beta tests is gaining feedback from testers.
This has become less prevalent in recent years with the Internet becoming a faster response and
software distribution tool. With larger beta tests such as Windows 7 OS and World of Warcraft then
this is certainly the case as automatic error reports can be sent, however with smaller pieces of
software then gaining user feedback is still very much an issue.
To get round this then many companies are making beta testers sign agreements when using the
software. Other than this some developers are enticing testers to give more feedback by giving away
a free copy of the final product.12.
This however could go the other way where the developers do not take account of the tester’s
feedback, a situation similar to Microsoft during its Windows 7 OS beta test.
Another issue or criticism of beta testing is that while beta testing can provide feedback on
improvements to the software, do the developers have time to implement these changes
effectively? Since beta testing is at the end of the development cycle then it can be argued that
anything other than error correction is too late on in the development lifecycle.
Another issue is knowing when the software is ready for a beta test. If the developer releases the
software before all the features are complete is this testing still beta testing? Due to the
advancements in user feedback then it seems that beta testing is carried out earlier and earlier to
allow for changes to the software. 13. For instance a new MMMORPG, APB, opened a beta test in
October 2009 where the game is incomplete in terms of features. This goes against the trend of beta
testing where the software is complete and it is more like companies are now using real users as
alpha testers.
Alternatively then software can go the other way and end up staying as a beta version while
developers endlessly tinker with their products. A good example of this is applications done by
Google, both Gmail and google apps stayed as beta versions for several years before they became
official release versions. 14. This is mainly due to companies wanting to not take responsibility for
any errors that may occur with their newly developed software.
7
A final issue is that of knowing when a beta test ends and the live version of the software should
begin. With the potential for a beta version for a piece software to be close to the actual version of
the software and the amount of updates that continue after release then you can ask if the software
ever really is not in a beta phase? With free software then this can be more of an issue as the line is
not always clear. It is, however, easier to identify with software that costs money as the beta will
end when the software is at a sufficient standard that the developer feels that they can charge
money for it – anything after this is maintenance of the software.
Conclusion
Beta testing is a fundamental part of the testing cycle and allows developers to see how their
software will interact in a real environment. However with so many issues surrounding beta testing
then it appears that the idea of beta testing may be subtly changing in the coming years.
8
References
1. Wikipedia article on beta testing
http://en.wikipedia.org/wiki/Software_release_life_cycle#Beta
2. A Practical Guide to Usability Testing - S. Dumas and Janice Redish
3. Wikipedia article on IBM Product Test
http://en.wikipedia.org/wiki/IBM_Product_Test
4. Taken from SQA.net – based on NASA’
http://www.sqa.net/
5. Windows 7 news announcing the limited beta test
http://windows7news.com/2009/01/08/windows-7-beta-to-be-released-on-january-9/
6. Gamespot giving news of the World of Warcraft beta test
http://uk.gamespot.com/news/6101498.html
7. Journal Entry for Softner 1.2 software on beta testing
http://www.namesuppressed.com/syneryder/2004/betapostmortem.shtml
8. Windows 7 beta test closure reminder
http://windowsteamblog.com/blogs/windows7/archive/2009/02/09/reminder-generalavailability-for-windows-7-beta-ending-this-week.aspx
9. Microsoft Connect website
http://beta.microsoft.com/
10. A news article on Microsoft responding to criticisms that they do not take on beta testers
feedback http://windows7news.com/2009/03/02/windows-7-beta-testing/
11. A news article on Netscape realising their beta test early http://news.zdnet.com/21009588_22-141230.html
12. Starcraft II beta test opt in agreement
http://starcraft.incgamers.com/gallery/data/520/Beta_Profile_Settings_Page_small.jpg
13. A news article on beta tests in a modern age by a company called
tidbitshttp://www.tidbits.com/tb-issues/TidBITS-328.html#lnk5
14. A news article on google aps as a beta http://googleblog.blogspot.com/2009/07/googleapps-is-out-of-beta-yes-really.html
9