Software
Testing
LAB COMPENDIUM FOR
SOFTWARE TESTING
TDDD04
LABORATORY 2015
LAB 3
AUTHORS:
Ola Leifler
Usman Dastgeer
Per Böhlin
Mohamed Abu Baker
Mohsen Torabzadeh-Tari
Kristian Sandahl
DEPT. OF COMPUTER AND INFORMATION SCIENCE
LINKÖPING UNIVERSITY, SWEDEN
SPRING 2015
TDDD04 Software Testing
2015
1 Introduction (common for lab 1 & 3)
Read through the lab instructions before going to the lab. You will benefit from it. There are 12
hours scheduled in the labs (PC PULs). The total time expected to spend on this lab series,
including the reflection reports (see below), is approximated to about 50 hours. This mean that
you are expected to put in a lot of hours on your own. Coming prepared to the scheduled hours
will make things better for all of you. You will be able to ask intelligent questions and we (the
lab assistants) can spend our time giving relevant answers.
The lab should be conducted in pairs (two students). Pair programming 1 should be practiced
during the lab and is considered part of the exercise. Studies suggest that pair programming is
most efficient when switching driver (the person typing) every 20 minutes. It is not necessary
for you to switch that often but try to switch every hour or at least after each completed task.
The lab series consists of two labs, each broken down into a number of exercises. The first lab
(exercises 1-4) is expected to take about 16 hours (4 of the scheduled hours), including writing
the reflective report. The second lab (exercises 1-4) is expected to take about 34 hours (8 of the
scheduled hours) including writing the reflective report. Each exercise is to be demonstrated to
a lab assistant. After completing all the exercises for a lab, code as well as a reflective report
should be submitted to your lab assistant for final review. Details on requirements and
submission are presented at the end of the labs’ descriptions.
Sharing working code solutions between groups, or directly copying complete solutions from
the Internet is not allowed and is considered cheating and can result in a report to the
disciplinary committee2.
The programming exercises are fairly simple but you are expected to have at least basic
understanding of C++, Java and object oriented programming. Familiarity to Visual Studio and
Eclipse is also expected. If you feel uncertain about these areas you might have to spend some
extra time catching up. Please ask your lab assistant if you need help finding suitable resources.
Important! Due to the large number of students in this course, we do not have time to handle
multiple resubmissions. Therefore, please take great care to make the first version good. There
is only 1 resubmission opportunity and in case someone misses the resubmission deadline or
require a 2nd resubmission, he/she would have to wait until the retake period in August.
Deadlines
Deadlines for the labs will be posted on the course web page.
1 Read more (not mandatory) about pair programming in Kent Beck’s book Extreme
Programming Explained – Embrace Change (2002 –2nd ed.); Williams’ and Kessler’s paper
“All I Really Need to Know about Pair Programming I Learned In Kindergarten” (1999)
http://collaboration.csc.ncsu.edu/laurie/Papers/Kindergarten.PDF; or on Wikipedia:
http://en.wikipedia.org/wiki/Pair_programming
2 The disciplinary committee decides on matters regarding cheating and other kinds of
misconduct. Cheating can result in a formal written warning or up to six months suspension
from all university activities –including lectures and exams.
1
TDDD04 Software Testing
2015
http://www.ida.liu.se/~TDDD04/labs/index.en.shtml
Setting up the Lab Environment
How to setup the lab environment is described on the course web site. Please see
Labs Homepage: http://www.ida.liu.se/~TDDD04/labs/index.en.shtml
Helping material/hints: http://www.ida.liu.se/~TDDD04/labs/extra_lab_instructions.shtml
2
TDDD04 Software Testing
2015
Lab 3: Exploratory Testing, Test
Implementation and Regression Testing
Planning,
Test
You are allowed to work on your own project in this third lab, should you choose to do so. If
you choose your own project, you may focus on using advanced techniques such as mutation
testing, test case generation using symbolic execution or model-based testing, but you must
justify the use of those techniques with respect to the goals stated for each exercise in this lab
(listed as Passing Requirements for each exercise) and also explain that your proposed
exercises will be of similar scope and with similar general goals as the exercises using the
Black Lagoon example project. The remainder of this instruction assumes you use our example
project the Black Lagoon.
1.1 Introduction and Preparations
The purpose of lab 3 is for you to apply the principle and tools from lab 1 on a larger scale.
Therefore, you are expected to have learnt how to use a unit testing toolchain such as
CppUnitTestFramework. As part of the preparations, you should read the included references
as well as chapters 13, and 14 in the course book.
In this lab you are expected to apply the knowledge you have acquired in the first lab; about the
tools and practices in a creative and independent manner. Since this lab is not about the tools
themselves but what to test, how to test it and why; this lab will be slightly more difficult. The
tools are just tools; means to an end. Therefore, it will require you to work more independently
and think for yourself about the testing strategies applied to a greater extent than the first lab.
The subject of testing for lab 3 is a simple 2D top-down scrolling game called Black Lagoon.
The topics of concern in lab 3 are:

Exploratory Testing, (exercise 1)
An approach of “simultaneous” learning, by designing and executing tests while exploring
the System Under Test (SUT)
The interesting one definition: “Exploratory testing is simultaneous learning, test design,
and test execution.” In other words, exploratory testing is any testing to the extent that the
tester actively controls the design of the tests as those tests are performed and uses
information gained while testing to design new and better tests.
Read: course book chapters 13 and 14.

Automated Unit Tests, (exercises 2 & 3)
Revise: the material about unit testing from lab 1.

Stubs, (exercises 2 & 3)
o “A dummy component used to simulate the behavior of a real component. The test of a
component X which calls a component Y, which is not already tested implies the
replacement of Y by a dummy component called stub. A specific stub is written if it
3
TDDD04 Software Testing

2015
simulates Y’s behavior relatively to X use. The dummy component uses the same calling
sequence but provides “canned” outcomes for its processing. A realistic stub is used if it
simulates Y in every way. Realistic stubs can correspond to obsolete, but reliable,
implementations of the stubbed component”3.
Regression Testing, (exercise 4)
An important part of testing, in which the aim is to prevent bugs that have been taken care
of from reoccurring by comparing test results before and after implementing bug fixes and
feature extensions.
1.2 THE GAME - Introduction
Black Lagoon is a simple 2D top-down scroller game of two levels, which is designed for one
player. The player controls a hovercraft, with the mission of reaching the end of each level.
Along the river, there are several rogue military units (the enemy) that will try to stop the
player.
The player can move the hovercraft using the arrow keys (forward, back, right, and left). The
hovercraft can also be rotated left and right using the Q and E keys. The up, down, right and
left arrow keys will always move the hovercraft relative to the global coordinate system. W, S,
A and D key can also be used to move the hovercraft forward, back, left and right, but will
move the craft relative to its current rotation.
The hovercraft has a twin gun that it can use to shot down enemy vessels. Some enemy units
have the ability to shoot back. Shots hitting a vessel (enemy or player) will cause it to lose
health. If the player’s health is reduced to zero, the player dies and the game is over.
Enemy vessels that have its health reduced to zero will simply disappear. Depending on the
type of military unit: money, extra ammo or health will be left in its stead.
The number of enemies killed is displayed on the scoreboard along with game score, health,
money, ammo left and game time.
Besides being fired upon by the rogue military units, there are other factors that cause
reduction in the hovercraft health. If the hovercraft collides with the enemy, or with the bank of
the river from any side, this will cause a reduction in health. The hovercraft can improve its
health by picking up health packages floating in the river or dropped by sunken enemy ships.
There are other control keys that can be used in this game. The P key, which causes the game
to pause, and un-pause, ESC exits the game. When the player starts a new game, the hovercraft
will always starts with health, ammo and money set to 100, and score, and kills set to zero.
When the game is over, the score is saved to a high score list where the top ten results is stored.
3 Thierry Jéron, Jean-Marc Jézéquel, Yves Le Traon, and Pierre Morel - Efficient Strategies for Integration and
Regression Testing of OO Systems, Proceeding 10th International Symposium on Software Reliability
Engineering (Cat, No.PR00443) 260-9, 1999
4
TDDD04 Software Testing
2015
The source code is partially badly written and should not be taken as an example of good code.
It suffers from improper and lack of use of const, bad naming of classes and functions, almost
no error handling, among other things.
1.3 THE GAME – Specification
GENERAL
I-key extra information about the application status: FPS (Frames per second), applications
internal clock and if the game is paused (P) or not.
MENU SCREEN
ESC-key exits the game application.
Left arrow key move to previous in list.
Up arrow key move to previous in list.
Down arrow key move to next in list.
Right arrow key move to next in list.
Selecting Play on the menu and hitting enter or space should start a new game.
Selecting High Score on the menu and hitting enter or space should show the high score screen.
Selecting Credits on the menu and hitting enter or space should show the credits.
Selecting Exit on the menu and hitting enter or space should exit the game.
HIGH SCORE SCREEN
Hitting the ESC, return or space key when displaying the high score should return to menu.
The high score shows a list of the top 10 results in descending order (best score first).
The last achieved score (if on the list) should be highlighted in red.
Scores of zero should not be added to the list.
CREDITS SCREEN
Hitting the ESC, return or space key when displaying the credits screen should return to menu.
GAME PLAY
ESC-key quits the game session as if the last level was finished or the player died.
Space fires a shot.
F-key fires a shot.
Left arrow key moves hovercraft to the left relative to the global coordinates.
Right arrow key moves hovercraft to the right relative to the global coordinates.
Up arrow key moves hovercraft forward relative to the global coordinates.
Down arrow key moves hovercraft back relative to the global coordinates.
5
TDDD04 Software Testing
2015
Q-key rotates the hovercraft to left.
E-key rotates the hovercraft to right.
W-key moves the hovercraft forward relative to rotation.
S-key moves the hovercraft back relative to rotation.
A-key moves the hovercraft left relative to rotation.
D-key moves the hovercraft right relative to rotation.
P-key pause and un-pause the game.
Starting a new game session should reset score as follows: Score: 0, Money: 100, Ammo: 100,
Health: 100, Time: 00:00:00, Kills: 0
Small boats should require 3 shots to be killed.
Gun boats should require 8 shots to be killed.
Large gun turrets should require 10 shots to be killed.
Small gun turrets should require 5 shots to be killed.
The player should die when health reaches zero.
When the player dies, the game ends, and the high score screen is shown.
Gun turrets should track the player the shortest path in all directions (except turrets mounted on
gun boats that cannot aim directly behind due to the high control cabin at the center of the
ship).
Health package should increase player’s health by 25.
Money package should increase player’s money by 100.
Ammo package should increase player’s ammo by 20.
Firing a shot should decrease player’s ammo by 1.
Hitting enemy vessels should cause collision damage on both player and enemy.
Hitting the bank of the river (the terrain) should cause collision damage.
Collision occurs when the outer boundaries of sprites overlap.
Killing an enemy should increase the number of kills on the scoreboard by one.
Killing an enemy should increase the score on the scoreboard by 10 points.
Hitting the red mark by the end of the river causes the level to end.
At end of level, the next level should be loaded, unless last level, then the game ends and the
High Score screen is shown.
The game uses a left-handed coordinate system:
6
TDDD04 Software Testing
2015
1.4 Playing the game
Play the game and explore it! Run it to the end (two levels)!
In order to find the bugs you will need to be creative and use the knowledge you have acquired
from the course lectures and course literature. Use the specification as hints for what to test for.
In the Game\Levels directory, there is an extra level file, “test_level.lev”. Temporarily replace
the real level file with the test file to make it easier to test some aspects of the game.
A “correct” version of the game has been included in the Game folder as reference. Any bugs
also appearing in the correct version can be ignored in the assignment.
Exercise 1.
Manual Exploratory Testing
In this exercise you will perform the concept of Exploratory Testing by manually explore and
test the Black Lagoon (using the game’s graphical interface), and record (write down) all the
performed tests using the template provided on the next page. You should record the action
taken to execute the test, the expected result, the actual result, and if the test was considered
pass or fail. If fail, why?
Record all defects found and explore their cause. For each defect, try to debug the application
(using Visual Studio’s built in debugger4) until you have found the general area in the code that
causes the defect. Try to analyze the code and understand what is causing the defect.
Write a test plan (using the provided template on the next page) for at least FOUR (4) of the
defects. The plan should contain what automated unit tests should be implemented, and how
they should be designed. The plan need to cover at least four of the defects detected giving a
general description of the symptom, description of the code, state and context that caused the
defect. It should also contain a description on how to expose the defect with automated unit
test; how the test should be designed with regards to what input should be used and expected
output for a working system.
Note: The same code defect will sometimes cause multiple symptoms. The general rule for this
exercise is that errors in the same method/function are considered part of the same defect.
Hence, you need to select four unrelated defects for your test plan.
Note: Before proceeding to the next exercise, discuss your test plan and the defects
selected with your lab assistant. After approval, email the test plan to your lab assistant.
4 http://dotnetperls.com/debugging
7
TDDD04 Software Testing
2014
TEMPLATE - TEST RECORD
Test
Nr
1
2
3
4
Item
Menu
Menu
Menu
Menu
Action
Down key
Down key when on last menu item
Hit enter when Credits selected
Hit enter when the Exit selected
Result
Expected
Next menu item selected
First item selected
Show Credits screen
Exit the game
Actual
Next menu item selected
First item selected
Shows Credits screen
No action is taken
Pass/Fail Comments
P
P
P
Game::updateGame() should return
F
true in order for the game to exit.
5
…
TEMPLATE - TEST PLAN
Defect Nr:
Class/Function:
Test Nr:
1
Game.cpp
4.1
Defect Description (symptom):
The game does not exit as intended when hitting the return/space key when the Exit option is selected in the game menu.
Hitting the return or space key when the Exit option is selected in the game menu should exit the game.
Code, state and context description that triggered the defect:
The switch statement, case: “m_menuComponent->EXIT” in Game.cpp, line 165, should be followed by a return true statement, making the Game::updateGame_menu
function return true.
Input to trigger defect:
The game object should be instantiated with at least a none-null input device and menu component.
m_inputDevice->select(), should return true
m_menuComponent->select() should return m_menuComponent->EXIT
Expected output from working system:
Game::updateGame_menu should return true when the exit option is selected on the menu component and the input device signals select.
TASKS
1. Use exploratory testing to find defects in Black Lagoon.
2. Use both the original level file as well as the test level file in the Game\Levels directory.
8
TDDD04 Software Testing
2014
3. Record all tests performed (both passing and failing) using the template provided (or your own format, at least containing the same items
as the template).
4. For all defects discovered, try to locate their general location in the source code and the cause of the defect.
5. Select at least four unrelated defects and write a test plan for them. The plan should outline how the defects can be exposed using automated
unit tests and how the test should be designed. Use the provided template (or your own format, at least containing the same items as the
template).
PASSING REQUIREMENTS
 Test record of performed exploratory tests.

Each item in the game specification has been tested.

Defects have been detected – a reasonable number of defects (the number of expected defects to be found is only known to the lab assistants).

Test plan for at least four unrelated defects.
HINTS
 Learn from the provided examples, on how to use the templates.
9
TDDD04 Software Testing
Exercise 2.
2014
Implement Automated Unit Tests from Test Plan
You should only start this exercise after first exercise has been approved by your lab assistant. In this part of the lab you will implement THREE
(3) of the unit tests that you have defined in your test plan from exercise 1. You need to implement the unit tests using CppUnitTestFramework.
Verify good coverage of the code area using Visual Studio coverage tool. The unit tests should be designed so that they fail due to the defect and
pass once the defect has been corrected, but keep in mind that you will not fix the bugs at this stage, and (most of 5) your test cases should therefore
fail.
A test project has been included in the same solution as the game. Add your own test classes that implements the tests specified in your test plan.
In addition to the tests for the defects identified in the test plan, your unit tests should also cover other parts of the class under test that you find
relevant.
Due to the code structure of the program, you will sometimes be required to use existing components in some test cases. In some cases you will
need to write your own stubs. Use your judgment for when to use what.
TASKS
1 Implement three (3) of the unit tests specified in your test plan from Exercise 1, using CppUnitTestFramework.
2
Use your knowledge about different testing strategies such as boundary testing and equivalent class testing, whenever appropriate to produce
good test cases. Not only from the perspective of code coverage but from the perspective of actually exposing defects.
3
Create a test class for each class that is to be tested.
4
Verify that the tests give good coverage of the problem area using Visual Studio coverage tool. If coverage is unsatisfactory, add more test
cases.
5
The tests should be designed so that they fail due to the defect and once the defects have been corrected the tests should pass. Remember that
you will not fix the found bugs in this stage, and that your tests should fail.
5 Its good idea to write test cases that test surrounding code of where the defect is. Testing already working part (functionality) is important to
make sure the fixes (that you make later in Exercise 3) do not introduce new (unintended) defects in already working parts.
10
TDDD04 Software Testing
2014
6
In most circumstances it is a good idea to write tests that test working areas of a class under test, not only the defect. This way you will be
able to say to greater certainty you have not introduced new bugs once fixing the exposed defects. Write some test cases that confirm existing
correct behavior. This can also help to bring up code coverage.
7
Demonstrate to your lab assistant before proceeding to the next exercise.
PASSING REQUIREMENTS
 Three tests from the test plan should be completely implemented.

Reasonable (only less than 100 % for very good reasons) code coverage of the problem code areas.

No fixes for the found defects.

All test cases specifically designed to expose defects should fail.

Some additional test cases confirming correctly working parts in the problem area.

The tests should be organized into test fixtures (individual classes) for each class that are to be tested for defects.
HINTS
 Use one of your test classes from lab 1 as a template to get started quicker.

The term problem area has intentionally been loosely defined. It refers to the code segment that is critical for the functionality that the defect
impairs. It could be a complete function/method if it is only a few lines long or if it lacks branches. In a larger function/method the problem
area might be a single code block or sets of nested code blocks. You have to assess the situation and make a judgment yourself on how much
you think should be tested. But be prepared to make improvements if your lab assistant does not share your assessment.
11
TDDD04 Software Testing
Exercise 3.
2014
Fix the Code Defects
In this part of the exercise, you will use the gained knowledge about the found defects in exercise 1, and tested and failed in exercise 2, to fix the
code that causes the defects. Now the automated implemented unit tests that failed in exercise 2 should pass, and the functionality of the fixed
code should run with the correct behavior in the game’s graphical interface.
After a bug fix has been applied, the code coverage for the problem area should still be good (100% if not for very good reasons). If you introduce
logic as part of the bug fix, that logic should be thoroughly tested with new test cases if not properly covered by the original tests. If you during
the implementation/bug fix discover that your original tests were incorrect, you are of course allowed to improve them.
In addition to fixing the bugs identified by the automated test cases, also fix any other bugs that you have a good understanding of.
TASKS
1 Fix the defects identified and tested in exercise 2.
2
Verify that the automated unit tests from exercise 2 now pass.
3
Verify that the intended functionality is performing in the natural way using the graphical interface.
4
Try to fix other bugs that you found in exercise 1, but did not write test cases for.
5
Clearly mark and comment the fixed places in the source code.
6
Demonstration to a lab assistant before proceeding to the next exercise.
PASSING REQUIREMENTS
 All three defects covered by tests are fixed.

All failed automated tests in exercise 2 should pass.

Reasonable (only less than 100% for very good reasons) code coverage of the problem code areas.

Correct behavior in the game’s graphical interface for the three found defects.
12
TDDD04 Software Testing

2014
Any other bugs fixed should make the game behave according to the specification and not introduce new errors.
13
TDDD04 Software Testing
Exercise 4.
2014
Regression Testing
The objective of this exercise is to learn about the advantages of Regression Testing. You will perform the regression testing in the graphical
interface of the game using your initial test record from exercise 1. Verify that the bug fixes that you made have had the intended effect and that no
new defects have been introduced by mistake.
TASKS
1 Use the test record from exercise 1 and do regression testing of the application.
2
Document the test results in a new Test Record. Each item of it should match the same item in the first test record.
3
If any new defects are discovered, make a note about it and explain whether it is a new issue or an old defect just not found the first time.
4
Demonstrate the test record to your lab assistant and discuss the result.
PASSING REQUIREMENTS
 Test record of performed regression testing, with items matching the tests in the initial exploratory tests.

All the fixed bugs should result in the game behaving in the intended way without introducing any new defects.
14
TDDD04 Software Testing
2014
1.5 Reporting Lab 3
To pass this lab you need to both demonstrate each of the exercises to your lab assistant as well as hand in a written reflection report of what you
have learnt together with the source code changed or written by you.
You should send an email based on the following instructions to amir.aminifar@liu.se with TDDD04 in the subject of the email. One email per
group!
o The email must include:
o A test record for the exploratory testing filled in with test cases ID, tested item and actions, results (expected & actual), pass or fail, and
the related comments from Exercise 1.
o Test plan for the discovered defects with all related information from Exercise 1.
o All source code written by you or changed by you (clearly marked, and well explained) – from Exercise 2 & 3.
o A test record for the regression testing filled in with the same test cases from the exploratory test with its new results after the bug fixes
from Exercise 4.
o A reflective report should be written in accordance to the directions below.
Copying each other’s reports or directly copying other sources such as books or web pages are considered cheating and can result in a
report to the disciplinary committee.
After demonstrating your solution to your lab assistant, and handing in the source code, you will only get a “Complete” in webreg. To get a
“Passed” you need to have your individual report handed in and passed as well.
If the groups’ solution (source code) or the individual reflective lab report is not “Passed” and you were asked to hand in an updated version of it
then you must also attach the old version of the source code/lab report.
15
TDDD04 Software Testing
2014
DEMONSTRATION
During the demonstration you are expected to show your lab assistant a working solution in accordance to the requirements specified for the
individual exercises. Both group members should be able to answer detailed questions regarding the solution implemented as well as the topics
specified in the requirements. Both group members are expected to participate equally as this is an oral and practical examination.
16
TDDD04 Software Testing
2014
REFLECTIVE REPORT
You are required to submit (via email), a short discussion (2-3 pages) regarding:
1. Pros and cons of exploratory testing.
2. Your thoughts about regression testing.
3. Pros and cons of debugging and automated unit testing. From your own point of view, write a short discussion in terms of time/cost,
usability, and general complexity of writing good unit tests.
4. How did you find the four planed test scenarios? What was the trigger that led to each defect? How did you locate the place of the defect in
the source code?
5. How many defects did you find in total? How many did you fix? How did you find them? Why didn’t you find the defects (if any) that
were pointed out by the lab assistant?
6. Why did you choose to test the three approved defects? If you were asked to rate these defects (easy/medium/hard), how would you do
that? Why?
7. Did you experience any unexpected behavior during your regression test? If Yes! What was it? How did you find it? How do you think
your fixes affected that part of the code? How did you treat it?
8. Did your coverage measures reach 100%? Yes/No, why?
9. Was it necessary to reach 100% for all of your test cases? What is your opinion regarding requirements of “full coverage” in realistic
settings?
OBS: List any references used, such as books or websites, in accordance to normal academic standard. Your reports will be checked via
URKUND.
17