Integra(on tes(ng
TDDD04 So1ware tes(ng
Yes, this is a new ppt template
måndag 25 maj 15
Levels of tes(ng
Maintenance
Validate Requirements, Verify Specification
Requirements
Acceptance Test
(Release testing)
System Design
Verify System Design
System Testing
(Architecture,
High-level Design)
Module Design
(Program Design,
Detailed Design)
(Integration testing of modules)
Verify Module Design
Module Testing
Verify Implementation
Implementation
of Units (classes, procedures,
functions)
(Integration testing of units)
Unit testing
Project Management, Software Quality Assurance (SQA), Supporting Tools, Education
2
måndag 25 maj 15
Outline
• Informal example
• Theory:
– Decomposi(on-­‐based integra(on
– Call-­‐graph based integra(on
– Path-­‐based integra(on
måndag 25 maj 15
Example: Local bus card reader
Sell (ckets
Show balance
Register travel
Choose (cket
and
User buMons
Display
Func(on group,
aka Capability
Read RFID
Check balance
Deduct money
Check validity
Communication with server
Power supply
måndag 25 maj 15
Dependency
Example: Organic integra(on plan
Sell (ckets
Show balance
Registrer travel
Chose (cket
Services
User buMons
Display
User interface
Read RFID
Check balance
Server func(ons
Deduct money
Check validity
Communicate with server
Power supply
måndag 25 maj 15
Communica(on
Hardware and supply
Big Bang integra(on tes(ng
Sell (ckets
Show balance
Chose (cket
Registrer travel
User buMons
Display
Bring everything together
Switch on the current
Try to buy a (cket
!
Read RFID
Check balance
Check validity
Deduct money
måndag 25 maj 15
Power supply
Communicate with server
Is Big Bang smart?
• Well-­‐defined components and interfaces
• No extra so1ware needed
Problems:
• It is very difficult to isolate the defects found, as it is very difficult to tell whether defect is in component or interface.
• Defects present at the interfaces of components are iden(fied at very late stage.
• There is high probability of missing some cri(cal defects which might surfaced in produc(on.
• It is very difficult to make sure that all the cases for integra(on tes(ng are covered.
(www.Tes(ngGeek.com)
måndag 25 maj 15
BoMom-­‐up integra(on 1
Environment crea(on
Install components
Draw cables
Measure – compare with calcula(on Power supply
måndag 25 maj 15
Hardware and supply
BoMom-­‐up integra(on 2
Communica(on possible?
Drivers
Environment
Rudimental client
Rudimental server
Communicate with server
Power supply
måndag 25 maj 15
Communica(on
Hardware and supply
Driver
• A pretend module that requires a sub-­‐system and passes a test case to it
setup driver
SUT(x)
verifica(on
driver
SUT
SUT
black-­‐box view
SUT – System Under Test
måndag 25 maj 15
Is boMom-­‐up smart?
• If the basic func(ons are complicated, error-­‐prone or has development risks
• If boMom-­‐up development strategy is used
• If there are strict performance or real-­‐(me requirements
Problems:
• Lower level func(ons are o1en off-­‐the shelf or trivial
• Complicated User Interface tes(ng is postponed
• End-­‐user feed-­‐back postponed
• Effort to write drivers.
måndag 25 maj 15
Top-­‐down integra(on 1
Sell (ckets
Show balance
Registrer travel
Chose (cket
Services
Envrionment:
Modules for the services
Driving a prototype interface
Test end-­‐users
Is it possible to perform services in a good way?
måndag 25 maj 15
Top-­‐down integra(on 2
Sell (ckets
Show balance
Registrer travel
Chose (cket
Services
User buMons
Display
User interface
Read RFID
Func(on 1
Func(on 2
Stubs
måndag 25 maj 15
Func(on 2
Rudimentary
Server func(ons
Are all our test scenarios fulfilled?
Stub
• A program or a method that simulates the input-­‐output func6onality of a missing sub-­‐
system by answering to the decomposi(on sequence of the calling sub-­‐system and returning back simulated or ”canned” data.
SUT
Service(x) SUT
Stub(x)
return y;
end
Stub
måndag 25 maj 15
Is top-­‐down smart?
• Test cases are defined for func(onal requirements of the system
• Defects in general design can be found early
• Works well with many incremental development methods
• No need for drivers
Problems:
• Technical details postponed, poten(al show-­‐stoppers
• Many stubs are required
• Stubs with many condi(ons are hard to write
måndag 25 maj 15
Decomposi(on-­‐based integra(on
The func(onal decomposi(on tree:
• hierarchical order of processes
• reflects the lexical inclusion of units, in terms of the order in which they need to be compiled måndag 25 maj 15
Func6onal Decomposi6on of the SATM System
Example: SATM from Jorgensen
1
A
D
2
3
10
E
4
5
11 12 13
6
7
8
C
B
9
14
15
16
18
19
22
F
17
20
21
23
24
25
26
Unit Level Unit Nam Unit Level Unit Name B 1.3 Terminal Sense & Control 1 1 SATM System 14 1.3.1 Screen Driver A 1.1 Device Sense & Control 15 1.3.2 Key Sensor D 1.1.1 Door Sense & Control C 1.4 Manage Session 2 1.1.1.1 Get Door Status 16 1.4.1 Validate Card 3 1.1.1.2 Control Door 17 1.4.2 Validate PIN 4 1.1.1.3 Dispense Cash 18 1.4.2.1 GetPIN E 1.1.2 Slot Sense & Control F 1.4.3 Close Session 5 1.1.2.1 WatchCardSlot 19 1.4.3.1 New Transac6on Request 6 1.1.2.2 Get Deposit Slot Status 20 1.4.3.2 Print Receipt 7 1.1.2.3 Control Card Roller 21 1.4.3.3 Post Transac6on Local 8 1.1.2.3 Control Envelope Roller 22 1.4.4 Manage Transac6on 9 1.1.2.5 Read Card Strip 23 1.4.4.1 Get Transac6on Type 10 1.2 Central Bank Comm. 24 1.4.4.2 Get Account Type 11 1.2.1 Get PIN for PAN 25 1.4.4.3 Report Balance 12 1.2.2 Get Account Status 26 1.4.4.4 Process Deposit 13 1.2.3 Post Daily Transac6ons 27 1.4.4.5 Process Withdrawal Table 1: SATM Units and Abbreviated Names måndag 25 maj 15
27
17
Three level func(onal decomposi(on tree
A
C
B
E
måndag 25 maj 15
Level 1
F
D
G
Level 2
H
Level 3
Big-­‐Bang tes(ng
A
Level 1
C
B
E
F
D
G
Level 2
H
Environment:
A, B, C, D, E, F, G, H
Level 3
Unit test A
Unit test B
…
Unit test H
måndag 25 maj 15
System-­‐wide test
BoMom-­‐up tes(ng
A
C
B
E
Level 1
F
D
G
Level 2
H
Level 3
General formula:
Number of drivers: (non-­‐leaves)
Number of sessions: (non-­‐leaves)+edges
Environments:
Session 1: E, driver(B)
S2: F, driver(B)
S3: E, F, driver(B)
S4: G, driver(D)
S5: H, driver(D)
S6: G, H, driver(D)
S7: E, F, B, driver(A)
S8: C, driver(A)
S9: G, H, D, driver(A)
S10: E, F, B, C, G, H, D, A
Number of drivers: ? 3
Number of sessions: ? 10
SATM: 10 drivers, 42 sessions
måndag 25 maj 15
Top-­‐down tes(ng
A
C
B
E
Level 1
F
D
G
Level 2
H
Level 3
General formula: Number of stubs: (nodes – 1)
Number of sessions: (non-­‐leaves)+edges
Environments:
Session 1: A, stub(B), stub(C), stub(D)
S2: A, B, stub(C), stub(D)
S3: A, stub(B), C, stub(D)
S4: A, stub(B), stub(C), D
S5: A, B, stub(E), stub(F), C, D, stub(G), stub(H)
S6: A, B, E, stub(F), C, D, stub(G), stub(H)
S7: A, B, stub(E), F, C, D, stub(G), stub(H)
S8: A, B, stub(E), stub(F), C, D, G, stub(H)
S9: A, B, stub(E), stub(F), C, D, stub(G), H
S10: A, B, E, F, C, D, G, H
Number of stubs: ?
Number of sessions: ?
SATM: 32 stubs, 42 sessions
måndag 25 maj 15
7
10
Sandwich tes(ng
Taget level
A
C
B
E
Level 1
F
D
G
Fewer stubs and drivers
Risk driven
Small-­‐bang at target level
More complicated
måndag 25 maj 15
Level 2
H
Level 3
Environments:
Session 1: A, stub(B), stub(C), stub(D)
S2: A, B, stub(E), stub(F), stub(C), stub(D)
S3: A, stub(B), C, stub(D)
S4: A, stub(B), stub(C), D, stub(G), stub(H)
S5: E, driver(B)
S6: F, driver(B)
S7: E, F, driver(B)
S8: G, driver(D)
S9: H, driver(D)
S10: G, H, driver(D)
S11: A, B, E, F, C, D, G, H
Number of stubs: 3
Number of drivers: 2
Number of sessions: 11
Poten(al problems
?
•
•
•
•
Ar(ficial: Assumes correct units and interfaces
Test correct structure only
Investment in stubs and drivers
Retes(ng
måndag 25 maj 15
Call-­‐graph integra(on tes(ng
• Use the call-­‐graph instead of the decomposi(on tree
• The call graph is directed
• Two types of tests:
– Pair-­‐wise integra(on tes(ng
– Neighborhood integra(on tes(ng
• Matches well with development and builds
• Tests behaviour
måndag 25 maj 15
5
Pairwise integra(on of SATM
1
7
One session per edge
Real code
40 sessions, but no extra code
20
21
9
22
16
4
10
13
12
11
17
18
19
23
24
26
14
måndag 25 maj 15
15
6
8
27
2
25
3
Neighborhood integra(on of SATM
5
1
7
Integra(ng direct neighbors of nodes
Number of sessions: nodes – sinknodes
(a sink node has no outgoing calls)
SATM: 11 sessions
20
21
9
22
16
4
10
13
12
11
17
18
19
23
24
26
14
måndag 25 maj 15
15
6
8
27
2
25
3
Poten(al problems
?
• ”Small-­‐bang” problems: fault isola(on in large neighborhoods
• Retes(ng needed if a node is changed
• Assumes correct units
måndag 25 maj 15
Path-­‐based integra(on
• Tes(ng on system level threads
• Mo(vated by overall system behaviour, not the structure
• Smooth prepara(on for System level tes(ng
måndag 25 maj 15
Example: An MM-­‐Path
module A calls module B, which in turn calls module C A
C
B
1
1
1
2
2
2
3
4
3
3
4
4
5
5
6
måndag 25 maj 15
Extensions to Defini6ons
Defini(ons
•
Defini6on: A source node in a program is a statement fragment at which program execu(on begins or resumes.
•
Defini6on: A sink node in a program is a statement fragment at which program execu6on halts or terminates. •
Defini6on: A module execu6on path (MEP) is a sequence of statements that begins with a source node and ends with a sink node, with no intervening sink nodes.
•
Defini6on: A message is a programming language mechanism by which one unit transfers control to another unit. måndag 25 maj 15
Defini6ons for Integra6on Tes6ng
More defini(ons
• Defin(on: An MM-­‐Path is an interleaved sequence of module execu(on paths (MEP) and messages. • Defini(on: Given a set of units, their MM-­‐Path graph is the directed graph in which nodes are module execu6on paths and edges correspond to messages and returns from one unit to another. måndag 25 maj 15
Example (cont.): source nodes and sink nodes
Source nodes
A
1
2
Sink nodes
4
3
5
6
måndag 25 maj 15
Example (cont.): source nodes and sink nodes
Source nodes
B
1
Sink nodes
2
3
4
måndag 25 maj 15
Example (cont.): source nodes and sink nodes
Source nodes
C
1
Sink nodes
2
3
4
5
måndag 25 maj 15
Example (cont.): source nodes and sink nodes
Source nodes
A
C
B
1
1
1
2
Sink nodes
2
2
3
4
3
3
4
4
5
5
6
måndag 25 maj 15
Example (cont.): source nodes and sink nodes
• Module A:
– Source nodes: Nodes 1 and 5
– Sink nodes: Nodes 4 and 6
• Module B:
Source nodes: Nodes 1 and 3
– Sink nodes: Nodes 2 and 4
– • Module C:
– Source nodes: Node 1
– Sink nodes: Node 5 måndag 25 maj 15
Example (cont.)
module execu(on path (MEP)
?
•
•
•
•
•
•
•
MEP (A, I) = < 1, 2, 3, 6 >
MEP (A, II) = < 1, 2, 4 >
MEP (A, III) = < 5, 6 >
MEP (B, I) = < 1, 2 >
MEP (B, II) = < 3, 4 >
MEP (C, I) = < 1, 2, 4, 5 >
MEP (C, II) = < 1, 3, 4, 5 >
måndag 25 maj 15
Example (cont.)
module execu6on path (MEP)
A
C
B
1
1
2
MEP(B,I)
MEP(A,II)
2
4
3
3
MEP(A,I)
5
6
måndag 25 maj 15
1
MEP(A,III)
MEP(C,II)
2
MEP(C,I)
3
4
MEP(B,II)
4
5
Example (cont.): MM-­‐Path graph
Messages
MEP (A, II)
MEP (B, I)
MEP (A, I)
MEP (C, I)
Returns
MEP (B, II)
MEP (A, III)
MEP (C, II)
Test cases are selected to cover these paths
måndag 25 maj 15
Problems
?
• more effort is needed to iden6fy the MM-­‐Paths, but the effort is probably offset by the elimina(on of stub and driver development.
måndag 25 maj 15
Data-­‐driven integra(on tes(ng
Input
Input
MEP (A, II)
MEP (B, I)
MEP (A, I)
MEP (C, I)
MEP (B, II)
MEP (A, III)
MEP (C, II)
41
måndag 25 maj 15
Some demos
42
måndag 25 maj 15
Tradi(onal unit tests
@Test
public void testItemServiceGetMethod(){
Input
ItemId itemid = new ItemId(2600L);
String searchText = new String("Batman");
String itemType = new String("ebook");
List<Item> items = testSubject.getItems(itemid, searchText, itemType);
Assert.assertNotNull(items);
Assert.assertEquals(2, items.size());
itemid = new ItemId(3600L);
searchText = new String("SuperMan");
itemType = new String("journal");
items = testSubject.getItems(itemid, searchText, itemType);
Assert.assertNotNull(items);
Assert.assertEquals(1, items.size());
}
43
måndag 25 maj 15
Data-­‐driven tests
44
måndag 25 maj 15
Mock objects and loose coupling
45
måndag 25 maj 15