Lecture 02
Activities of a Test Engineer
Outline
•
•
•
•
•
Test Engineer
Levels of Testing
Example Software Failures
Beizer’s Testing Levels
Automation of Testing Activities
1
July 24, 2016
Test Engineer
•
•
•
•
•
Designing test inputs
Producing test case values
Running test scripts
Analyzing results
Reporting results to developers and managers
2
July 24, 2016
Test Engineer
• Every developer sometimes wears the test engineer’s “hat”
• Test-driven development
• Each module has set of tests defined for it
test
engineer
3
July 24, 2016
Test Manager
•
•
•
•
In charge of one or more test engineers
Sets test policies and processes
Interacts with other managers
Helps engineers do work
4
July 24, 2016
5
July 24, 2016
Overall Testing Process
•
•
•
•
•
Test requirements transformed into actual values and scripts
Tests run against software (P in diagram)
Results evaluated to reveal faults
Carried out by several people
Tests monitored by manager
6
July 24, 2016
Formal Coverage Criterion
• Test engineer’s most powerful tool
• Give test engineers ways to decide what test inputs to use
• More likely to find problems
• Provide greater assurance of software reliability and quality
• Provides stopping criteria
• Technical core of course and book!
7
July 24, 2016
Ammann & Offutt Book…
•
•
•
•
Types of coverage criteria
How supported by tools
How best applied
How to integrate into development process
8
July 24, 2016
Outline
•
•
•
•
•
Test Engineer
Levels of Testing
Example Software Failures
Beizer’s Testing Levels
Automation of Testing Activities
9
July 24, 2016
Levels of Testing
•
•
•
•
•
Acceptance testing
System testing
Integration testing
Module testing
Unit testing
10
July 24, 2016
Acceptance Testing
• Assess software with respect to
requirements
• Does SW meet customer’s needs
(requirements)?
• “Highest” level of testing
• Often done by domain experts
Acceptance Testing
System Testing
Integration Testing
Module Testing
Unit Testing
11
July 24, 2016
System Testing
• Assess software with respect to
architectural design
• Does system work as a whole?
• Assumes all pieces work
• Expensive place to find lowerlevel faults
• Not often done by programmers,
but by testing team
Acceptance Testing
System Testing
Integration Testing
Module Testing
Unit Testing
12
July 24, 2016
Integration Testing
• Assess software with respect to
subsystem design
• Do modules interact correctly?
• Assumes all modules work
correctly
• Responsibility of development
team
Acceptance Testing
System Testing
Integration Testing
Module Testing
Unit Testing
13
July 24, 2016
Module Testing
• Assess software with respect to
detailed design
• Do individual modules work
correctly in isolation?
• Checks component interactions
and data structures
• Responsibility of programmer
Acceptance Testing
System Testing
Integration Testing
Module Testing
Unit Testing
14
July 24, 2016
Unit Testing
• Assess software with respect to
implementation
• Testing done without knowledge
of encapsulating software
application
• “Lowest” level of testing
• Responsibility of programmer
Acceptance Testing
System Testing
Integration Testing
Module Testing
Unit Testing
15
July 24, 2016
16
July 24, 2016
*does not assume waterfall process
Object-Oriented Software
• Blurs distinction between unit / module testing
• OO literature has slight variations:
•
•
•
•
intramethod testing- testing on individual methods
intermethod testing- testing on pairs of methods within same class
intraclass testing- tests for single entire class
interclass testing- tests more than one class at same time
• First three types are unit / module testing
• Last is integration testing
17
July 24, 2016
Regression Testing
• Part of maintenance phase of software development
• Done after changes made to software
• Ensures that updated software still possesses functionality it
had before updates
18
July 24, 2016
Test Early and Often
• Important for testing planning to proceed concurrently with
requirements analysis and design
• DO NOT put off until late in the project
19
July 24, 2016
Outline
•
•
•
•
•
Test Engineer
Levels of Testing
Example Software Failures
Beizer’s Testing Levels
Automation of Testing Activities
20
July 24, 2016
Example: 1994 Intel Pentium Microprocessor
• Problem: chip gave incorrect answers for certain FP divisions
• Intel claims problem occurs 1 in 9,000,000,000
• Problem was omission of 5 numbers in table of 1,066 values
• Easy problem to make, difficult to catch
• Difficult to find during system testing
• Could have been caught during unit testing?
• Multi-million dollar mistake
21
July 24, 2016
Example: Ariane 5 Rocket
• Launch failure of Ariane 5 rocket
• Exploded 37 seconds after liftoff (1996)
• Problem: unhandled FP conversion exception in
inertial guidance system
• Guidance system worked fine in Ariane 4 rocket
• Engineers reused inertial guidance system from Ariane 4
• Did not reanalyze software for Ariane 5’s vastly different flight trajectory
• System test would have found problem?
• How to system test a rocket??
22
July 24, 2016
Example: Failure of NASA’s Mars Climate Orbiter in 1999
• Climate orbiter burned up in Martian atmosphere
• Crashed due to integration failure between two modules
(created by two different groups)
• One module created thruster data in English units
• Forwarded data to another module that expected metric units
23
July 24, 2016
Outline
•
•
•
•
•
Test Engineer
Levels of Testing
Example Software Failures
Beizer’s Testing Levels
Automation of Testing Activities
24
July 24, 2016
Beizer’s Testing Levels
• Based on test process maturity
• Characterized by goal of test engineers
• Levels 0, 1, 2, 3, 4
25
July 24, 2016
Beizer’s Testing Level 0
• No difference between testing / debugging
• Natural view adopted by undergrad CS majors
• Get code to compile, then debug with a few inputs
• No distinction between incorrect program behavior or mistake
within the program
• Does very little to help develop software that is safe and
reliable
26
July 24, 2016
Beizer’s Testing Level 1
• Purpose of testing: show that software works
• Show correctness.
• Goal of proving complete correctness is impossible
• If ALL tests pass -> good software or bad tests?
• Usually no strict goal, stopping rule, or formal test techniques
27
July 24, 2016
Beizer’s Testing Level 2
• Purpose of testing is to show that software doesn’t work
• Show failures.
• Negative goal
• Puts testers “against” developers
• What to do if no failures found?
28
July 24, 2016
Beizer’s Testing Level 3
• Purpose of testing is not to prove anything specific, but reduce
the risk of using the software
• Testing can show presence, not absence, of failures
• Accept fact that we incur risk whenever use software
• Risk may be small / inconsequential…
• … but risk is always there
• Testers / developers work together to reduce risk
29
July 24, 2016
Beizer’s Testing Level 4
• Testing is a mental discipline that helps all IT professionals
develop higher quality software
• Analogy: spell checker
• Purpose is to improve our ability to spell, not just find misspelled words
• Learn from our mistakes
30
July 24, 2016
Beizer’s Testing Levels
• We probably start at level 0, 1, or 2
• Goal is to reach 3 and 4 (eventually)
• What level are you?
31
July 24, 2016
Outline
•
•
•
•
•
Test Engineer
Levels of Testing
Example Software Failures
Beizer’s Testing Levels
Automation of Testing Activities
32
July 24, 2016
Automation of Test Activities
• Testing is expensive and labor intensive
• 50% of development costs
• More for safety-critical applications
• Goal: automate as much as possible
• Reduce cost
• Minimize human error
• Make regression testing easier
33
July 24, 2016
Candidates for Automation
• Excise tasks
• Do not contribute directly to solution of the problem
• Examples:
•
•
•
•
Compiling a class into object code
Rerunning test scripts
Comparing expected vs. actual results
(Things the require minimal thought)
34
July 24, 2016
NOT Candidates for Automation
• Revenue tasks
• Contribute directly to the solution of the problem
• Example
• Determining methods for a class
• Developing the software’s architecture
• (Things that require thought)
35
July 24, 2016
Next Steps
• Familiarize yourself with Blackjack
• Work on homework_01
36
July 24, 2016