0.20
Quality Manager
(Orchestrazione automazione test del Sw)
1
Test Management
2
© Leonardo - Finmeccanica - Società per azioni
1. Test Management
In this module we will address the following topics:
1. Test Management goals
2. Validation vs Verification: Defining the right system vs constructing
the right system
3. Traceability in the lifecycle
4. Requirement & Requirement testing: Taxonomies & applicability
5. Test Types
6. Test confidence, reliability & residual defects
3
© Leonardo - Finmeccanica - Società per azioni
Brief Discussion
What are typical problems faced by development teams and why?
Are you working with all informations?
Have you complete informations to work?
If not, can you have prompt answers to your questions/issues?
Will things change on-the-fly when you are working on it?
Have you sufficient time to complete on time the
assigned work?
Did you feel confident that on delivery your artifacts
are complete and do not need more changes?
What improvements can be made?
To the tools?
To process?
To training?
4
© Leonardo - Finmeccanica - Società per azioni
Symptoms
User or business needs not met
Insufficient requirements
Boiling Requirements
Ambiguous communication
Modules not integrating
Brittle architectures
Difficulties with maintenance
Overwhelming complexity
Late discovery of flaws
Undetected inconsistencies
Poor quality of end-user experience
Poor performance under load
No coordinated team effort
Build-and-release issues
Poor testing
Subjective assessment
Waterfall development
Uncontrolled change
Insufficient automation
5
© Leonardo - Finmeccanica - Società per azioni
Root Causes to Symptoms
User or business needs not met
Insufficient requirements
Boiling Requirements
Ambiguous communication
Modules not integrating
Brittle architectures
Difficulties with maintenance
Overwhelming complexity
Late discovery of flaws
Undetected inconsistencies
Poor quality of end-user experience
Poor performance under load
No coordinated team effort
Build-and-release issues
Poor testing
Subjective assessment
Waterfall development
Uncontrolled change
Insufficient automation
6
© Leonardo - Finmeccanica - Società per azioni
Trace symptoms to root causes
Symptoms
Resolving
Root causes
Address
Best practices
Needs not met
Insufficient requirements
Boiling Requirements
Ambiguous communication
Develop iteratively
Modules do not fit
Brittle architectures
Manage requirements
Hard to maintain
Overwhelming complexity
Late discovery of flaws
Undetected inconsistencies
Poor quality
Poor testing
Model visually (UML)
Poor performance
Subjective assessment
Continuously Verify quality
Colliding developers
Waterfall development
Build-and-release
Uncontrolled change
Insufficient automation
© Leonardo - Finmeccanica - Società per azioni
Use component architectures
Manage change
7
Discussion
Testing models vary widely:
–What types of testing do you do?
–What do you test?
–When do you test?
–How formal is your process?
8
© Leonardo - Finmeccanica - Società per azioni
Test Management Goals
Organizing resources and artifacts involves organizing and maintaining an
inventory of items to test and assembling a group of testers.
Define specific
test cases, and
Determine why,
develop test
what, where, and
scripts to
when to test.
implement the
test cases.
Monitor the test
effort, analyze test
results,
communicate
project status and
product quality
Run test
scripts and
test suites,
and submit
defects.
The planning, estimating, monitoring, and control of test
activities, typically carried out by a test manager.
© Leonardo - Finmeccanica - Società per azioni
− ISTQB Glossary
s
9
Test management challenges
How do you determine the current quality of your application under test
(AUT)?
How do you define, measure, and track quality goals?
How do you coordinate the work of all people who are involved in the testing
effort?
How do you track dependencies and relationships among test assets?
How do you plan, execute, and evaluate your testing iteratively?
10
© Leonardo - Finmeccanica - Società per azioni
Review requirements to reduce rework
Relative Cost to Correct a Defect
120
100
80
60
40
20
0
© Leonardo - Finmeccanica - Società per azioni
Rework can consume
30% - 50% of your
development costs:
Requirements errors
account for 70% 85% of rework costs.
Preventing
requirements errors
and catching them
early has a huge
leveraging effect on
reducing rework.
Wiegers, 2003.
11
Requirements are an input to test planning
What’s new in
this release?
What should I
test?
Does the system
meet security
expectation?
What is the intended
system behavior for
this operation?
Does the system
meet reliability
expectations?
What are the
performance
expectations?
Test Manager
Does the system
meet safety
expectations?
Does the system
meet robustness
expectations?
© Leonardo - Finmeccanica - Società per azioni
Are all compliance
criteria satisfied?
Does the system
meet usability
expectations?
12
Quality stabilization process
Defects
Test Case
Satisfy
Design
Process
Validation
Requirement
Verification
Specification
Test Results
Document Results
Changes
The
Product
Drive Implementation Of
Implementation
© Leonardo - Finmeccanica - Società per azioni
13
..But we have more perspective to capture with
requirement sets (each requirement lives in a specific set)
Is Verified By
Needs
UR
Verify
System
Verification
System
Design
Verification
Design
Validation
User
Acceptance
Implementa
tion
14
© Leonardo - Finmeccanica - Società per azioni
Overall System and Software Quality by design
Applicable Standards
(Guide and
Compliance)
User
Requirements
Context of use
Satisfy
Satisfy
Satisfy
The Product
Validation – Proves that the specified product design is coherent
Verification – Measures that the real product meet specifications
15
© Leonardo - Finmeccanica - Società per azioni
…Standards origin
1994
EIA/IS632
Systems
Engineering
1969
MIL-STD499
1974
MIL-STD499A
1994
MIL-Std499B
(Not
Released)
1980
MIL-STD1679A
(Interim)
1998
IEEE 1220
1998
1994
IEEE 1220
1994
(Trial Use)
Software
Engineering
1968MIL-STD1679
1998
ANSI/EIA632
1985
DOD-STD2167
Data Item
Descriptions
1968-
1995
ISO/IEC
12207
1988
DOD-STD2167A
1987
DOD-STD1703
1988
DOD-STD7935A
1994
MIL-STD498
1995
IEEE 1498
/EIA 640
(Draft)
1995
EIA/IEEE
J-STD-016
2003
ISO/IEC
19760
Guide
1999
EIA/IS- CMMI
731
(SE-CM)
2003+
ISO/IEC
15288
IEEE
1220
Harmon
2003+
ISO/IEC
15288
12207
Harmon
1999-2002
Instructions/
Handbooks/
Manuals/
Guides
Supersedes
Derived From
(Interim)
2003+
DIDs
Defense
Specifications
1998
IEEE/EIA
12207
2002
ISO/IEC
15288
DIDs
1995
MIL-STD961D
2003
MIL-STD961E
16
© Leonardo - Finmeccanica - Società per azioni
Adapted from John O. Clark INCOSE 2003 Speech
Standard Normalization
Satisfy
Stakeholder Need
System
Software
System
User Requirement
ORD, ICD
TLR
SSS, Internal IRS
System Requirement
SRD, SS,
External IRS
SRS + (SW)IRS
Subsystem
Requirement
SSS, Internal IRS
Architecture
SDD, IDD
Design Requirement
SSDD, DBDD,
SRS, HRS
Detailed Design
Implementation
Requirement
Drawings
Code
Implementation
IEEE 15288, IEEE 12207
MIL-STD 499 / EIA632
MIL-STD 498
17
© Leonardo - Finmeccanica - Società per azioni
Test Traceability
Verification
Solution against expectations
User Requirement
System Requirement
Subsystem
Requirement
User Acceptance
System against requirements
System Testing
and Qualification
Subsystems against requirements
Collaborations
Components against requirements
Integration Testing
Design Requirement
Implementation
Requirement
Component against requirements
Unit Testing
Intrinsic Product Quality Area
In-context Product Quality Area
18
© Leonardo - Finmeccanica - Società per azioni
In-context quality
+ Robustness
19
© Leonardo - Finmeccanica - Società per azioni
Intrinsic Product Quality
+ Compliance with standards
20
© Leonardo - Finmeccanica - Società per azioni
Quality Impact
21
© Leonardo - Finmeccanica - Società per azioni
How measure the missing quality - Terms
Issue
Is measured as the difference between actual (prescribed) and
expected
Impact
Is the effect of the issue in the context of use
Defect
Is the origin that originates the issue
Error
Is what is introduced by one missing or failed activity
Remediation
Is the proposed (then accepted) fix that shall be implemented
Implementation
Is the fix implementation and will be released in product lifecycle
22
© Leonardo - Finmeccanica - Società per azioni
Verification
Verification Level
System
( Element )
Subsystem
( Equipment )
Unit
Test Kind
Destructive
Typical to check robustness over design limits
Non-Destructive
All other cases
Post-Mortem
Robustness of weared out items
Verification Methods:
Analysis (A) – Typical in R&D Phase & Qualification
Prove by design review the existence of the requirement implementation
Inspection (I) – Typical in Production
Prove by inspecting the produced element
Demonstration (D) – Typical in Qualification
Demonstrate the presence of the requirement by direct measure
Test (T) – Any R&D and Production phase
Measure pragmatically using a specific procedure
23
© Leonardo - Finmeccanica - Società per azioni
Different test levels addressed
System Level
and Qualification
•
Verifies that the system requirements (functional
and QoS) are present in the product.
•
Verifies that the designed parts integrates
correctly satisfying the designer expectations.
(functional and QoS)
Verifies the interface requirements match the
implementation
Unit Level
•
Verifies that the single unit match the expected
behavior and QoS
….. Tests
•
Verifies that the asset are in compliance with
process guidelines
Integration Level
24
© Leonardo - Finmeccanica - Società per azioni
Test Types
Black Box Testing
Injected Condition
Context integration
White Box Testing
System Test
Integration Test
Unit Test
25
© Leonardo - Finmeccanica - Società per azioni
Typical test types usage
System Testing (Black Box)
It verifies that (1) the system satisfies the given
requirements and does not have unexpected behavior
or break when used incorrectly (functional robustness).
Used in system and software system also for
qualification.
System Integration Testing (White Box)
Verifies and prove how the internal states and parts
influences the behavior and qualities of the subsystem
or part. Examples are safety, reliability, performance.
Used in proofing the final system or part design in pr
black box testing, certifications or R&D
Integration Testing (White Box)
Unit Testing (Black Box)
Verify and prove selected parts integration on how they
fit together on interfaces design, integrated
collaboration, robustness, integrity, performance. Used
in R&D to proofing selected parts integration and
robustness post unit test and before testing the whole
block
It is used in post-production to verify and prove that the
built part and implementation satisfy or exceed
specified design requirement. Used in R&D and
factories after implementation before making the part
available for integration or assembly
26
© Leonardo - Finmeccanica - Società per azioni
Requirements in the RM application
Requirement
collection
In the Requirements Management application:
– A requirement collection is a set of high-level
requirements that are grouped for a specific
purpose, such as a particular release or functional
area.
– A requirement defines a specific capability that the
application under test must provide.
– Requirement types include features, non-functional
requirements, and user stories.
Requirement
27
© Leonardo - Finmeccanica - Società per azioni
Test Confidence, Reliability and Residual Defects
The verification process, by verifying the item allocated requirements,
may address:
The correct behavior and context integration interfaces
Limits and boundary conditions
Integrated error and failure protection mechanisms and handling
Safety (countermeasures verification)
Performance (Volumes, Responses times, Capacity)
Reliability (prediction of fault probability, in-range conditions)
Integrity (confidence)
Robustness (resilience, breaking conditions, out-of-range and exceeded
assumptions behaviors, non damage limits)
…
Each verification context may develop test plans to address quality
scope or which defect size, if existent, may be pass undetected.
28
© Leonardo - Finmeccanica - Società per azioni
Reliability: Each QoS can influence others
Function
State
Hazard
Safety
Issue
Performance
Security
Incident
QoS
Break
Robustness
Integrity
Issue
29
© Leonardo - Finmeccanica - Società per azioni
Requirements-based testing – Testing confidence
The requirements-based testing (RBT) process addresses
two major issues:
– first, validating that the requirements are correct, complete,
unambiguous, and logically consistent;
– and second, designing a necessary and sufficient set of test
cases from those requirements to ensure that the design and
implementation fully meet those requirements.
30
© Leonardo - Finmeccanica - Società per azioni
Requirements-based testing strategy
The overall RBT strategy is to integrate testing throughout the development life
cycle and focus on the overall quality addressing the following benefits:
– Early defect detection
– Defect prevention, not just defect detection
– Reduced cost of defect detection and remediation
– Reduced time to delivery
– Increased probability of successfully delivering the right solution
31
© Leonardo - Finmeccanica - Società per azioni
2
Quality Manager & Team
Concert Integration
Overview
32
© Leonardo - Finmeccanica - Società per azioni
2. Quality Manager & Team Concert Integration Overview
In this module we will address the following topics:
1. Tools vs Toolchains: activites vs Process
2. Lifecycle project, QM Project Area, CM Project Area, RM Doors,
Rhapsody Designer
3. Test Management and Change management integration within
Rational Quality Manager and Rational team Concert planning
33
© Leonardo - Finmeccanica - Società per azioni
Lifecycle project
A set of related project areas that are linked
Centrally managed through the Lifecycle Project Administration (LPA)
application
Quality
Management
project area
Quality
management
tasks
Defects
Change and
Configuration
Management
project area
Requirements
Implementation
requests / tasks
Requirements
change
requests
Requirements
Management
project area
34
© Leonardo - Finmeccanica - Società per azioni
Project area
The context in which teams perform their work
Provides access to all project artifacts, such as plans, work items,
requirements, and test cases
Controls project access, membership, roles, and permissions
Created by a user with JazzAdmins or JazzProjectAdmins repository
permissions
Managed by a local project administrator
Quality Management
project area
Quality
management
tasks
Defects
Change and
Configuration
Management
project area
© Leonardo - Finmeccanica - Società per azioni
Quality
Management
project area
Requirements
Implementation
requests / tasks
Requirements
change requests
Requirements
Management
project area
Artifacts
Project access
and members
Roles and
permissions
35
Banner bar (1 of 2)
Banner elements are consistent among Jazz applications
Project (Application)
Help Menu
Logged in user
Home Menu
Use the Home menu to:
Navigate among Quality Management
projects.
Access Change and Configuration
Management and Requirements
Management projects.
Access the Jazz Team Server Home or
Lifecycle Project Administration application.
36
© Leonardo - Finmeccanica - Società per azioni
Banner bar (2 of 2)
Use the Profiles menu to:
View information about
your user profile
Set user preferences
Log out
Use the Admin menu to:
Access Quality Management project
properties and timelines
Manage the Quality Management project
area
Create or manage lifecycle projects
Go to the Jazz Team Server home page
Profiles menu
Admin menu
37
© Leonardo - Finmeccanica - Società per azioni
Search
This search found four
resources that contain the text
bill :
A test script
A test case
A test case execution record
A test case result
Select the types of
resources that you want
to include in the search .
38
© Leonardo - Finmeccanica - Società per azioni
Filter
Use the filter box to limit a list to items that contain the specified text.
Clear Filter Text
39
© Leonardo - Finmeccanica - Società per azioni
Quality management test artifact map
Development
plan
The Quality Management application uses artifacts to contain and
organize project information. Links create relationships between artifacts.
Requirement
collection
Test plan
Requirement
Test case
Development
work item
Test script
Test suite
Test suite
execution record
Test suite result
Test case
execution record
Test case result
Defect
Test
environment
Relationships
Bidirectional link
Parent / child
© Leonardo - Finmeccanica - Società per azioni
Test data
40
Quality management capability menus
Use the capability menus to:
– Open a dashboard.
Browse, create,
and import
– Review requirements.
artifacts
– Manage test plans.`
– Develop test cases and test scripts.
– Review test execution records.
– Run reports.
– Create and review change requests.
See recently
viewed artifacts
and lists
41
© Leonardo - Finmeccanica - Società per azioni
The Quality Management project dashboard
Save changes automatically
Click an
artifact to
open it.
Tasks and reviews
assigned to the
logged in user.
Work items
Report data that shows
test execution status
Project timelines
and members
Log of events that have occurred
in the QM application
The dashboard displays a variety of widgets that provide views into projects.
42
© Leonardo - Finmeccanica - Società per azioni
Hover for Details
Hover over an artifact link
in a list to see information
about the artifact.
© Leonardo - Finmeccanica - Società per azioni
Review summary information about
an artifact before you decide whether
to click and open the artifact.
43
About the Mini Dashboard
Add Widget
Pin or Unpin
Functions like the main dashboard
Persists across applications and QM pages
Quick show and hide (or pin open, docked
on left or right )
To open, click the vertical Mini
Dashboard bar on the left.
To close, click anywhere
outside the Mini Dashboard.
44
© Leonardo - Finmeccanica - Società per azioni
3
Test Activities Planning
45
© Leonardo - Finmeccanica - Società per azioni
3. Test Activities Planning
In this module we will address the following topics:
1. Test Plan Description: Objectives, Time, Resources and Risks
2. Test Plan Templates
3. Assigning Quality Tasks to teams
4. Formalizing Test Plans
5. Risk Management in test plans: risk profiles, probability and impact
6. Test Schedule & Test Estimation
7. Test Platform & Test Environment
8. Review and approval of test plans
9. Test Plan Reports & affected requirements
10.Snapshot & audit trails
11.Test Plans and release management timeline
46
© Leonardo - Finmeccanica - Società per azioni
Orientation: Rational Quality Manager
Start planning the test effort by identifying allocated requirements.
Identify
requirements
Develop a
test plan
Develop test
cases and
test suites
Develop test
scripts
Run test
cases and
test suites
Report
defects
Generate
reports
47
© Leonardo - Finmeccanica - Società per azioni
Test artifacts related to the test plan
Define the test effort
with quality goals
Define what to test and
the test procedure
Detail how to execute the
test case
Test plan
Test suite
Group related test
cases to be executed
together
Test case
Test script
48
© Leonardo - Finmeccanica - Società per azioni
Test plan
Test plan
A document defining the scope, approach, resources, and schedule of intended
test activities… It is a record of the test planning process.− ISTQB Glossary
A Quality Management test plan:
– Defines scope, objectives, risks, schedules, work estimates,
entry, and exit criteria, and team members
– Is a quality contract for the entire team
– Is the foundation of planning the test effort
– Includes static and dynamic content
– Has associations with requirement collections, development
plans, test cases, test suites, test environments, and child test
plans
– Can have one or more child test plans or one parent test plan
49
© Leonardo - Finmeccanica - Società per azioni
Test motivators
Test motivators are anything that the test team uses to help
determine what to test.
System
specifications
Use-case
model
Design
specifications
© Leonardo - Finmeccanica - Società per azioni
Requirement
s
Change
requests
Test plan
Iteration plan
Risk lists
50
Test plan header and summary section
Execution Progress
ID and Name
State, Action,
Originator,
Owner, and
Priority
Description
Selected
section area
51
© Leonardo - Finmeccanica - Società per azioni
Test plan sections
A set of predefined sections may be included in the Test Plan as needed. Each
section brings specific content and is shown inside the section area
Rich text
Predefined content
52
© Leonardo - Finmeccanica - Società per azioni
Test plan customization
Test Plans can be customized to suit the specific target or test effort.
Customization are made by:
– Add sections
– Remove sections
– Reorder sections
– Create custom sections
Create a category values.
53
© Leonardo - Finmeccanica - Società per azioni
Add, remove, and reorder test plan sections
1
Click to hide or
show available
sections
Create
Section
Move
Up
Move
Down
2
Sections not in
the test plan
Sections included in
the test plan
Add
Add All
Remove
Remove All
© Leonardo - Finmeccanica - Società per azioni
3
54
Create a custom test plan section
1
2
3
4
5
The rich text
editor opens
55
© Leonardo - Finmeccanica - Società per azioni
Rich text editor toolbar
Bold / Italic / Underline / Strikethrough
Subscript / Superscript
Paragraph Format
Font Name
Text Color
Background Color
Remove Format
Font Size
Insert Table
Insert Horizontal Line
Align Left / Right / Center / Justified
Increase / Decrease Indent
Numbered / Bulleted List
Cut / Copy
Paste / Paste as plain text /
Paste Special
Undo / Redo
Find
Maximize
Validate Content
Insert Image
Insert Attachment
Insert Link
Remove Link
56
© Leonardo - Finmeccanica - Società per azioni
Add values to a category
1
3
2
4
5
Optional: Set one value as
the default value.
Click to add
another value.
6
7
57
© Leonardo - Finmeccanica - Società per azioni
Artifact Template
• Templates define structure only, not content (all sections
are empty).
• Templates can include predefined and custom sections.
• Templates are provided for these artifacts:
– Test plans
– Test cases
– Test suites
• A template is an ordered set of artifact sections.
• Use templates to standardize the structure of test artifacts.
58
© Leonardo - Finmeccanica - Società per azioni
Standardization of structure and contents
• To standardize the content in test artifacts, create a
baseline artifact (or skeleton) as you will then save as
“Template” item.
• When you need the artifact, duplicate it to create new
artifacts.
• To help compiling the artifact as your intent, include test
group standards and instructions to testers in the baseline
artifact.
59
© Leonardo - Finmeccanica - Società per azioni
Manage templates
Click Admin > Manage Artifact Templates.
Grouped by:
Ungrouped
Type
Filter for specific text
All templates that include
the filter text, grouped by
artifact type
60
© Leonardo - Finmeccanica - Società per azioni
Manage templates toolbar
Use the toolbar to complete template operations
Copy
Template
Refresh Templates
Copy Template and
Set as new default
are enabled when a
template is selected.
Set as new
default
Archive
Change Display Settings
Archive is enabled only
if you have permission
to archive a template.
61
© Leonardo - Finmeccanica - Società per azioni
Create a test plan template (1 of 2)
To create a
template, either
create a
template…
1
2
2
… or copy an existing
template, and then edit
it to suit your needs.
1
62
© Leonardo - Finmeccanica - Società per azioni
Create a test plan template (2 of 2)
Add, remove, and
reorder test plan
sections in the
template
Use the Up and
Down buttons to
reorder sections.
3
Sections not in
the template
Click to make this
template the
default template.
Sections included
in the template
Add
Add All
Remove
Remove All
4
63
© Leonardo - Finmeccanica - Società per azioni
Create test plan template toolbar
New
Edit
Down
Up
Delete
Edit is available only when a
custom section is selected.
Delete is available only if you
have permission to delete a
template section.
Click New to create a custom
section.
64
© Leonardo - Finmeccanica - Società per azioni
Validating copied text
Use the Validate Content function to clean up copied
text after pasting into a rich text section of the test plan.
This will clean-up all unsupported formatting tags that
may have been copied with the text
1
2
3
65
© Leonardo - Finmeccanica - Società per azioni
Quality objectives
Measurable criteria that define the overall quality goals for athe
test plan
Use quality objectives to:
– Establish measurement criteria and enter status updates
– Define entry and exit criteria for testing
– Measure progress on the overall level of quality
– Respond to the question, “Are we ready to release?”
66
© Leonardo - Finmeccanica - Società per azioni
Add quality objectives to the test plan (1 of 2)
1
2
Ctrl+click to select
multiple quality
objectives
3
67
© Leonardo - Finmeccanica - Società per azioni
Add quality objectives to the test plan (2 of 2)
Click Evaluate Quality
Objectives to see current
actual values (for predefined
quality objectives only).
Selected quality objectives
are added to the test plan.
1
2
68
© Leonardo - Finmeccanica - Società per azioni
Create a custom quality objective
1
2
3
4
69
© Leonardo - Finmeccanica - Società per azioni
Lab 1: Create a test plan template
• Complete these tasks:
Review the test plan template
reference.
Copy a test plan template and rename
the copy.
Add, remove, and reorder test plan
sections.
Make the new test plan template the
default test plan template.
Create a Baseline Artifact
• Add quality objectives.
• Create a quality objective
70
© Leonardo - Finmeccanica - Società per azioni
Quality Tasks - Assign test plan sections to team members
1
2
3
Caution:
The task work item is
not created until the
test plan is saved.
4
71
© Leonardo - Finmeccanica - Società per azioni
Task created in the Change Management application
Click the link in the test plan
section to open the task work
item in the Change
Management application.
72
© Leonardo - Finmeccanica - Società per azioni
Hover on Quality Task to Monitor test plan sections
assigned to team members
Tip:
A test plan cannot be reviewed and approved until
all associated tasks (work items) are complete.
The Test Plan Work Items list
displays tasks that are
associated with sections of the
test plan.
Owner
Status
Due date
Link to related test plan
73
© Leonardo - Finmeccanica - Società per azioni
Different test plan objectives
• Iterations and Release (Verification)
– Regression Test Plans
– Non Regression Test Plans (Change)
• Specific Qualities to be tested
– Safety Countermeasures Test Plans
– Security Measures Test Plans
– Performance/Volumes Test Plans
– Reliability Test Plans
–…
74
© Leonardo - Finmeccanica - Società per azioni
Grouping Test Plans
• Test Plans can be grouped in a parent-child relationship
(one level only)
• Change Management drives Non-Regression test plans
• Configurations drive Regression Test Plans
Rel n Changes
Master
Test Plan
+
Non Regression
Test Plan
Child Plan
Functional Test
Plan
Child Plan
Performance Test
Plan
Child Plan
Safety
Countermeasures
Test Plan
Rel n-1 Tests
=
Rel n+1 Regression Tests
Regression Test
Plan
Regression Test
Plan
Functional Test
Plan
Functional Test
Plan
+
Performance Test
Plan
Safety
Countermeasures
Test Plan
=
Performance Test
Plan
Safety
Countermeasures
Test Plan
Test plans are the correct test artifacts to group tests for objective
© Leonardo - Finmeccanica - Società per azioni
75
Create a test plan
1
2
3
4
5
76
© Leonardo - Finmeccanica - Società per azioni
Detail a test plan
Create a test plan – results in a mostly empty shell
Detail a test plan – involves providing details for each section
The information for test plan sections often comes from related
documentation that is already available.
If you want to create a Child Test Plan, use the Child Test Plans
section
77
© Leonardo - Finmeccanica - Società per azioni
Labs 2, 3 and 4: Customize a test plan and delegate
sections
• Complete these tasks:
Remove a test plan section.
Create a test plan section.
Create a table in a test plan section.
Add a test plan section.
Attach a document to the test plan.
Create a category value.
Change a category value.
Create tasks to direct other team
members to complete sections of the
test plan.
78
© Leonardo - Finmeccanica - Società per azioni
Test plans are the base to organize tests
• Test Plans maintain test cases together
• Test Plans cover the specific test verification objective
• Test Plans document the whole test maturity and coverage
79
© Leonardo - Finmeccanica - Società per azioni
Test planning and risk related to the test
• Capturing risks in the test plan can:
– Prompt early risk assessment and mitigation planning
– Provide a basis for test prioritization
– Enable a project manager to assign resources where they are
most valuable
– Focus the project team on reducing risk whenever possible
80
© Leonardo - Finmeccanica - Società per azioni
Definitions
• Risk: A factor that could result in future negative consequences;
usually expressed as impact and likelihood.
•
• Risk analysis: The process of assessing identified risks to estimate
their impact and probability of occurrence (likelihood).
• Risk-based testing: An approach to testing to reduce the level of
product risks and inform stakeholders of their status, starting in the
initial stages of a project. It involves the identification of product risks
and the use of risk levels to guide the test process.
• − ISTQB Glossary
Standard glossary of terms that are used in Software Testing, Version 2.1
International Software Testing Qualifications Board
81
© Leonardo - Finmeccanica - Società per azioni
Test-Related Risks
Twenty risks are defined in two types:
business and technical. You can
define additional risks as required.
Risks and risk
profiles are
defined in the
Quality
Management
project properties.
82
© Leonardo - Finmeccanica - Società per azioni
Risks in the test plan
The overall risk assessment (for the
artifact) is calculated by taking the
average of all individual risk assessment
scores weighted by importance.
Current impact is only
included in the average if
it is high, medium, or low.
average
83
© Leonardo - Finmeccanica - Società per azioni
Risk profiles
A risk profile is an artifact-specific set of risks.
Risk profiles provide standardization and efficiency.
One risk profile is predefined
for each artifact type. You can
define additional risk profiles
for each artifact type as
required.
Risks and risk
profiles are defined
in the Quality
Management
project properties.
Each risk profile includes
one or more risks.
NOTE: You can create specific risk profiles for different test plan types
© Leonardo - Finmeccanica - Società per azioni
84
Initial risk assessment
Typically by test manager or test plan owner
Provides overall risk score for the artifact
Steps:
– If required: Switch to a more appropriate risk profile.
– Delete inappropriate risks, and add relevant risks.
– Evaluate (edit) risks.
– Optional: Create work items for testers to complete the My Risk
score and provide comments.
85
© Leonardo - Finmeccanica - Società per azioni
Risk Assessment toolbar
Switch Risk Profile
Change Display Settings
Edit Risk
Remove Risk
Add Risk
86
© Leonardo - Finmeccanica - Società per azioni
Switching a risk profile
1
2
3
Replace substitutes the
risks in this profile for the
existing risks in the test
plan
Append adds the risks in
this profile to the existing
risks in the test plan
Risks from the selected risk
profile replace existing risks in
the test plan.
87
© Leonardo - Finmeccanica - Società per azioni
Adding a risk to the test plan
1
Risks by type
Technical risks
2
3
Business risks
2
Select a risk type and risk.
Evaluate the likelihood and
impact of the risk.
Identify an action to mitigate
the risk.
88
© Leonardo - Finmeccanica - Società per azioni
Editing a risk and setting mitigation actions
2
1
Evaluate the likelihood and
impact of the risk.
Identify an action to mitigate
the risk.
3
4
5
6
Assess the current impact of the risk.
Calculates the importance of the risk.
© Leonardo - Finmeccanica - Società per azioni
89
Community risk assessment
Typically by test team members
May include other members of the project team
Produces calculated community risk score
Steps:
1. Select a My Risk score by clicking one of the five
circles.
When you save the artifact,
1
your comment is added to
2. Optionally, type a comment.
2
the discussion and your My
Risk score is added to the
community risk.
90
© Leonardo - Finmeccanica - Società per azioni
Lab 5: Define and assess test risk
• Complete these tasks:
Capture risks for a test plan.
Assess risk as a test manager.
Assess risk as a tester, adding to the
community risk assessment.
91
© Leonardo - Finmeccanica - Società per azioni
Test schedules
A test schedule lists the activities, tasks, or events that make up a test
plan.
These groups of activities are often called milestones or iterations.
Test iterations often mirror development iterations, but can also include
iterations that are specific to the test effort.
A point value is an estimate of the execution effort for the iteration.
92
© Leonardo - Finmeccanica - Società per azioni
Test Schedules section of a test plan
Two sections show how key dates align with, or relate to, test iterations.
Test schedules
define iterations
within the test effort.
Key dates reflect high-level,
product-related events, such
as beta version delivery or
ship-readiness.
© Leonardo - Finmeccanica - Società per azioni
93
Test Schedules toolbar
Show Archived
Browse (select an iteration)
Change Display
Settings
Clear
(clear all
iterations)
94
© Leonardo - Finmeccanica - Società per azioni
Timelines and iterations
One project timeline is enabled (by default).
A timeline hierarchy includes child iterations.
Iterations with a release
scheduled can be added to the
test schedule for a test plan.
Note:
To create or edit a timeline, you must be
an administrator of the project.
95
© Leonardo - Finmeccanica - Società per azioni
Change display settings
To see more of the
name and description
of each test schedule,
change the display
settings.
Add Select
Add All
Remove Select
Remove All
Reallocate
horizontal space
Wrap text instead ellipsing (…)
Reduce vertical space
Reduce font size
96
© Leonardo - Finmeccanica - Società per azioni
Define test schedules on test plan (1 of 3)
1
2
3
97
© Leonardo - Finmeccanica - Società per azioni
Define test schedules (2 of 3)
4
5
6
7
8
98
© Leonardo - Finmeccanica - Società per azioni
Define test schedules (3 of 3)
Instead of using auto
generation, you can click
(Add Row) to add rows
manually and insert progress
points at whatever intervals
suit your reporting needs.
Auto Generation spreads points over
the duration of the iteration in
increments: daily, weekly, or monthly.
9
10
11
Points attempted and points
completed both increase
over time and the gap
between attempted and
completed decreases to zero
by the end of the milestone.
12
99
© Leonardo - Finmeccanica - Società per azioni
Define key dates
1
2
Examples of key dates:
– End of iteration boundary minus one week (a checkpoint)
– Translation drop
– Beta delivery
– Ship-readiness
100
© Leonardo - Finmeccanica - Società per azioni
Test estimation
Estimates are often based on what is known about the project
requirements.
Provide high-level estimates of the time that is required to
– Complete your test planning activities
– Run all of your tests
Update estimates as the test effort progresses.
Execution estimates for individual test cases are captured in the test
case weight.
101
© Leonardo - Finmeccanica - Società per azioni
Lab 6: Set schedule and estimates
• Complete these tasks:
Establish the test schedule for a test
plan.
Estimate the test effort for a test plan.
102
© Leonardo - Finmeccanica - Società per azioni
Platform coverage
Represents the sum of all hardware and software configurations that:
– The product will support
– The test effort will cover
Enables generation of specific test environments
103
© Leonardo - Finmeccanica - Società per azioni
Define platform coverage
1
Tip:
Define platform coverage broadly
by including all environment types
and their attributes that you
anticipate covering. You can
selectively remove specific test
environments later.
2
3
4
When you select an
environment type, attributes for
that type are displayed in the
Available list (and the
Selected list, if any attributes
have been selected).
Selected attributes
are listed.
104
© Leonardo - Finmeccanica - Società per azioni
Test environment coverage examples
Does the business require full coverage of all
environment types or all permutations?
Example testing matrix:
– Operating systems: Microsoft Windows 32-bit, Linux
– Databases: Apache Derby, IBM DB2®
All permutations
All environment types
Windows 32 Linux Derby
Env 1
Env 2
Windows 32 Linux Derby
DB2
Env 1
Env 2
Env 3
Env 4
© Leonardo - Finmeccanica - Società per azioni
DB2
105
Test Plan Lifecycle
Actions move an artifact from one state to another.
Draft
ready for
review
This model applies also to:
• Test plans
• Test cases
• Test scripts
• Test suites
• Test case results
• Test suite results
reject
Under
Review
approve
reopen
Approved
retire
reopen
Tip:
All work items associated with an artifact must be
resolved
before
the artifact can be Under Review.
© Leonardo - Finmeccanica
- Società per
azioni
Retired
106
Test plan review process
The test manager determines that the test plan is
sufficiently developed to get feedback from the test
team and other project members.
Draft
Assign
approver
Approve
Assign
reviewer
Review
Each Approver changes the
review status from Pending to
Approved or Rejected.
Under
Review
Approved
Each Reviewer changes the
review status from Pending to
Reviewed.
The test manager also completes the test plan review
task, and approves the test plan.
107
© Leonardo - Finmeccanica - Società per azioni
Prepare a test plan for review
2
3
This review is for … and shall …
4
1
5
Review and
Approval follow the
same actions but
have different
means
6
8
7
108
© Leonardo - Finmeccanica - Società per azioni
Starting from Dashboard Widget
Review a test plan (1 of 2)
1
3
2
109
© Leonardo - Finmeccanica - Società per azioni
Review a test plan (2 of 2) – E-Signature
4
5
A project administrator can
configure a precondition to
require an e-signature when an
artifact is reviewed or approved.
110
© Leonardo - Finmeccanica - Società per azioni
Test Plan Reports and Affected Requirements
111
© Leonardo - Finmeccanica - Società per azioni
Recall: Test plan approval activities and Checklist
Is the review complete?
Task-review history shows
actions by reviewers and
approvers
Test plan Show Sections
button shows all test plan
sections on one page
Create a PDF file to review
the test plan offline
Create a snapshot
Approve and then save the
test plan
Lock the test plan to prevent
reopening it
Check approvals state
Check task-review history
Resolve task-review
Review test plan in Show Sections
Optional: Create test plan PDF file
Create a snapshot of the test plan
Change test plan status to Approved
Save the test plan
Optional: Lock the test plan
112
© Leonardo - Finmeccanica - Società per azioni
About snapshots
Read-only version of an artifact
Create snapshots of these artifacts:
–
–
–
–
Test plan
Test case
Test script
Test suite
Captures the artifact at a point in time:
– Provides version history, audit trail
– Facilitates progress tracking
– Enables roll-back
Roll-back: copy a snapshot to create an editable copy of a previous
version of the artifact (new version, not replace)
Test plan
© Leonardo - Finmeccanica - Società per azioni
A copied test plan snapshot does not retain
the contents of the Formal Review, Test
Schedules, and Test Environments sections
113
Create a snapshot
1
2
3
4
New snapshot
is added to the
snapshots list
114
© Leonardo - Finmeccanica - Società per azioni
About locking
Any artifact can be:
– Locked by a user with Lock permission
– Unlocked by a user with Unlock permission
Lock and unlock permissions are granted by role and by
artifact type.
A locked artifact cannot be edited.
To lock an artifact, click the lock icon.
To unlock an artifact, click the lock icon.
115
© Leonardo - Finmeccanica - Società per azioni
Lab 7: Approve the test plan
• Complete these tasks:
Check the test plan review status.
Resolve the test plan review task.
Approve the test plan.
Create a PDF file of the test plan.
Create a snapshot of the test plan.
116
© Leonardo - Finmeccanica - Società per azioni
Test Plans & Release Management Timeline
Development Timeline (RTC)
Defect
Testing Timeline (RQM)
Iterations
Requirements
Dev
Release 1.0
Test
Release 1
117
© Leonardo - Finmeccanica - Società per azioni
Ship
4
Constructing test cases
118
© Leonardo - Finmeccanica - Società per azioni
4. Constructing test cases
In this module we will address the following topics:
1.
2.
3.
4.
5.
Creating and detailing test cases
Test requirement coverage generation
Relating test cases to other CLM artifacts
RQM Test case vs RTC workitems relationship
Validating Requirement, Change and Test
119
© Leonardo - Finmeccanica - Società per azioni
Orientation: Rational Quality Manager
Identify
requirements
Develop a
test plan
Develop test
cases and
test suites
Develop test
scripts
Run test
cases and
test suites
Report
defects
Generate
reports
120
© Leonardo
- Finmeccanica
- Società per azioni
Now
we create
test cases.
Recap: Were we are
Development
plan
Requirement
collection
Test plan
Requirement
Test case
Development
work item
Test
script
Test
data
Test suite
Test
environme
nt
Test suite
execution
record
Test suite
result
Test case
execution
record
Test case
result
Defect
Relationships
Bidirectional link
Parent / child
121
© Leonardo - Finmeccanica - Società per azioni
Test artifacts related to the test case
Define the test effort
Test plan
Test suite
Group related test cases
to be executed together
Test cases group can be
executed concurrently
or sequentially.
Define what is to be
tested
Test case
Detail how to implement
the test case
Test script
Test
environment
Define where a test is to
be performed
122
© Leonardo - Finmeccanica - Società per azioni
Test Case
A Test Case is a measure that shall be taken and compared with expected
result (that is the Verification Point) to prove requirement implementation.
Test Case at least have:
Test Case Name
Input Data (may use equivalence classes)
Verification Method (Inspection, Test, Analysis, Demonstration)
Test Procedure/Description (a procedure that ends with the measure
that shall prove pass or fail)
Test Script (the sequence of actions that implements the test procedure)
Trace to Verified Requirement (if test verification point pass, the
requirement is present and implemented)
It is important noticing that, if the verification point cannot be reached, the
test is Inconclusive
123
© Leonardo - Finmeccanica - Società per azioni
Test case summary
ID and Name
Artifact type
State, Originator,
Owner, and Priority
Description
Selected
section
Test case weight is measured
in points and reflects
execution effort.
Start defining a standard,
such as one hour of execution
effort equals 100 points.
124
© Leonardo - Finmeccanica - Società per azioni
Test cases view
Click a column name:
• Once to sort ascending
• Once more to sort descending
• Once more to remove the column sort
Click the Change icon to
change the column value
for all selected test cases.
125
© Leonardo - Finmeccanica - Società per azioni
Test cases view filtered by category
Filter the list of test cases by one or more hierarchical categories
Test cases of
type Dividend…
…sorted in descending
order by Priority
126
© Leonardo - Finmeccanica - Società per azioni
Test cases view grouped
Group test cases by selected field
Expand and collapse groups
Number of test
cases in each group
127
© Leonardo - Finmeccanica - Società per azioni
Test cases traceability view
Use the traceability view to see:
• The requirements each test case validates
• The development items each test case tests
128
© Leonardo - Finmeccanica - Società per azioni
Test cases toolbar
Manage Test Case Categories
Refresh
Download as spreadsheet(.csv)
Change
Display
Settings
129
© Leonardo - Finmeccanica - Società per azioni
Test cases action menu
Perform selected actions on one or more selected test cases
130
© Leonardo - Finmeccanica - Società per azioni
Lab 8: Create a test case
• Complete these tasks:
Create a test case from the
Construction menu.
Create a test case from within a test
plan.
131
© Leonardo - Finmeccanica - Società per azioni
Generate test cases automation
B
E
F
O
R
E
A
F
T
E
R
Requirement
collection
Test plan
Requirement
Requirement
Requirement
Requirement
collection
Test plan
Requirement
Requirement
Requirement
Test case
Test case
Test case
The test case generation wizard:
1. Generates a test case for each
requirement in the requirement
collection that is linked to the test
plan.
2. Adds new test cases to the test
plan.
3. Links each new test case to its
corresponding requirement.
132
© Leonardo - Finmeccanica - Società per azioni
Generate test cases from requirements (1 of 7)
Select one or more requirement
collections and then click
(Reconcile Requirements in
Collections)
2
1
133
© Leonardo - Finmeccanica - Società per azioni
Generate test cases from requirements (2 of 7)
Use the filter box to find
specific requirements
without test coverage.
3
Select some or
all of the listed
requirements.
Click Next as
required to review all
listed requirements.
4
134
© Leonardo - Finmeccanica - Società per azioni
Generate test cases from requirements (3 of 7)
Use the filter box to find
specific requirements
for which you want to
generate test cases.
5
Select all or only
some of the listed
requirements.
Click Next as
required to review all
listed requirements.
6
135
© Leonardo - Finmeccanica - Società per azioni
Generate test cases from requirements (4 of 7)
Select values that are
appropriate for all the test
cases to be generated,
and leave other values
unassigned.
7
8
9
10
By default, the test plan is
the one in which you
clicked
(Reconcile
requirement collections,
generate new test cases).
136
© Leonardo - Finmeccanica - Società per azioni
Generate test cases from requirements (5 of 7)
The wizard:
1. Creates one test case for each
selected requirement.
2. Links each new test case with its
corresponding requirement.
3. Adds all of the new test cases to
the test plan that you specified.
137
© Leonardo - Finmeccanica - Società per azioni
Generate test cases from requirements (6 of 7)
Wait until all
actions are
complete…
…and then
click Next.
11
138
© Leonardo - Finmeccanica - Società per azioni
Generate test cases from requirements (7 of 7)
12
13
139
© Leonardo - Finmeccanica - Società per azioni
Test case generation results (1 of 2)
Generated test
cases are
added to the
test plan.
140
© Leonardo - Finmeccanica - Società per azioni
Test case generation results (2 of 2)
A link to the
corresponding
requirement is added to
the generated test case.
141
© Leonardo - Finmeccanica - Società per azioni
Lab 9: Generate test cases from requirements
• Complete these tasks:
Generate test cases from a
requirement collection.
Verify new test cases and links to
requirements.
142
© Leonardo - Finmeccanica - Società per azioni
Relationships between Test Cases and CLM Artifacts
Requirement
Change
Request
Implementation
Request
Task
Implement
Workitem
Workitem
RTC
Verify
Changes
Workitem
Verify
Doors / DNG
Test case
Changes
Requirement
Tests
Configuration
RQM
RTC
Workitem
Quality Task
RTC
143
© Leonardo - Finmeccanica - Società per azioni
Validation of Test Cases
•
•
Each Test Case shall trace to one requirement
Each Test Case shall trace to one development item if implemented
144
© Leonardo - Finmeccanica - Società per azioni
5
Using test suites
145
© Leonardo - Finmeccanica - Società per azioni
5. Using Test Suites
In this module we will address the following topics:
1. Creating and defining Test Suites: functional and strategic view
2. Monitoring test activities
3. Parallel vs Sequential execution of the test suite
146
© Leonardo - Finmeccanica - Società per azioni
Orientation: Rational Quality Manager
Now we want to group test cases into test suites as appropriate.
Identify
requirements
Develop a
test plan
Develop test
cases and
test suites
Develop test
scripts
Run test
cases and
test suites
Report
defects
Generate
reports
147
© Leonardo - Finmeccanica - Società per azioni
Recap: Test artifacts related to the test case
Define the test effort
Test plan
Test suite
Group related test cases
to be executed together
Test cases group can be
executed concurrently
or sequentially.
Define what is to be
tested
Test case
Detail how to implement
the test case
Test script
Test
environment
Define where a test is to
be performed
148
© Leonardo - Finmeccanica - Società per azioni
Execution variables in Test Suites
Another important aspect of the Test Suites is the
capability to use Execution Variables in the owned test
cases
Replaces literal values in a test script
Values can be changed at run time
Facilitate passing data:
From a test suite to a test case in the test suite.
From one test case to another in a test suite (sequential
execution only).
From a test suite or test case to a test script or keyword script.
Highest level values prevail: Execution variable values
in a test suite supersede those in test cases in the test
suite
© Leonardo - Finmeccanica - Società per azioni
149
Develop a test suite (1 of 5)
Create the test suite.
6
2
1
3
4
5
150
© Leonardo - Finmeccanica - Società per azioni
Develop a test suite (2 of 5)
Add the test suite
to a test plan.
7
8
9
151
© Leonardo - Finmeccanica - Società per azioni
Develop a test suite (3 of 5)
Complete test suite
sections and add test
cases to the test suite.
10
12
11
13
14
152
© Leonardo - Finmeccanica - Società per azioni
Develop a test suite (4 of 5)
15
Recalculate the
test suite weight.
Remember: as initial
performance, 100 = 1 hour
153
© Leonardo - Finmeccanica - Società per azioni
Develop a test suite (5 of 5)
Define execution variables.
16
17
18
19
154
© Leonardo - Finmeccanica - Società per azioni
Lab 10: Develop a test suite
• Complete these tasks:
Create a test suite.
Add the test suite to a test plan.
Associate test cases with the test suite.
155
© Leonardo - Finmeccanica - Società per azioni
Monitoring test activities – Consolidated View
156
© Leonardo - Finmeccanica - Società per azioni
Monitoring test activities – Detailed View
157
© Leonardo - Finmeccanica - Società per azioni
Suites and Test Case Execution Strategy
Test cases execution can be setup for Sequential Execution (order is fixed)
Function
Function
Function
Function
1
2
3
4
Test case 1
Test case 2
Test case 3
Test case 4
or for Parallel Execution (order is casual)
Function
Function
Function
Function
1
2
3
4
Test case 1
Test case 2
Test case 3
Test case 4
158
© Leonardo - Finmeccanica - Società per azioni
Where to use suites
Sequential Suites
If one test completion is one precondition of another (chained
requirements)
If one tester alone is testing one part of the system
Parallel execution Suites
If the test shall verify a whole set of requirements together
If you have many testers involved in the same plan and want to allocate
test case execution to different testers
Each Test Case Execution will produce a Test Case Execution Record
(TCER)
The suite have a Test Suite Execution Record that brings all Test Case
Execution Records inside
Remember: Test cases Result can be Pass, Fail or Inconclusive
© Leonardo - Finmeccanica - Società per azioni
159
6
Test Scripts
160
© Leonardo - Finmeccanica - Società per azioni
6. Test Scripts
In this module we will address the following topics:
1.
2.
3.
4.
5.
Test Scripts: Difference between Test Script and the test case
Manual Test Script: Goals and Definition
Manual Test Script: Supporting Manual Test Script Creation
Manual test Script: Keywords
Automatic Test: Example of automatic test tool integration
161
© Leonardo - Finmeccanica - Società per azioni
Test Script vs Test Case
Test script
is a step-by-step test execution that ends with the Verification Point. Reaching
and passing the Verification Point will demonstrate that the test case passes.
Each Test Environment may require different script steps implementation. For
each step, scripts can have related images and annotations to help the
execution. Each step will require a “pass” to continue the test. Script can be
executed with the same result for same equivalence class
Test Case
Is the container of the measure with all the information related. It will contain one
or more script for different test contexts if the interaction will change. Defining the
test cases implies that the equivalence classes for the requirement are identifed.
SS The system shall determine the drop
order on the aircraft when load dropped
TC Verify that the system correctly determines
the drop order of the aircraft when load
dropped
Test Inputs / Conditions:
•Aircraft loaded with 5 x 200 kg
•Flight Condition: 10.000 feet
SCRIPT
•Ensure Current Balance for the actual load
•Drop one of the 200kg weight load from the aircraft
•Verify that the drop sequence is calculated correctly
162
© Leonardo - Finmeccanica - Società per azioni
Equivalence Classes
To do some test compression, we can divide dimensions in four distinct disjoint sets:
In-set values (more than 18)
values in this set represent an expected case of the requirement. All values in the in-set are
represented by one central value of the set.
The user shall have more than 18 years to drink beer
In-set values: 25
Out-set values (not more than 18)
values in this set represent a negative of the expected case expressed in the requirement.
All values out-set are represented by one value of the set.
The user shall have more than 18 years to drink beer
Out-set values: 10
On-Boundary values ( between 17 and 18)
values in this set represent limits that are close to In-set and Out-set values. Because
defects tends to be generated also by rounding and flooring of values, these On-Boundary
values may give some reliability of the behavior and meta-stable behavior.
The user shall have more than 18 years to drink beer
On-Boundary values: 17.9999999 and 18.0000001 (if 1 exp -9 precision floats used)
Invalid-set values
Values that cannot be part of the requirement by domain
The user shall have more than 18 years to drink beer
Off-set values: -0.01, NaN, Infinite
© Leonardo - Finmeccanica - Società per azioni
163
Manual Test Script: Goals and Definition
1
2
3
© Leonardo - Finmeccanica - Società per azioni
164
Define test case design
You can describe the test procedure also using steps.
3
1
2
165
© Leonardo - Finmeccanica - Società per azioni
Manual Script Automatic Generation
Specifies terms that indicate when a test case design statement
should be a reporting step within the generated manual test script
Should be reviewed and updated as required after designing a test
case and before generating a manual test script from a test case
design
1
3
2
166
© Leonardo - Finmeccanica - Società per azioni
Labs 11: Detail a test case and Update the manual script
preferences
• Complete these tasks:
Complete sections of the test case.
Define a test case design.
Update the manual script preferences.
167
© Leonardo - Finmeccanica - Società per azioni
Keywords in Rational Quality Manager
Keywords are reusable test scripts that consist of one statement or more
related statements. Think of keywords as “mini-scripts”.
You can reuse keywords in multiple test scripts.
Keywords provide one location for frequently-used sequences, and therefore
reduce test script maintenance.
You can use keywords exclusively or combined with local statements in a
test script.
Keywords can be associated with manual or automated test scripts.
168
© Leonardo - Finmeccanica - Società per azioni
Keywords view
The Keywords view is a repository for reusable content. In this view, you can
do these activities:
– Create keywords
– Filter keywords by name and tag
– Edit keyword name and tags
169
© Leonardo - Finmeccanica - Società per azioni
Creating keywords
Click Construction > Browse Keywords
On the toolbar, click Create Keyword
Type a name and tags for searching
Associate the keyword with a test script, or
create keyword content
170
© Leonardo - Finmeccanica - Società per azioni
Creating keywords from existing statements
Shift+Click to select a series of statements in a test script
Drag the selected statements to the Keyword View
2
1
171
© Leonardo - Finmeccanica - Società per azioni
Creating keywords from existing statements
(continued)
Type a name and tags
Choose whether to replace statements in the test script with the new
keyword
3
4
The new keyword is added
to the Keyword View and
inserted into the test script
172
© Leonardo - Finmeccanica - Società per azioni
Keywords in a script
Keywords have these characteristics:
– More indentation than local statements
– The keyword icon
– Read-only in the test script
– Changes to the source content (the keyword script) are reflected
immediately
The keyword icon and the name
of the keyword are followed by
statements in the keyword script
173
© Leonardo - Finmeccanica - Società per azioni
Insert keyword example
2
1
174
© Leonardo - Finmeccanica - Società per azioni
7
Test Execution
175
© Leonardo - Finmeccanica - Società per azioni
7. Test Execution
In this module we will address the following topics:
1. Test Case Execution Record
2. Test Execution Record and Test Context
3. Automatic Test Execution environments
4. Integrating RQM with Automatic Test Environments
5. Tracking results with IBM Rational Toolchain
6. Verification of the test results
7. Doors-RQM Integration for detecting suspects
8. Doors and DNG (Doors Next Generation)
9. History of test execution with TCER/Plan Execution Records
10.Regression and Non-Regression Testing
11.Automatic Test Script Execution tools
176
© Leonardo - Finmeccanica - Società per azioni
Run a single Test Case
Run a test case or a test case execution record.
Run the test case
In the Test Case
Execution Records
run the test
177
© Leonardo - Finmeccanica - Società per azioni
Manual script execution view
Progress indicator
Script execution toolbar
Current step
pointer
Work through a manual test by
alternating between the execution
view and the AUT, performing steps
and applying results.
178
© Leonardo - Finmeccanica - Società per azioni
Script execution toolbar
Pass
Click to Pause Execution
Fail
Apply All
Create New Defect
Show/Hide
Comments
Column
Comments
Attachments
Link To Existing Defect
179
© Leonardo - Finmeccanica - Società per azioni
Test run completed
The test is finished running when:
– Execution completed is displayed at the end of the script steps and in
the upper-right corner of the script.
– The progress indicator shows 100%.
180
© Leonardo - Finmeccanica - Società per azioni
About execution results
Execution
result
Generated when a test run is complete
Two types:
– Test case results (TCER)
– Test suite results (TSER)
Linked to the execution record (TCER or TSER)
Sections of the result artifact contain information about the test run
Each result is added to previous results, providing a complete history
181
© Leonardo - Finmeccanica - Società per azioni
Reviewing test results: Result details (1 of 2)
Summary information
You can change the
actual result value to fit
your own interpretation
of the test run.
Result
sections
Step results by type
182
© Leonardo - Finmeccanica - Società per azioni
Reviewing test results: Result details (2 of 2)
Result applied to each step
Manual test script steps are
shown with their results.
Actual results recorded
during the test run
Requirement that
the step validates
Links to attachments
and defects that were
associated to the step
during the test run
183
© Leonardo - Finmeccanica - Società per azioni
Reviewing test results: Weight distribution
Use weight distribution sliders to reflect different results for portions of the test.
25 Passed points
+13 Failed points
38 Attempted points
In this example, 38 points were attempted, and the remaining points were
blocked by the failed step.
184
© Leonardo - Finmeccanica - Società per azioni
Associate a blocking defect with a TCER
Link To
Existing
Defect
1
2
Create New
Defect
3
4
5
The resulting defect
has a Blocks link to
the TCER in RTC
185
© Leonardo - Finmeccanica - Società per azioni
Lab 12: Run the test case
• Complete these tasks:
Run the test case
Access the Test Case Execution
Record
Access the Execution Result of the Test
Case Execution Record
186
© Leonardo - Finmeccanica - Società per azioni
Lab 13: Examine a test result
• Complete these tasks:
Open and review a manual test case
result.
Change the actual result and weight
distribution.
Make the associated defect a blocking
defect.
187
© Leonardo - Finmeccanica - Società per azioni
Test Execution Record and Test Context
…Test Plan
Test Suite…
Test Suite
Execution Record
(TSER)…
Test Case
Execution Record
(TCER)…
188
© Leonardo - Finmeccanica - Società per azioni
Automatic Test Execution Integration: RFT
Click Start > All Programs > IBM Software Delivery
Platform > IBM Rational Functional Tester > Adapter
to Rational Quality Manager > Configure Adapter.
2
3
1
4
5
189
© Leonardo - Finmeccanica - Società per azioni
Run a Rational Functional Tester test (1 of 2)
1
The Script Type column
identifies the tool in which
the test script was created.
© Leonardo - Finmeccanica - Società per azioni
190
Run a Rational Functional Tester test (2 of 2)
Adapter status
information
The computer
and adapter
on which the
test script will
be run
2
3
191
© Leonardo - Finmeccanica - Società per azioni
Rational Functional Tester test script playback
The execution view and the Rational Functional Tester playback monitor.
The execution view shows the
computer details, the adapter that is
in use, the test environment, and a
progress indicator.
On RFT Client
The automated test script that is running
The active test script statement
192
© Leonardo - Finmeccanica - Società per azioni
Reviewing Rational Functional Tester test
results (1 of 2)
The overview and weight distribution sections are similar to those for a manual
test result.
193
© Leonardo - Finmeccanica - Società per azioni
Reviewing Rational Functional Tester test
results (2 of 2)
Verification
point summary
Distribution of
step results
Each Rational Functional Tester test
script step is listed with its script line
number and the result of performing the
step.
Rational Functional Tester logs
are attached.
© Leonardo - Finmeccanica - Società per azioni
194
Rational Functional Tester simple log
The simple log is a text-based
log of events that occurred
during the run of the Rational
Functional Tester test script. It
opens in a web browser.
195
© Leonardo - Finmeccanica - Società per azioni
Rational Functional Tester detailed log
The detailed log opens in the Rational
Functional Test Log viewer (available only if
Rational Functional Tester is installed on
the computer on which you want to view the
log).
196
© Leonardo - Finmeccanica - Società per azioni
Tracking result with Rational Toolchain (RQM) Live
197
© Leonardo - Finmeccanica - Società per azioni
IBM Rational RELM
When reports are too flat, you can use RELM*
IBM Rational RELM is the tool to layout graphical queries result
transversing overall asset showing their status
Interfaces and extract artifacts from
Quality Management
Requirement Managmenet
Change Management
Design Management
Extract artifacts using query and filters
Queries and draws links between artifact types
Free-form dashboard to visually track results and status
Analyzing artifacts dependencies using ontologies
Understanding what’s wrong
(*) Requires RELM license
© Leonardo - Finmeccanica - Società per azioni
198
Tracking result with Rational Toolchain (RELM)
199
© Leonardo - Finmeccanica - Società per azioni
Example Using Ontologies
Verification of the test results
Suite:
200
© Leonardo - Finmeccanica - Società per azioni
Doors-RQM Integration for detecting suspects
When the requirement in Doors changes, the test case is marked as Suspect
Is up to the test designer understand what the difference is and accordly update
the test case and test case script
201
© Leonardo - Finmeccanica - Società per azioni
Doors and Doors Next Generation
Doors is the mature rich-client based Requirement Management Tool
Doors Next Generation is the web-based rich-client Requirement
Management Tool
202
© Leonardo - Finmeccanica - Società per azioni
Doors Next Generation Capabilities
Figure 1: (Formal) Modules
Figure 3: Processes and flows
© Leonardo - Finmeccanica - Società per azioni
Figure 2: Graphical UI Sketches
Figure 4: Device UI Sketches and Storyboards
203
Test Traceability in Doors and Doors NG
Doors
Doors NG
204
© Leonardo - Finmeccanica - Società per azioni
History of Test Execution With TCER and Test Plan
For the single Test Case
Newest
Oldest
For the Test Plan
Newest
Oldest
© Leonardo - Finmeccanica - Società per azioni
205
Regression and Non-Regression Testing
Regression Testing
Verifying that the system continues to work
Non-Regression Testing
Verifying that the new added changes works as expected
@Release 1
Release 1
Verification
Plan
Release 1
NonRegression
Verification
Plan
Test Cases
@Release 2
+
Release 2
Verification
Plan
Release 2
NonRegression
Verification
@Release 3
+
Release 3
Verification
Plan
Release 3
Nonregression
Verification
206
© Leonardo - Finmeccanica - Società per azioni
Integrating RQM with Automatic Test Environments
RQM
TC
Script Ref
RQM can integrate with any external test engine
using one agent.
Many agents can be registered on the RQM
application inside the QM Project Area
If not already existent, may be implemented (requires
API usage and code implementation)
Agent Hub
External Test
Engine
Run
TCER
Result
Create
Define
Maintain
Adapter
Browse / References
External Test Script
Repo Script
Third Party Tool
207
© Leonardo - Finmeccanica - Società per azioni
Automating Tests Script Execution
Out of the box, you can run automated test scripts that were created with:
Rational Functional Tester
Rational Performance Tester
Rational Service Tester for SOA Quality
Rational Robot
Rational AppScan Tester Edition
Command-line adapter (any tool/technology)
Third-party Adapters
NI Test Integration for TestStand and LabView (National Instruments)
Squish for QT (Frologic)
Configure and run the corresponding adapter.
The test script in the Quality Management application contains a reference
to the test script that was created in the external tool.
208
© Leonardo - Finmeccanica - Società per azioni
8
Reporting & Collaboration
209
© Leonardo - Finmeccanica - Società per azioni
8. Reporting & Collaboration
In this module we will address the following topics:
1.
2.
3.
4.
5.
6.
Test Team Load and Test Completeness Reports
Dashboards
Measuring Test Coverage of Requirements and Releases
Reporting using Report Designer
Reporting using IBM Rational Publishing Engine
Generating MIL-STD 498 documents
210
© Leonardo - Finmeccanica - Società per azioni
Reporting with RQM
In RQM Reports may be shared or personal
Shared reports can be generated by the user selecting already deployed
reports or developing customized one (need Rational for Developer
Intelligence)
Each report may have specific parameters to filter in informations to render
on it
211
© Leonardo - Finmeccanica - Società per azioni
Shared reports view
Select a report folder to see its individual reports.
Report types
Filter reports
list
Execution reports
212
© Leonardo - Finmeccanica - Società per azioni
Reports toolbar and actions
Import Custom Report
Create Folder
Create Report
Refresh
Delete Report
Move/Copy Report
Delete
Report
Edit
Move/
Copy
Report
213
© Leonardo - Finmeccanica - Società per azioni
Report parameters
Each report has its own set of parameters.
214
© Leonardo - Finmeccanica - Società per azioni
Parameter dependencies
Some parameters are dependent on the values selected for a related
parameter.
Some test milestones
are listed twice because
they are associated with
two test plans.
Dependency indicator
Selecting a test plan limits the Test
Milestones list to test milestones that are
associated with the selected test plan.
© Leonardo - Finmeccanica - Società per azioni
215
Report descriptions
Read the report description to understand the intended purpose of a report.
216
© Leonardo - Finmeccanica - Società per azioni
The Quality Management project dashboard
“My Task” Widget
217
© Leonardo - Finmeccanica - Società per azioni
Add a widget to the dashboard
You can customize your dashboard
by adding widgets.
1
5
2
3
4
218
© Leonardo - Finmeccanica - Società per azioni
Add a page to the dashboard
Click the Add New Tab
button to create a page
for the dashboard.
Copy a tab to paste to
another dashboard.
Double-click to edit
the page name.
Duplicate a tab within
this dashboard.
Change the
page layout.
219
© Leonardo - Finmeccanica - Società per azioni
Configuring shown widgets
You can change the appearance or content of a widget.
Appearance options
are the same for all
widgets.
Settings options vary by the
type of widget. The settings
that you see apply to the type
of content that the widget
displays.
© Leonardo - Finmeccanica - Società per azioni
220
Dashboards
There are:
One personal dashboard for each Jazz Member
One team dashboard for each Jazz Team
One Project Area Dashboard for each Project Area
Widget Sources
External Widget Sources
Example
© Leonardo - Finmeccanica - Società per azioni
221
Measuring Test Coverage of Requirements and Releases
RQM Test Case
© Leonardo - Finmeccanica - Società per azioni
Requirement
RTC
Workitem
Script
Requirement
Closed
Missing
New/Open
Script Pass
Script Fail
Missing
No result
Missing Script
222
Reporting using Report Designer
Starting from Jazz 6.0 platform version, a Report Designer is included in the
Jazz Plaform
With Report Designer, starting from any artifact you can:
Design tabular reports and graphical reports
Extract data following link and presenting in the reports
Adding any attribute data of the involved artifacts to the report
Add calculated value columns to the report
It is very simple to use
223
© Leonardo - Finmeccanica - Società per azioni
Report Designer Creation Wizard - Data
https://<jazz-server:port>/rs
1
Choose Report Type
2
Choose an Artifact
3
Choose Relation
224
© Leonardo - Finmeccanica - Società per azioni
Report Designer Creation Wizard - Formatting
225
© Leonardo - Finmeccanica - Società per azioni
Report Designer Creation Wizard – Saving Definition
226
© Leonardo - Finmeccanica - Società per azioni
Resulting Report
227
© Leonardo - Finmeccanica - Società per azioni
Report Designer Creation Wizard – Graphical Trend
1
2
3
228
© Leonardo - Finmeccanica - Società per azioni
Report Designer Creation Wizard – Graphical Trend
229
© Leonardo - Finmeccanica - Società per azioni
Rational Publishing Engine Overview
Rational Publishing Engine is a generic tool to extract
information from external data sources (as tools, files,
databases) and render it on a document, following one
design template.
It needs to know and understand the domain that will
import
For each tool / repository supported, RPE will have an
Input Loader that can implicit reference an input
schema or, for example using XML or REST, need one
definition for it.
230
© Leonardo - Finmeccanica - Società per azioni
Rational Publishing Engine (RPE)
Document
Template
Document
Template
Document
Template
Document
Specification
Input
Loaders
Core
Output
Formatters
Data
Output
Document template
– Defines structure of input sources
– Defines output document content and hierarchy
– Relates input to output using queries
Document specification
– Associates concrete data with the input sources of document template(s)
– Defines concrete output document types
© Leonardo - Finmeccanica - Società per azioni
231
RPE Output Formatters
Word – Microsoft® Word®
– Generates documents in 2
Word formats – ‘97-2003
(.doc)’ and ‘2007 (.docx)’
– Documents can be opened by
any version of Word that
supports either of those
formats
– Supports *.dot stylesheets
– Can run a macro after
document generation, which is
defined in the *.dot file
PDF – Adobe® Portable
Document® files
XslFo – XSL Formatting
Objects
– XML based standard defined
by W3ORG
– 3rd party tools available for
converting XSL-FO to various
other formats
HTML – Browser web pages
– Supports *.css style sheets
232
© Leonardo - Finmeccanica - Società per azioni
RPE Input Loaders
DOORS Database – Database structure from Rational® DOORS®
DOORS – Requirement modules from Rational® DOORS®
Rational Publishing Engine supports Rational® DOORS® version 9.1 and above
Tau – projects containing UML-2 models and diagrams
Rational Publishing Engine supports Rational® Tau™ version 4.2.0.1 and above
Generic XML – information stored in XML files
any XML file matching the information structure definition of the document template
REST – Information provided by a RESTful web-service interface with the needed
capabilities for Rational Publishing Engine (Rhapsody is supported through REST
Integration and domain.xsd document)
233
© Leonardo - Finmeccanica - Società per azioni
Doors Database Input Loader
Worldwide unique database ID
Follow this URL to open database in a
DOORS browser
Recursive folder structure allows
reporting with recursive queries
Projects have “true” here
Modules inside Folder/Project
No information about baselines, yet
Fullname and id allow configuration of
dynamic data source
“Formal”, “Link” or “Descriptive”
Follow URL to open the module
© Leonardo - Finmeccanica - Società per azioni
234
Rational Quality Manager Reporting API
Feed Schema
https://[host]:[port]/[contextRoot]/service/com.ibm.rqm.integration.service.IIntegrationService/schema/feed.xsd
Quality Manager Schema
https://[host]:[port]/[contextRoot]/service/com.ibm.rqm.integration.service.IIntegrationService/schema/qm.xsd
Feed Query Endpoint
https://[host]:[port]/[contextRoot]/service/com.ibm.rqm.integration.service.IIntegrationService/resources/[Project Area name]/[object]
Objects
adapter
catalog
configuration
executionresult
executionworkitem
jobscheduler
objective
requirement
resourcegroup
teamarea
testphase
testsuitelog
© Leonardo - Finmeccanica - Società per azioni
attachment
category
contributor
executionscript
instructions
keyword
projects
reservation
step
template
testplan
workitem
builddefinition
categoryType
datapool
executionsequence
job
labresource
remotescript
resource
suiteexecutionrecord
testcase
testscript
buildrecord
channel
executionelementresult
executionsequenceresult
jobresult
labresourceattribute
request
resourcecollection
tasks
testcell
testsuite
235
RPE - Template Document Principle
236
© Leonardo - Finmeccanica - Società per azioni
9
Teamwork for Testing
237
© Leonardo - Finmeccanica - Società per azioni
9. Teamwork for testing
In this module we will address the following topics:
1. Defining and allocating test resources on test activities
2. Test Activity management
3. Review and approvals
238
© Leonardo - Finmeccanica - Società per azioni
Rational Quality Manager Labs
To manage labs and resources, Rational Quality Manager uses this simple mechanism:
Lab
Inventory
Lab Resource
Group
Request
Cell
Lab Resource
Reservation
Test Suite
Run
Test
Environment
239
© Leonardo - Finmeccanica - Società per azioni
When Running Test Suites
For Environment
In Cell
240
© Leonardo - Finmeccanica - Società per azioni
Devices Inventories
241
© Leonardo - Finmeccanica - Società per azioni
The cell where the test will take place…
242
© Leonardo - Finmeccanica - Società per azioni
Planning the availability of the devices…
243
© Leonardo - Finmeccanica - Società per azioni
Test Activity Management – Lifecycle sample
Define Test Plan
Goals
Define Test
Cases
Define
Review
Define
Review
Define Test
Suites
Define Test
Scripts
Run Tests
Define
Define
Approve
Approve
Review
Review
Approve
Approve
Run
Review
Approve
244
© Leonardo - Finmeccanica - Società per azioni
Review and approval
Peer Reviewers
Approver
One or more Approvals can be requested for different means
© Leonardo - Finmeccanica - Società per azioni
245
Customizable special behaviors (samples)
Auto-lock Triggers
When the object is moved to Approved state, a lock is applied.
When the object is moved to Draft state, the lock is removed.
…
Save Restrictions
If the object is in Approved state, prevent save
Do not permit Under Review state until all Quality Task items are
resolved
Do not approve if review pending
…
246
© Leonardo - Finmeccanica - Società per azioni
250
© Leonardo - Finmeccanica - Società per azioni