IEEE Software Engineering Standards Testing
Table of Contents
Introduction .....................................................................................................................................1
Testing Objectives .......................................................................................................................1
Aspects of Standards Testing ......................................................................................................1
Two Viewpoints on Standards Testing....................................................................................1
4-level Model for Standard Maturity .......................................................................................2
Standard Types ........................................................................................................................2
Characteristics of Standards Testing ...........................................................................................3
Static vs. Dynamic Evaluation ................................................................................................3
Objectivity of Testing ..............................................................................................................3
“Positiveness” of Testing ........................................................................................................3
Standards Maturity Levels ...............................................................................................................4
Level 1: Need Satisfaction...........................................................................................................4
Viewpoint: Contents ................................................................................................................4
Viewpoint: Presentation ..........................................................................................................4
Level 2: Correctness ....................................................................................................................4
Viewpoint: Contents ................................................................................................................4
Viewpoint: Presentation ..........................................................................................................5
Level 3: Effectiveness .................................................................................................................5
Viewpoint: Contents ................................................................................................................5
Viewpoint: Presentation ..........................................................................................................6
Level 4: Attractiveness ................................................................................................................7
Viewpoint: Contents ................................................................................................................7
Viewpoint: Presentation ..........................................................................................................7
Annex 1. Popularity of Standards ................................................................................................8
Introduction
Testing Objectives
The permission letter entrusting the standards testing to the student laboratory in the Ukrainian
Software Engineering Training Center states that “the goal of the workshop is to make software
engineering standards more efficient by testing them in a real production environment and
thereby getting recommendations and comments on using them.”
In our opinion the ultimate goal of the testing is to make the standards more useful and popular.
Aspects of Standards Testing
After some practical work on the standards testing we gained an understanding that the standards
should be considered in several aspects and several points of view, such as: presentation and
contents, standard “maturity”, standard type. Let’s consider them in more detail.
Two Viewpoints on Standards Testing
Below we will use the word “Viewpoint” having in mind standard Presentation and standard
Contents. Presentation includes document’s structure, design and appearance while contents is
the ideas behind the text.
Testing of the Contents is more important for us since IEEE Standards Board and members of
the balloting pool already paid much attention to the standards’ Presentation.
4-level Model for Standard Maturity
We found convenient to use a CMM-like model for testing (evaluating) the standards. We have
distinguished four levels of a standard maturity. The model can be applied in both staged and
continuous representation. The analogue for the CMMI “Process Area” is Viewpoint – contents
or presentation.
Level 1. Need Satisfaction
 Viewpoint: Contents
The standard satisfies actual need(s) of the industry; it addresses some real problems.
 Viewpoint: Presentation
The standard satisfies primary needs of its user (contains necessary sections etc.).
Level 2. Correctness
 Viewpoint: Contents
The standard does not contain false statements.
 Viewpoint: Presentation
The standard does not contain ambiguities, self-contradictions; it is in line with formal
requirements like IEEE Standards Style Manual or S2ESC Guide for Working Groups.
Level 3. Effectiveness
 Viewpoint: Contents
The standard makes positive effect while being applied in practice; the effect of using the
standard is more valuable as compare to alternative solutions.
 Viewpoint: Presentation
The standard is usable enough.
Level 4. Attractiveness
 Viewpoint: Contents
The standard is a unique source of specific information.
 Viewpoint: Presentation
The standard’s appearance and design appeal to particular qualities of human perception,
and do it successfully.
The maturity levels are cumulative just as CMMI ones, e.g. compliance with 2nd level means
satisfaction of both 1st and 2nd level requirements. And in the same way as in CMMI, assignment
of a particular requirement to a certain level is to a great extent subjective – e.g. some
requirements could be attributed to Level 1, but we included them into Level 3 set of
requirements etc.
Standard Types
Unlike traditional way of classifying IEEE standards (Standard, Recommended Practice, Guide)
we will consider a slightly different set of types that are more useful from the testing point of
view:
 Uniformity Statement (could be also “vocabulary”, “dictionary”)
 Concept Description
 Document Description (“template”)
 Practical Guide
On any given maturity level the requirements to different types of standards are slightly
different, and the methods of testing are also different.
Characteristics of Standards Testing
Static vs. Dynamic Evaluation
We found convenient to distinguish 2 kinds of standards testing – static evaluation and dynamic
one (similar to software verification techniques). By static evaluation we mean just reading and
inspection of the standard. Conversely, dynamic evaluation means application of the standard in
practice.
Dynamic evaluation is more important for us since IEEE Standards Board and members of the
balloting pool already inspected the standards’ text.
It is mandatory to perform dynamic evaluation wherever it is possible. In practice that means that
we should apply the standard in a production environment first, and only then analyze
satisfaction of the requirements of maturity levels.
Dynamic evaluation is difficult for Uniformity Statements and Concept Descriptions though.
Objectivity of Testing
In any case testing results should be objective. That means, in particular, measurability of testing
results and conformance of testing method to a preconcerted set of requirements wherever
possible.
“Positiveness” of Testing
The testing results should contain some useful suggestions rather than pure critique for the
defects found.
Standards Maturity Levels
Level 1: Need Satisfaction
Viewpoint: Contents
Requirement1. The standard addresses some real problems of the industry.
How to Test
1. Make the list of ALL the problems resolved by the standard (let’s call it List of
Addressed Problems).
2. For each problem estimate its urgency for the industry.
It’s advisable to get measurable evaluation, e.g. numeric results of expert poll.
3. For each problem estimate the degree of satisfaction of the corresponding need by the
standard.
It’s also advisable to get numeric results.
Requirement2. The standard addresses all the problems related to the concerned problem area.
How to Test
1. Make precise definition of the concerned area.
The Overview section of the standard can be used to get initial understanding of the
standard’s scope. Although we cannot fully trust the declared scope, and after the practical
use of the standard we should revise the definition of the concerned area.
2. Make the list of the problems related to the concerned area (let’s call it List of Related
Problems) for all categories of the standard’s users (target audiences).
3. Compare the List of Addressed Problems with the List of Related Problems.
“Positive” testing. It’s advisable to make some suggestions of how to address the items
from the List of Related Problems which are not present in the List of Addressed Problems.
Viewpoint: Presentation
Requirement1. The standard includes Index section if necessary.
IEEE Standards Style Manual says that “Indexes are discouraged unless the document is
very long or complicated”. However we experienced Index to be useful in most paper
documents (equivalent to”Find Text” feature in electronic documents).
How to Test
1. If the index is present ensure it contains all the helpful words and phrases (including
keywords declared on a Title page).
Level 2: Correctness
Viewpoint: Contents
Requirement1. The standard does not contain false statements which are illogical or not proved
to be true in practice.
How to Test
1. Apply each statement in practice if possible.
It’s advisable to make some numeric evaluations which prove the correctness of the
statement, e.g. some statistics.
Requirement2. For each problem addressed by the standard all the necessary aspects are
covered.
How to Test
1. The questions should be answered: which statements are not complete, what is not taken
into account.
Requirement3. The standard is approved by well-known industry experts.
How to Test
1. At least one prominent expert was invited to the Working Group or Balloting Pool.
It’s difficult to say whether the particular person is a world- known expert or not. One of
the possible methods is to search amazon.com for the books of this author; at least one of
them has to be so popular that the Spotlight Review section displayed instead of Customer
Review (that means that only part of reviews are displayed).
Requirement4. The standard is harmonized with other IEEE and ISO/IEC standards.
How to Test
1. Review all other software engineering standards concerning the same subject area.
At least the standard should be tightly adjusted to the clauses of IEEE/EIA 12207.0.
2. Suggest possible harmonization steps.
Requirement5. The standard is harmonized with other sources of information (literature, web
resources) concerning the same subject area.
How to Test
1. Review the most known books and web resources on the subject.
It’s important to take into account the sources of information from Agile and OpenSource
communities.
2. Suggest possible harmonization steps.
Viewpoint: Presentation
Requirement1. The title of the standard reflects its scope and purpose.
If the title is not adequate suggest more suitable one.
Requirement2. The standard doesn’t contain ambiguities and self-contradictions.
Suggest replacement for each ambiguity or self-contradiction found.
Level 3: Effectiveness
Viewpoint: Contents
Requirement1. The ideas expressed in the standard are easy to apply in practice.
Any suggestions on how to simplify the methods proposed in the standard should be stated.
Requirement2.Using the standard makes positive effect for all target audiences.
How to Test
1. Determine the target audience(s) of the standard.
2. For each audience make practical experiments on using the standards.
It’s advisable to get measurable results (some statistics).
Requirement3. The effect of using the standard is more valuable as compare to alternative
solutions.
How to Test
1. The questions should be answered: is it possible to behave slightly different from what is
suggested in the standard? is it possible to behave inversely? What would be the effect?
It’s advisable to make practical experiments and get some statistics.
Requirement4.The artifacts appeared as a result of using the standard are useful; they satisfy
corresponding needs.
It’s advisable to get measurable results (some statistics).
Viewpoint: Presentation
Requirement1. The structure and design of the standard takes into account needs of all
categories of the standard’s readers.
Don’t confuse “category of readers” with one of users (target audience). “Users” is a
more broad term embracing not only direct readers but also people who gain the knowledge
of standard from readers.
How to Test
1. List user categories and corresponding information needs.
2. Estimate the degree of satisfaction of each need by means of the proper structure and
design.
3. The following question should be answered: is it possible to present the same ideas in
more convenient form?
Requirement2. The standard contains necessary “how to use” hints.
How to Test
1. Ensure that the following questions are answered in the standard: who (target audience),
when, where, under what circumstances and how will use it.
2. Find out is it possible to use the standard more widely.
Requirement3. The relation of the standard to IEEE/EIA 12207.0 is indicated.
The best solution is mapping table containing references to the corresponding clauses of
12207.
Requirement4. The criteria of correspondence to the standard are explicitly stated.
Requirement5. The standard is detailed enough, easy to overview and easy to understand.
How to Test
1. The questions should be answered: is it possible to say the same more briefly? is it
possible to remove some text without negative impact on perception of the ideas?
Requirement6. Optionality of each clause of the standard is indicated.
It’s better to have some “optionality scale” rather than binary marks like
required/optional.
Requirement7. The standard contains feedback form within it.
Our suggestion is to provide readers with some incentives for doing comments, like “if
the feedback is valuable enough its author gets a permission to join the balloting pool even
if he/she is not a member of the IEEE-SA, and his/her name will appear in the list of
balloting pool members in the next version of the standard”.
Requirement8. The major statements (ideas) are marked out explicitly.
How to Test
1. Create a list of the major ideas.
Requirement9. The standard contains brief description of all clauses.
It’s also advisable to indicate which clauses are the most important.
Requirement10. Meaning of all keywords is clear.
How to Test
1. Ensure all necessary keywords are displayed at the Title page.
2. Ensure all of them are present in the Definitions section.
Level 4: Attractiveness
Viewpoint: Contents
Requirement1. The standard is a unique source of specific information.
2 different cases are possible: either the original ideas are expressed in the standard, or
it summarizes the ideas expressed in all the accessible sources of information on given
subject. In the latter case it’s advisable to reference all the sources of information on given
subject.
Viewpoint: Presentation
Requirement1. The standard contains the List of Addressed Problems.
Requirement2.The standard contains sufficient number of examples.
Requirement3.The form of presentation of information is convenient enough.
How to Test
1. Consider different forms of presentation for different audiences: brief description, table,
picture, diagram etc.
Requirement4.The standard contains information on where to get (buy) it.
Annex 1. Popularity of Standards
If we consider wide application and popularity of the standards to be the ultimate goal it would
be interesting to estimate the degree of popularity of the existing set of standards. We attempted
to do that using number of references on the standard’s name in Google as indicator.
Here are the raw data sorted by number of references in Google. In order to make some
conclusions as regards the key parameters for standard’s popularity the data needs to be further
analyzed taking into account standard types, subject areas, and number of years in use.
Standard
IEEE 1016
IEEE 830
IEEE 829
IEEE 1028
IEEE 1058
IEEE 1471
IEEE 1220
# of
references
10,900
10,700
8,190
7,130
5,800
5,300
5,220
IEEE 828
4,220
IEEE 1362
IEEE 1233
IEEE 1008
IEEE 1490
IEEE
610.12
IEEE 1465
IEEE 1219
IEEE 1062
IEEE 730
IEEE 1063
IEEE 1012
3,770
3,730
2,930
2,920
2,610
IEEE 1074
IEEE 1540
IEEE 1228
IEEE 1061
IEEE 1045
1,530
923
897
814
788
IEEE 1462
IEEE
12207.0
IEEE 1044
IEEE
12207.1
IEEE 1517
669
553
Software Quality requirements
Software Maintenance
Software Acquisition
Software Quality Assurance
Software User Documentation
Software Verification and
Validation
Software Life Cycle Processes
Software Risk Management
Software Safety
Software Quality Metrics
Software Development
Productivity
Selection of CASE tools
Software Life Cycle Processes
531
438
Software Anomalies
Software life cycle processes
409,000
1,300,000
11
7
389
Software Reuse Processes
472,000
5
2,610
2,500
2,280
2,220
2,060
1,940
Subject Area
Software Design
Software Requirements
Software Testing
Software Review
Software Project Management
Software Architecture
Software System Engineering
Process
Software Configuration
Management
Software System Definition
Software System Requirements
Software Unit Testing
Software Project Management
Software Engineering
# of Subject
Area references
18,700,000
13,400,000
10,600,000
16,800,000
10,700,000
9,190,000
4,570,000
# of years
in use
21
20
21
15
29
8
10
5,520,000
21
10,100,000
10,200,000
3,370,000
9,050,000
10,200,000
9
8
17
6
14
7,020,000
8,880,000
5,610,000
2,980,000
7,060,000
548,000
6
12
11
25
17
18
1,220,000
6,780,000
7,510,000
508,000
3,290,000
13
3
10
12
12
3,690,000
1,300,000
9
8
Standard
# of
references
IEEE 982.1 364
IEEE
356
1420.1
IEEE
355
12207.2
IEEE
257
1420.1b
IEEE 1012a 114
IEEE
1320.1
IEEE
1420.1a
IEEE
14143.1
IEEE
1320.2
IEEE
1175.1
IEEE 2001
# of Subject
Area references
6,750,000
52,600
# of years
in use
16
9
1,300,000
7
Reuse Software Library
Interoperability
Software Verification and
Validation
IDEF0
52,600
5
548,000
18
18,100
11
52,600
8
74
Reuse Software Library
Interoperability
Software Measurement
5,390,000
6
63
IDEF1X
6,900
10
46
CASE Tool Interconnections
55,300
12
?
Web Page Engineering
7,030,000
5
106
81
Subject Area
Reliable Software
Reuse Software Library
Interoperability
Software life cycle processes