6 January 2014
Responses to SYS-13-005.R1 Referee Review Comments
Note: The title of the paper has been changed to “The Requirements Entropy Framework in
Systems Engineering”.
2nd Reviewer Comment
Author Response
1. p.3 line 39 missing end quote
The end quote has been added.
2. check consistent use of def of “reqs
Section 1.2 has been rewritten to clarify the difference
volatility” as stated p.3 line 55ff
between requirements changes and requirements
volatility as defined in this paper.
Other revisions
have been made to the text of the paper to ensure
consistency and avoid misunderstanding of the
difference
between
requirements
changes
and
requirements volatility (e.g. Section 6 title, discussion
of figure 8, general discussion of the requirements
trend figures in Section 6). Part of the confusion may
have arisen because of references and data discussed
previously that refer to requirements changes as
requirements
volatility.
Requirements
volatility
requires change, however not all requirements changes
result in requirements volatility. One of the objectives
of this paper is to emphasize the difference between
“good” requirements changes that move the system
development in the right direction and “detrimental”
requirements changes that result in volatility and have
the opposite effect.
As discussed previously, this
paper
considers
the
REF
special
case
where
requirements changes either increase requirements
quality or have a neutral effect. Another paper on the
REF general case that accounts for the possibility of
requirements volatility effects is being written.
3. p.13 line 25 incorrect pronoun
The pronoun has been changed to “them”.
antecedent: “it” vs. “them”. Quality
attributes are bound to the
“requirements” (pl.) rather than to the
“end state” (sing.)
4. p.13 lines 32-46 The concept of
This part of the paper has been revised to more clearly
stability applied to q14 is correct, but
communicate the reason maximum stability only
the reason for it is incorrect: “…is the
applies to the q14 level and the reason P = 1 only
least random state, because there is
applies
only one possibility for requirements
relationship between rn, N and P is important to
occupying it.” This reason is equally
understand in the context of the process where the
valid for state q0. The number of
input of engineering effort E increases Q relative to
available possibilities for each state
the desired end state Qmax, which occurs when R
q0..q14 is a simple binomial function
occupy q14. For requirements occupying q14, N = 1
based on selecting some “q” number of
because there is only one quality level available for
quality attributes out of 14. Suggest
those requirements to occupy when E is input. For the
minor wording change.
special case discussed in this paper that does not
to
requirements
occupying
q14.
The
include requirements volatility or negative “ripple
effects” as a result of a revision to an existing
requirement, the requirements occupying q14 will
remain there for the duration of the requirements
engineering effort. This is intuitive because validated
requirements that have maximum quality do not
require any more work in the process. Therefore by
equation (4) and equation (5), P = 1. This is not the
case for requirements occupying the other quality
levels in the REF even though there is only one state
per quality level. This point is made by the example
in the paper following this discussion where R = 4 and
the requirements occupy q0 and therefore P = 3060,
which reflects more than one possible arrangement for
the requirements among the 15 available quality levels
when E is input into the open process. The reviewer is
correct that the REF in this paper has only one state
per level for requirements to occupy, however this
does not mean that there is only one possible
arrangement available for the requirements during the
process as E is input and requirements increase in
quality
according
to
the
specified
probability
distributions. The desired end state q14 is the least
random, and therefore, the least uncertain state
because q14 is the only state in the REF with one
possibility for requirements to occupy (P = 1). The
problem of interest discussed in this paper is an
estimate of the additional engineering effort ΔE that
must be input into the open process to transition R
from their current state to the desired end state q14, not
the engineering effort that has already been input into
the process which is reflected by the current
requirements quality distribution and Q. This may be
of interest and can be estimated or measured using the
REF but this is not the focus of this paper. This is a
topic of another paper that would require detailed
retrospective analysis of real SE program data for
comparison with the model. The initial distribution of
requirements among the quality levels, or states, is
determined by the probability mass function of the
binomial random variable given by the probability of a
requirement having a quality attribute which is treated
as a success.
More detail on the probability
distribution assumptions is provided in the response to
question 7.
5. p.14 line 57. Reference to “figure 4”
The sentence has been revised to reference the REF in
appears to be no longer correct. Figure
figure 2, not figure 4. A new figure has been added to
4 is now a context diagram for the REF
the paper to address comment number 10. The figure
process. If this reference is intended to
numbers in this draft have been revised accordingly.
point to that figure, some explanation is
Additivity is an important property of entropy and is
necessary. Is this entire paragraph on
shown by the summation in equation (6).
additivity necessary?
paragraph also includes an example to help the reader
The
understand the calculation. Therefore, it is worthwhile
to include the point and the example in the paper.
6. p.15 lines 18-25. Perhaps a short
Text has been added to the paper on the likelihood that
mention is necessary that using this
<b> will vary among SE teams and organizations.
equation in the real world would
require calibration of the factor.
7. p.16 line 10. “…generated”? or The initial state requirements quality distributions
“…used”? Where these distributions were generated in MATLAB using the probability
inputs to the MATLAB, or results from mass function of a binomial random variable
it? If results, then what assumptions accounting for higher probability of a quality factor
and starting points generated them?
being bound to a reused requirement versus a new
requirement. This assumes reused requirements are
appropriately selected based on their merits. A brief
discussion in Section 5 was added to the paper on the
binomial distribution and the probability of a
requirement having a quality attribute ‘bound’ to it at
the start of the effort. Table II has been added to the
paper and includes information on the assumptions
and starting points for the MATLAB simulations. The
initial state distributions were generated in the same
manner as the initial state distributions for the
requirements trend analysis in Section 6 and
engineering effort analysis in Section 7. Sections 6
and
7
include
additional
information
on
the
probabilities that were used to generate the initial state
requirements
discussion
quality
of
the
distributions.
underlying
A
theory
detailed
and
all
assumptions, rationale and MATLAB implementation
considerations is beyond the scope of this paper. Such
a discussion would be suitable for a separate paper
focused specifically on the MATLAB software
implementation of the REF. Additional information
has been added to the paper to inform the reader that
for smaller programs (e.g. R < 400), the probability of
requirements reuse (preuse) was assumed to be a
maximum of 0.4, and the probability of reuse for
larger programs was treated as a uniform random
variable.
The probability of a reused requirement
having a quality attribute was assumed to be between
0.90-0.96 and the probability of a new requirement
having a quality attribute was significantly lower (0.6
maximum), which resulted in a probability << 1% that
a new requirement would have all fourteen quality
attributes at the start of the effort.
8. p.16 line 25. It appears that the
While it is true that the desired end state is reached
bottom pane of fig.3 is the desired end
when all requirements occupy q14, the bottom pane of
state for any initial distribution, not just
figure 3 is correct as shown because it is the actual
for the middle distribution.
desired end state for the MATLAB simulation run
corresponding to the initial requirements quality
distribution shown in the middle plot.
Note the
requirements quality levels are plotted against the
number of requirements, which vary from run to run.
9. p.17 line 41. A more explanatory
Table II has been added to the paper to summarize
introduction to section 6 would be
many
useful. Add a table to describe the
implementation of the REF. Additional text has also
assumed characteristics of the
been added to the paper in Sections 6 and 8 to explain
MATLAB runs. Use some text to
why the results from the Fractional Method and REF
explain the consequences of the
are different, even for the special case where changes
difference between REF and the
to existing requirements are either good or neutral.
fractional method. (e.g. fractional
Under such conditions, some might expect a high
method can predict the time at which
degree of correlation between the REF and Fractional
key
assumptions
for
the
MATLAB
requirements volatility approaches zero, Method but this is not the case. The hypothesis for the
so therefore predicting the completion
general case is that the difference between the
date of requirements effort. But REF
Fractional Method and REF is even more pronounced
also includes the quality of those
given the possibility that a single change can result in
requirements, thereby predicting a
significant negative “ripple effects” for existing
different completion date at which …)
requirements.
This reviewer has been through your
paper.
This will be discussed in a separate
paper three times to gain this
understanding; most readers will not
have that luxury.
10. p.17-18. The textual descriptions of
The textual descriptions of the MATLAB simulations
the MATLAB simulation runs do not
runs are correct for the requirements trend figures
always match the trending shown in
shown in the paper. Another figure has been added to
figures 5-10. Perhaps the problem is in
the paper in Section 5, which is now the new figure 4,
the figures, which often do not have
to show the theoretical nonlinear behavior of HR vs P
sufficient x-axis to show the
for R = 1000. Data from simulation runs 4 and 10 are
intersection point of REF with lowest
also included in figure 4 to show the nonlinear HR
entropy. Examples: Fig 6 says the REF
decrease as requirements quality increases and P
end interval =22, but a visual
decreases over time during the engineering effort.
extrapolation of the line goes far
This behavior is not immediately obvious without the
beyond 22. Likewise, fig.8 says the end
theoretical HR plot. A separate paper with more detail
interval= 18, but it is nowhere near
on the theory of information quality and theoretical HR
endpoint at interval 14.
behavior entitled “A Theory of Information Quality
and its Implementation in Systems Engineering”
[Grenn et al, 2013] has been recently published by the
IEEE Systems Journal and is now available on IEEE
Xplore if the reviewer would like more detail on the
theory. This reference has been added to the paper for
the reader’s benefit.
11. p.20 lines 22-34. This statistical
The premise that the REF and Fractional Method are
treatment of the difference between the
different, or should be different, is supported by the
REF and FM end points appears
statistical analysis of the results from the MATLAB
meaningless to this reviewer based on
simulation experiments. The reviewer is correct that
the theory. It is not your point or thesis
the REF and Fractional Method are different measures
that REF and FM are similar, but rather
of requirements trends, and the difference is
that they are dissimilar ways to treat the significant even for the special case that limits the
same measurement. Suggest removing
requirements change effects to good or neutral such
this paragraph, in favor of a new
that the quality of existing requirements either
paragraph at the end of section 6 that
increases or remains the same relative to the desired
actively contrasts the differences and
end state. If the two methods were not significantly
similarities in the two methods, seeking
different, then there would be little reason to consider
to explain why the simulation results
the REF despite the fact that it accounts for
are what they are. (e.g. REF measures
requirements
reqs quality, while FM only measures
encourage publication of results that show agreement
change counts)
between the Fractional Method and real program
quality.
Perhaps
this
paper
requirements trend data if such data exist.
will
The
difference between the two methods may not be
readily apparent to all readers, and some readers may
expect a high degree of correlation between the two
methods for the special case.
I expected a higher
correlation for the special case. The statistical analysis
of the simulation results clearly show that even for the
special case, there is a significant difference between
the REF and Fractional Method. More requirements
trend and engineering effort data needs to be published
in the SE literature to enable further insight into the
underlying mechanisms contributing to observed
macroscopic behavior. A few sentences have been
added at the end of Section 6 to further explain the
simulation results as recommended by the reviewer.