Rick Barneveld
Introduction
The title of the paper being discussed is: On the Perception of Software Quality Requirements during
the Project Lifecycle. The goal of this paper is to find the answer to two questions, namely:
•
•
Is there an increase of discussion about quality requirements as the project evolves?
Whether different software projects have similar levels of interest about quality
requirements.
The paper is based on a Ph. D dissertation by one of the authors (N. Ernst, 2012). From this
dissertation several conference proceedings are distilled.
Authors
The authors of the paper are Neil A Ernst and John Mylopoulos. Both the authors have an academic
background. They have written several publications together, and this is not their first one. Ernst
currently is doing post-doc research at the University of British Columbia in software engineering.
However, the cooperation between the authors began at the University of Toronto, where Ernst did
his Ph. D study, which was about Software Evolution. Mylopoulos is currently still involved at the
University of Toronto, where he is leading research projects. Mylopoulos actually supervised Ernst
during his Ph.D. research.
Method
The research is based on the use of quality attributes as defined in ISO/IEC 9126, so called F-R-U-E-MP (International Standards Organization, 2001). These stand for: Functionality, Reliability, Usability,
Efficiency, Maintainability and Portability. The paper uses eight different parts from an open source
software project for its research, namely Gnome. Gnome is extensively discussed by German (2003).
The steps undertaken to determine how often a certain quality attribute is discussed by developers
and users of a product during the project lifecycles is described in the beginning of the paper. First, a
number of sources where the discussions should be captured from were determined. These were the
developers email conversation of a project, the bug list of the project, etc. All the messages were
handled separately (so email threads were divided into distinct messages). Then, a catalogue of
words associated with each of the six quality attributes was made, to be used in an analysis of all the
messages. Two catalogues were made, “WN” and “ext”. One based on Wordnet, and another list that
extended the first one. WordNet is an online lexical database designed for use under program control
(Miller, 1995). It contains synonyms, meronyms and word relations. The “ext list” extended the “WN
list” with several terms specific to software engineering. After the lists were constructed, the
messages were analyzed with the word catalogues to see how often a certain quality attribute was
mentioned over the project lifetime. The last step of the process was to check how precise the
analysis was, and eliminate errors.
Example
Say a company wants to know if the perspectives of several development teams that work on a
project differ in terms of quality attributes. To do this, the company can use the method described in
the introduction. They can use either an external consultant, or use one of their own employees to
do the research. In this case, we will assume consultant is hired to do the job. The consultant first
needs to gather data to do the research on. He decides to use the commit logs of the project, as well
as the email conversation (from all teams) about the development. The next step is to make a list of
signifiers for the quality attributes. For this, he uses a Wordnet, to discover synonyms and
meronyms. In addition, he interviews some of the developers of every team, to see what terms and
words they associate with the quality attributes. After this, the third step can begin, which is to
analyze the gathered data with help of the list of signifiers. The next step is to figure out how the
false positive ratio is. When this is done, a comparison between the teams can be made. The
consultant can present the results to the board.
Deliverable: List of Signifiers
One of the deliverables of the case described above is a well-defined set of signifiers. This is one of
the most important deliverables, because when the list is incomplete, not every occurrence of a
quality attribute gets noticed, which impairs the research. Of course, the data which contains
occurrences of the quality attributes is also very important. A good starting point for the list of
signifiers is to choose standardized quality attributes, like the before mentioned ISO/IEC standard FR-U-E-M-P. The next thing to do is to get synonyms for all of these quality attributes. The researcher
should also try to find meronyms. These are words or concepts that are a part of some bigger
concept (i.e. room is a meronym for house). The last set of words that needs to be added to the list is
the set of words that are related to the quality attributes (e.g. speed for Efficiency). These relations
can be found via a website like Wordnet, but also by doing interviews with the developers of the
product that is the subject of the research. This last way will probably result in a better list of
signifiers, because now the signifiers that the developers associate with the quality attributes are
added. In the section Appendix – Template Deliverable: List of Signifiers, a template is shown of the
List of Signifiers.
Related literature
The paper is more or less an extension of another paper written by the authors (N. a. Ernst &
Mylopoulos, 2007), where they describe the past and the future of requirements for designing
software. To get a better view on how quality requirements may change during the project lifecycle,
another research was conducted. The method in the paper discussed in this description borders on
the field of requirements engineering, as is a method that investigates how the requirements of a
project may change over time. Requirements engineering is an important field in software
development, as there needs to be a good set of requirements in order to develop a good software
product (Sommerville & Sawyer, 1997).The results of the research however, are not conclusive.
Therefore they cannot be used in mainstream Requirement Engineering practice. That this is a
common problem is described by Kaindl et al. (2002).
Other methods to gather data about non-functional requirements are described by (Cleland-Huang,
Settimi, Zou, & Solc, 2006). However, this method is more an automated algorithm applied to
sources of data (like unspecified requirements documents and unstructured notes, memos and
meeting minutes). A more in-depth explanation about non-functional requirements is given by
Chung, Cesar, & Leite (2009). As requirements in an open source project like Gnome are not
generated by small team, but by all people that are collaborating in the project, they might be hard
to determine (Scacchi, 2001).
Process Deliverable Diagram
To create a clear overview of the method this section shows a Process Deliverable Diagram (PDD). A
PDD is in fact a diagram that consists of two diagrams, one that focuses on the activities used in a
method, and one that focuses on the deliverables that are created in the method(Van de Weerd &
Brinkkemper, 2008). The PDD merges these two diagrams and shows the activities on the left side,
and the deliverables on the right side. The section Process Deliverable Diagram shows the actual
PDD, while in the next two sections the Activities and Concepts are elaborated and explained. All the
processes in the PDD are executed by the same actor. This actor can be a single researcher, or it can
be a team of researchers. In this last case, the processes are executed by several people at the same
time.
Process Deliverable Diagram
0..*
MAIL LIST
0..*
1
CORPUS
BUG REPORT
1
Consists of
Determine Corpora
Select data sources
0..*
COMMIT LOG
1..*
Structure data sources
SINGLE MESSAGE
1..*
Create List of Signifiers
Select Quality Attributes
LIST OF QUALITY
ATTRIBUTES
Find related words and
concepts
CATALOGUE OF SIGNIFIERS
1
1
1
TEST DATA
1
Results in
Analyze data
NUMBER OF OCCURENCES
1..*
Evaluate results
Calculate number of false
positives/negatives
FALSE POSITIVE/NEGATIVE
RATIO
1
Calculate recall
RECALL VALUE
1
Gather results
1
RESULT
Figure 1 - PDD
1..*
Activity Table
Table 1 shows the Activity table that belongs to the Process Deliverable Diagram shown in Figure 1.
An activity table elaborates the activities that are in the PDD. All activities and its sub-activities are
explained and connected to concepts. Those concepts are explained in the next section: Concept
Table.
Activity
Sub-activity
Description
Determine Corpora
Select data sources
Select from all available data sources
of a project, the ones that contain
useful information, like COMMIT
LOGS, BUG REPORTS and the
EMAIL LIST.
Separate all data sources into
SINGLE MESSAGE’s
Define a LIST OF QUALITY
ATTRIBUTES to be researched. An
example can be the ISO/IEC 9126
standard F-R-U-E-M-P(International
Standards Organization, 2001)
Make
a
CATALOGUE
OF
SIGNIFIERS that are somehow
related to the quality attributes
(synonyms, meronyms, etc.).
Check for signifier occurrences. This
means see how many times the
signifiers occur in the data sources
Check how many times a signifier
was falsely linked to a quality
attribute.
Divide the number of correctly
classified occurrences by the total
number of occurrences.
Put together all the data and finalize
the research.
Structure data sources
Create list of Signifiers
Select Quality Attributes
Find related words and concepts
Analyze data
N/A
Evaluate results
Calculate number of false
positives/negatives
Calculate recall
Gather results
Table 1 – Activity table
Concept Table
Table 2 shows the Concept table that belongs to the PDD in Figure 1.
Concept
CORPUS
MAIL LIST
BUG REPORT
COMMIT LOG
SINGLE MESSAGE
LIST OF QUALITY
ATTRIBUTES
CATALOGUE OF
SIGNIFIERS
TEST DATA
NUMBER OF
OCCURRENCES
FALSE
POSITIVE/NEGATIVE
RATIO
RECALL VALUE
RESULT
Table 2 – Concept table
Description
Body of data about the project that is the subject of research.
Examples are MAIL LISTS, BUG REPORTS and COMMIT LOGS
(A Ernst & Mylopoulos, 2010)
Email list that the developers used to communicate with each other
about the project. Messages might include sentences like: “I am going
to work on the UI, to make it easier to use”.(A Ernst & Mylopoulos,
2010)
The list of bugs reported by user and testers of the project. This might
show sentences like: “The program was slow.”(A Ernst &
Mylopoulos, 2010)
The messages that developers submitted with the newly written
source code. Sentences that are in these logs might be: “I drastically
improved the performance speed of component X”.(A Ernst &
Mylopoulos, 2010)
The messages of each of the corpora split up. This is to make sure
reply emails do not result in the same mentioning of the quality
attribute occurs multiple times.(A Ernst & Mylopoulos, 2010)
The Quality Attributes that are being investigated. These usually are
based on an ISO/IEC standard. (International Standards Organization,
2001)
The list of signifiers that are used to see how many times a certain
quality attribute is mentioned in one of the SINGLE MESSAGES of
the CORPORA(A Ernst & Mylopoulos, 2010)
All the data that is needed to do the research, i.e. the LIST OF
QUALITY ATTRIBUTES, CATALOGUE OF SIGNIFIERS and the
SINGLE MESSAGES (A Ernst & Mylopoulos, 2010)
The number of times a certain quality attribute is mentioned in one of
the SINGLE MESSAGE. This number is calculated for each of the
quality attributes that are being researched.(A Ernst & Mylopoulos,
2010)
The number falsely classified occurrences. For instance, if a person’s
job is “usability engineer” and puts that in his email signature, it
should not be classified as an occurrence of the usability attribute.(A
Ernst & Mylopoulos, 2010)
The number of correctly classified occurrences by the total number of
occurrences.(A Ernst & Mylopoulos, 2010)
The results of the study, which shows the degree of change in how
many times a quality attribute is mentioned during the projects
lifecycle.(A Ernst & Mylopoulos, 2010)
References
A Ernst, N., & Mylopoulos, J. (2010). On the Perception of Software Quality Requirements during the
Project Lifecycle. Requirements Engineering Foundation for Software Quality 16th International
Working Conference REFSQ 2010 30 June2 July 2010, 143–157. Retrieved from
http://www.mendeley.com/research/perception-software-quality-requirements-during-projectlifecycle/
Chung, L., & Leite, J. do P. (2009). On non-functional requirements in software engineering. Conceptual
modeling: Foundations and …, 363–379. Retrieved from
http://link.springer.com/chapter/10.1007/978-3-642-02463-4_19
Cleland-Huang, J., Settimi, R., Zou, X. Z. X., & Solc, P. (2006). The Detection and Classification of NonFunctional Requirements with Application to Early Aspects. 14th IEEE International Requirements
Engineering Conference (RE’06) (pp. 39–48). Ieee. doi:10.1109/RE.2006.65
Ernst, N. (2012). Software Evolution: a Requirements Engineering Approach. Retrieved from
https://tspace.library.utoronto.ca/handle/1807/32707
Ernst, N. a., & Mylopoulos, J. (2007). Tracing software evolution history with design goals. Third
International IEEE Workshop on Software Evolvability 2007, 36–41. doi:10.1109/SE.2007.10
German, D. M. (2003). The GNOME project: a case study of open source, global software development.
Software Process: Improvement and Practice, 8(4), 201–215. doi:10.1002/spip.189
International Standards Organization. (2001). ISO/IEC 9126-1:2001 - Software engineering -- Product
quality -- Part 1: Quality model. Retrieved from
http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=22749
Kaindl, H., Brinkkemper, S., Bubenko Jr, J. a., Farbey, B., Greenspan, S. J., Heitmeyer*, C. L., Leite†, J. C. S. D.
P., et al. (2002). Requirements Engineering and Technology Transfer: Obstacles, Incentives and
Improvement Agenda. Requirements Engineering, 7(3), 113–123. doi:10.1007/s007660200008
Miller, G. a. (1995). WordNet: a lexical database for English. Communications of the ACM, 38(11), 39–41.
doi:10.1145/219717.219748
Scacchi, W. (2001). Understanding Requirements for Developing Open Source Software Systems, (July).
Sommerville, I., & Sawyer, P. (1997). Requirements engineering: a good practice guide. John Wiley & Sons,
Inc.
Van de Weerd, I., & Brinkkemper, S. (2008). Handbook of Research on Modern Systems Analysis and Design
Technologies and Applications. (M. R. Syed & S. N. Syed, Eds.). IGI Global. doi:10.4018/978-1-59904887-1
Appendix – Template Deliverable: List of Signifiers
List of Quality Attributes
Here should be a list of (preferably) standardized Quality Attributes of which the occurrence in the
data will be checked. An example might be the ISO-IEC standard F-R-U-E-M-P (Functionality,
Reliability, Usability, Efficiency, Maintainability and Portability).




Quality Attribute 1
Quality Attribute 2
Quality Attribute 3
Etc.
List of Signifiers
A signifier is a word that is connected to one of the quality attributes mentioned before. It might be a
synonym, a meronym or any other way two words can be related. For instance: “slow” might be a
signifier for “Performance”, while it is not a synonym or meronym. Eventually you will get a list of
words related to each of the quality attributes. Each list should be separate and there is no need for
each list to be of equal length.
Quality Attribute 1




Related word 1
Related word 2
Related word 3
Etc.
Quality Attribute 2





Related word 1
Related word 2
Related word 3
Related word 4
Etc.
Quality Attribute 3



Related word 1
Related word 2
Etc.
Quality Attribute 4




Related word 1
Related word 2
Related word 3
Etc.