ICT Standards and Guidelines
Segment 201
Quality Management
Main Document
(Version 2.0)
Disclaimer
The Office of the Minister of State for Administrative Reform (OMSAR) provides the
contents of the ICT Standards and Guidelines documents, including any component or
part thereof, submission, segment, form or specification, on an 'as-is' basis without
additional representation or warranty, either expressed or implied. OMSAR does not
accept responsibility and will not be liable for any use or misuse, decision, modification,
conduct, procurement or any other activity of any nature whatsoever undertaken by any
individual, party or entity in relation to or in reliance upon the ICT Standards and
Guidelines or any part thereof. Use of or reliance upon the ICT Standards and Guidelines
is, will be and will remain the responsibility of the using or relying individual, party or
entity.
The ICT Standards and Guidelines are works in progress and are constantly being
updated. The documentation should be revisited regularly to have access to the most
recent versions.
The last date of update for this document was June 2003.
Table of Contents - Quality Management
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
Executive Summary for Quality Management ........................................... 1
The Background of Quality Management Standardization ........................ 2
2.1
The Scope of Quality Management ...................................................... 2
2.2
The Objectives and Benefits of Quality Management .............................. 2
2.3
The Benefits of Standardization ........................................................... 2
2.4
Policies to Follow for Quality Management ............................................ 3
2.5
Risks Resulting from the Standardization Activities ................................ 3
2.6
Related Documents ........................................................................... 4
2.7
How to Use This Document ................................................................. 4
2.8
Related Terms and Acronyms ............................................................. 5
2.9
Related Segments and Cross References .............................................. 5
2.10 Related International Standards .......................................................... 5
2.11 All Segments in the ICT Standards and Guidelines ................................. 6
Introducing Quality Management ............................................................. 7
3.1
Quality Terms and Definitions ............................................................. 7
3.1.1 What is Quality? ...................................................................... 7
3.2
The Difference Between Grade and Quality ........................................... 8
3.3
The Issue of Implied and Stated Requirements ..................................... 9
Roles and Responsibilities in Quality Management ................................. 10
Quality Planning ....................................................................................... 1
5.1
The Quality Planning Template ............................................................ 1
5.2
Pre-Requisites of the Quality Plan ........................................................ 1
5.3
Risks in the Preparation of the Quality Management Plan ....................... 1
5.4
SOP 1: Preparing the Quality Plan ....................................................... 2
5.5
Good Practices: Defining Acceptance Criteria for SOPs ........................... 3
5.6
Good Practices: Project Documentation ................................................ 3
5.7
Good Practices: The Communications Plan............................................ 4
5.8
Good Practices: Use Performance Indicators ......................................... 5
Quality Assurance .................................................................................... 6
6.1
SOP 2: Quality Assurance Audits and Reviews....................................... 6
6.2
SOP 3: Project Management: Planning, Tracking, and Oversight ............. 8
6.3
SOP 4: Configuration Management .................................................... 12
Quality Control of Software Development .............................................. 13
7.1
What About Commercial Off the Shelf Software Applications? ............... 13
7.2
SOP 5: Qualification of Requirements................................................. 13
7.3
SOP 6: Qualification of Functional Design ........................................... 13
7.4
SOP 7: Qualification of Technical Design ............................................ 14
7.5
SOP 8: Software Application Testing .................................................. 14
Quality Control of the Deployment of Products and Systems .................. 15
8.1
SOP 9: Qualification of Technical Specifications ................................... 15
8.2
SOP 10: Qualification of Installation .................................................. 18
8.3
SOP 11: Qualification of Operation .................................................... 20
8.4
SOP 12: Qualification of Performance................................................. 22
Quality Control of ICT Operations .......................................................... 26
9.1
SOP 13: Maintenance Control ........................................................... 26
9.2
SOP 14: Data Entry Control .............................................................. 29
9.3
SOP 15: Training Control .................................................................. 31
Figures - Quality Management
Figure 1 : Variance Between Requirements and Deliveries ................................. 7
Figure 2 : The Quality Management Processes ................................................... 7
Figure 3 : Control the Project Triangle through Quality Management ................. 9
1.0
Executive Summary for Quality Management
The term “Quality” has become fashionable in the recent past. The use of the term has
consequently lost its meaning and is now wrongly associated with “excellence”, “high
grade” and other irrelevant associations.
The real meaning of the term affects ICT projects and operations in a major manner.
Quality is the ability of a product to meet stated requirements. For example, if a
software application delivers the functions exactly as stated in the Technical
Specifications, then it has high quality. Whether the functions are sophisticated or not is
not relevant in this case. What is important is that an ICT product or a service must have
its functions clearly stated and must have the requisite procedures to test whether these
functions have been delivered or not.
Quality Management consists of 3 major processes. These apply to ICT projects as well
to ongoing operations.
Quality Control: covers the testing or checking procedures needed to verify that a
specific product or service is performing or operating or has been delivered exactly as
stated in the functional or technical specifications. Quality Control is therefore presented
in this segment as a series of Standard Operating Procedures (SOPs).
Quality Assurance: is a meta-activity. It is the quality control of quality control. This
means that when the stakeholders wish to get “assured” that the Quality Control
procedures are valid, ie, can actually trap or identify variance from stated requirements,
Quality Assurance is brought in. It is therefore a process by which Quality Control
procedures are examined for their validity. Will the software test scripts actually
ascertain whether all functions have been implemented without error? Secondly, Quality
Assurance is a managerial tool designed at reviewing Quality Control activities to ensure
that they are actually taking place. Finally, Quality Assurance is resorted to while
planning Quality Management activities as well as when Quality Control procedures are
found to be inappropriate and would hence need improvement.
Quality Planning: covers the planning for all Quality Management activities that need
to take place in a project or an ongoing operation. A special template is provided for use
as the basis of such planning.
This segment covers a wide range of Quality Control Standard Operating Procedures
(SOPs), controlling such issues as specification, installation, operation, performance,
data entry, software testing, training and other procedures.
Quality Management
Page 1
2.0
The Background of Quality Management Standardization
This section presents key issues to be reviewed before reading the main sections for
Quality Management.
2.1
The Scope of Quality Management
In ICT environments, it has become common to think of Quality Management mostly
through Quality Management for Software projects. The Standard Operating Procedures
presented in this segment can also be applied to the following processes:







Hardware configurations: networks, minicomputer systems, etc.
Telecommunications systems
Networks (LANs and WANs)
Web based applications (eGovernment applications)
Office Technology applications (Workflow, groupware, internal exchange, etc)
Database Management Systems
Other ICT processes such as Training, Configuration Management and Disaster
Recovery
The Quality Management approach is universal and is based on the Total Quality
Management techniques.
2.2
The Objectives and Benefits of Quality Management
The benefits that Public Sector Agencies will reap from Quality Management are the
following:







2.3
Compliance with Quality aims
Improved deliveries due to earlier qualification of design and specifications
Improved controls over the testing procedures by standardizing testing scripts,
acceptance criteria, authorization levels and error reporting
Find critical defects earlier in the process and at a reduced cost to the
development
Adherence to the principles and good practices of documentation
Increased awareness of satisfaction of receivers of ICT process and project results
Applying the practical Quality Management processes will lead to an easier
implementation of wider scoped international Quality Standards
The Benefits of Standardization
The standardization process provides the following benefits:



Higher quality levels are achieved throughout the government
Higher quality levels are achieved through the interaction between various
Ministries and Agencies
When standardized through proper Quality Management techniques, practices
such as testing, delivery, installation, operation, performance, etc., will improve
throughout the government
Quality Management
Page 2


2.4
Quality Management practices will become easier to learn as practices are
standardized
Analysis of global performance indicators will provide a deeper understanding of
problems and trends.
Policies to Follow for Quality Management
The following Policies are proposed:





2.5
Quality Management must be part of all ICT processes and projects.
Quality Management must be applied on both internal and external processes.
Quality Assurance units must be setup to handle all such activities.
Follow up on all Quality Assurance audits and reviews must be performed.
Sufficient resources must be provided for such activities.
Risks Resulting from the Standardization Activities
The following risks may arise when the Ministry or Agency implements the Quality
Management into its projects and operations:







Misunderstanding the difference between Quality Assurance and Quality Control
(To be discussed in Section 3.1).
Standardized procedures might not be followed
Persons might be assigned to the job of Quality Assurance Unit who do not
possess adequate knowledge of Quality Management
Quality Management may be launched but is later abandoned during the life of
the project or operation
Not providing adequate time and resources for Quality Assurance
Lack of commitment of the Ministry or Agency’s management or that of the ICT
unit
Lack of coordination with other Agencies for learning, information sharing and
exchange of experience.
If the Ministry or Agency does not incorporate Quality Assurance into its projects and
operations, the following risks may arise:



Projects and operations will be executed but there will not be adequate checks
and controls on their ability to provide the required results.
Quality Management may yield gaps in the conformance, i.e., variance between
what is required and what is delivered without such gaps being attended to: no
corrective action, no improvement of processes.
Creating too many Quality checks and controls that result in a reduction of
efficiency.
By following the Guidelines and SOPs of this segment, many of the above pitfalls can be
avoided.
Quality Management
Page 3
2.6
Related Documents
This segment has three documents related to it:



The Quality Management project plan: A related document presents the
Quality Management Project Plan Template. This is a generic plan that is to be
used in its electronic form as the basis of all new Project Plans. The QU Unit will
make a copy of this template and fill it in with specific information and
instructions while planning the Quality Management Project.
Introduction to the CMM and Quality Management: Introduces the Model
and shows where within the model lies Quality Assurance.
Various Quality Management approaches: presents the inter-relation
between various Quality Management systems and standards.
These can be downloaded from OMSAR's website for ICT Standards and Guidelines at
www.omsar.gov.lb/ICTSG/QM.
2.7
How to Use This Document
The document needs to be reviewed the first time a Quality Management exercise is
taking place. Thereafter, the reader can go straight to the Quality Management planning
template and launch his or her Quality Management activities.
The following is a general list of all Standard Operating Procedures presented in this
segment. All are in found in the main Document. They are grouped as follows:
Quality Planning
SOP 1: Preparing the Quality Plan
Section 5.4
Quality Assurance
SOP 2: Quality Assurance Audits and Reviews
SOP 3: Project Management: Planning Tracking & Oversight
SOP 4: Configuration Management
Section 6.1
Section 6.2
Section 6.3
Quality Control of Software Development Process
SOP
SOP
SOP
SOP
5:
6:
7:
8:
Requirements Management Quality Control
Qualification of Functional Design
Qualification of Technical Design
Software Quality Control (Testing)
Section
Section
Section
Section
7.2
7.3
7.4
7.5
Section
Section
Section
Section
8.1
8.2
8.3
8.4
Deployment Processes
SOP
SOP
SOP
SOP
9: Qualification of Technical Specifications
10: Qualification of Installation
11: Qualification of Operation
12: Qualification of Performance
Operational Processes
SOP 13: Maintenance Control
Quality Management
Section 9.1
Page 4
SOP 14: Data Entry Quality Control
SOP 15: Training Quality Control
2.8
Section 9.2
Section 9.3
Related Terms and Acronyms
CMM®
CRF
IEC
IEEE
IQ
ISO
OQ
PMI
PQ
QM
QA
QC
SQAP
TQM
2.9
Capability Maturity Modeling
Change Request Form
International Electro-technical Commission
Institute for Electrical and Electronic Engineers
Installation Qualification
International Standards Organization
Operational Qualification
Project Management Institute
Performance Qualification
Quality Management
Quality Assurance
Quality Control
Software Quality Assurance Plan
Total Quality Management
Related Segments and Cross References
All segments relate to Quality Management. However, the following have a more direct
involvement with Quality Management:


2.10
Software Applications: Discussions related to software testing, software
development processes and other issues relate to QA. This segment can be
downloaded from OMSAR's website for ICT Standards and Guidelines at
www.omsar.gov.lb/ICTSG/SW.
Configuration Management: Change Control is a QA procedure and must be
reviewed in this document. This segment can be downloaded from OMSAR's
website for ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG/CM.
Related International Standards
The approach taken in this segment is to avoid using “parts” of ISO or other Quality
standards. Hazards have been faced before when organizations partially implement such
coherent standards.
The main approach taken is to follow the second Key Process Area of Level 2 of the
Capability Maturity Model of the Software Institute in the Carnegie Mellon University.
This is described in detail in a document called “Quality Management and the CMM”
which can be downloaded from OMSAR's website for ICT Standards and Guidelines at
www.omsar.gov.lb/ICTSG/QM.
Quality Management
Page 5
2.11
All Segments in the ICT Standards and Guidelines
OMSAR's website for ICT Standards and Guidelines is found at www.omsar.gov.lb/ICTSG
and it points to one page for each segment. The following pages will take you to the
home page for the three main project document and the 13 segments:
101
101
102
103
104
105
106
201
202
203
204
205
206
207
www.omsar.gov.lb/ICTSG/Global
www.omsar.gov.lb/ICTSG/Cover
www.omsar.gov.lb/ICTSG/Legal
www.omsar.gov.lb/ICTSG/HW
www.omsar.gov.lb/ICTSG/HW
www.omsar.gov.lb/ICTSG/NW
www.omsar.gov.lb/ICTSG/TC
www.omsar.gov.lb/ICTSG/DB
www.omsar.gov.lb/ICTSG/OS
www.omsar.gov.lb/ICTSG/EN
www.omsar.gov.lb/ICTSG/QM
www.omsar.gov.lb/ICTSG/SW
www.omsar.gov.lb/ICTSG/EV
www.omsar.gov.lb/ICTSG/SC
www.omsar.gov.lb/ICTSG/DE
www.omsar.gov.lb/ICTSG/RM
www.omsar.gov.lb/ICTSG/CM
Global Policy Document
Cover Document for 13 segment
Legal Recommendations Framework
Hardware
Hardware Systems
Networks
Telecommunications
Database Systems
Operating Systems
Buildings, Rooms and Environment
Quality Management
Software Applications
Evaluation + Selection Framework
Information Integrity and Security
Data Definition and Exchange
Risk Management
Configuration Management
Each page contains the main document and supplementary forms, templates and articles
for the specific subject.
Quality Management
Page 6
3.0
Introducing Quality Management
Due to the variety of usages of Quality Management terms, it is important to start this
segment with a clarification of each of the Quality Management terms and processes.
3.1
Quality Terms and Definitions
A major cause of dispute in ICT projects and operations is the three-way gap between
the actual requirements of the user, the promised and the delivered features of the
product. This “Variance” is the evil that needs to be removed. Achieving zero variance is
the aim of Quality Management.
Actual User
Requirements
Promised
Features
Delivered
Features
Figure 1 : Variance Between Requirements and Deliveries
3.1.1 What is Quality?
Quality Management refers to management functions that determine and implement the
systems affecting product quality. So what is Quality?
Quality = the characteristics of a system or a process that
affect its ability to satisfy "Stated" or "Implied"
requirements.
Quality ensures that what was promised was delivered. It ensures that there is a
conformance between requirement and delivery. It expects not to have a variance
between requirements and delivered products or services.
Quality Management is made up of 3 different activities:
Quality
Planning
Quality
Assurance
Quality
Control
Figure 2 : The Quality Management Processes
Quality Planning: Identify which processes are relevant to the project, determine how
to qualify them and document them in a formal Quality Management Plan.
Quality Management
Page 7
Quality Assurance: Consists of all activities necessary to assure all those concerned
with the product or service being produced that all quality control processes can actually
trap the variance and that such processes are taking place. Furthermore, Quality
Assurance gets involved in developing Quality Management plans, testing scripts and
improvement exercises.
Quality Control: Verifies all items, processes and systems correspond to predetermined
and stated requirements. This is a regulatory process through which actual results are
measured. They are then compared with requirements as stated during the Quality
Planning stage and used to upgrade the Quality Plan in order to achieve continuous
improvement.
An example:
1.
Quality Control is the procedure used to test a specific software module. It covers
such steps as setting up the environment, which test data to use, how to record
the test results and the final test results. The result of Quality Control is a pass or
fail situation for the particular software module.
2.
Quality Assurance covers the verification activities needed to complete the
following:



Check whether the testing procedures to be used by Quality Control will
actually trap all errors and prove that the requirements have been met.
Check whether the testing process is taking place with all its requirements
such as logging, approvals, proper personnel, etc.
Get involved in the improvement process. Improvement might be required
of the actual programming techniques, the testing process or even the
quality assurance process itself.
Warning: Up until ISO revised its definitions, Quality Assurance used to be the overall
process covering planning, control and improvement of Quality. It is the reason why
many projects still refer to the overall Quality Management process as “Quality
Assurance”. With the revised definitions, Quality Management now includes Quality
Assurance as a separate activity.
3.2
The Difference Between Grade and Quality
The common understanding of the term “Quality” confuses Grade with Quality. It has
become popular to define Quality as having a degree of excellence or superiority or
possession of many fine features or functions. This is not the managerial or
organizational meaning of Quality.
If the delivered product or service is short of what was promised, the result is a case of
Low Quality or a lack of compliance or conformance with requirements.
Grade is the level of promised features. The more and the better features are promised,
the higher the Grade.
An example: assume there are two software applications:
Quality Management
Page 8
1. A simple software application stores information about Vendors. It stores a
limited number of data elements. It has no bugs. The manual describes in exact
terms how the application works. It provides all reports defined in the brochure.
2. A high end product provides a very sophisticated database of Vendor profiles,
products, brands, experiences and team profiles. Some of the functions in the
brochure do not work. Others have bugs. The manual is not always clear and
does not describe all functions of the application.
The first product has low grade and high quality. The second product has high grade and
poor quality. One delivers what it promises even though the promises were not high. The
other does not.
3.3
The Issue of Implied and Stated Requirements
In the earlier definition, it was stated that Quality is the ability to meet “stated” or
“implied” requirements.
The stated requirements are easy to evaluate.
It is the implied needs that are problematic. These are the needs or the promises that
are “assumed” in the mind of the person requiring the product or the service. They are
expectations or requirements that the persons assume will be there and hence does not
see the need of stating them.
Usually, such “implied” needs would have been missed during the stage of Requirements
Analysis.
It is the task of any project or operations manager, whether from the Ministry or
Agency’s side or from that of the Vendor, to ensure that all implied needs be clarified
and brought out to become stated needs during the requirements stage of a project.
Quality Management
Page 9
4.0
Roles and Responsibilities in Quality Management
One person or a group of persons, often grouped under a Quality Assurance department,
shall be assigned to the project to perform the Quality Assurance.
For the sake of brevity, this segment shall refer to a Quality Assurance Officer or a QA
Officer to mean the whole QA Unit.
The QA officer is chosen from outside the project, business unit, or even from outside
the organization. The reason for such a separation is to seek objectivity and
independence. The QA Officer must have a reporting channel to top management
without going through the project manager.
The roles, responsibilities, and activities performed by the QA officer can be customized
according to the project. They need to be documented in the Quality Management Plan
of the project.
The following are the responsibilities of the QA Officer:










Assist in the preparation of the Quality Plan for each project
Assist the product managers and related stakeholders in the development of test
procedures or whatever is needed to complete the preparation of Quality Control
procedures.
Ensure that the stakeholders’ requirements from all products and services are
well documented and are conducive to being verified and audited against the final
products or services.
Review and audit the Quality Control procedures on a regular basis to ensure that
they will actually trap any variance between documented requirements and the
final products or services.
Conduct additional audits or QC reviews by demand of the project manager or
because of an event driven cause
Document all reviews and audits performed
Provide feedback to the project manager on a regular basis or on demand
Assist various stakeholders in the process of improving production processes, QC
processes or any other aspects of the ICT resources that need improvement.
Recommend updates to the Quality Plan to treat problems at hand
Keep a noncompliance log and ensure that recorded compliance issues are
resolved and corrected
The QA Officer will usually report to a post higher than that of the Project Manager but is
not to be regarded as an “opponent” of the Project Manager. The QA Officer is a team
member and should be considered as a friend of the project.
Quality Management
Page 10
5.0
Quality Planning
Quality Planning is the process by which all Quality Management activities are planned
and communicated to those concerned.
5.1
The Quality Planning Template
A separate template is provided to be used as the basis of future Quality Planning
exercises. It can be downloaded from OMSAR's website for ICT Standards and Guidelines
at www.omsar.gov.lb/ICTSG/QM.
5.2
Pre-Requisites of the Quality Plan
The following should be carried out before any planning takes place:
1.
The assignment of the Quality Officer: The Ministry or Agency must assign a
person responsible for the development and complete planning of the Quality
Management activities. This person is called the Quality Assurance or QA Officer.
The role of the Quality Officer was defined in Section 4.0.
2.
Establishing Project Management processes: Managing Quality on a project
is a critical task. Quality management should be considered as a sub-project,
having a plan, budget and resources, deliverables and outputs. The Quality
Officer should manage the Quality as a Project Manager manages the project.
3.
Assigning a Key Management person to Approve the Quality Plan: The
Quality Plan needs to be approved by a recognized key executive. This person
must be identified before the Quality Plan is commenced on and would be
indicated within the plan.
4.
Identify Applicable Norms and Standards: The Quality Plan must adhere with
or at least refer to relevant standards and regulations applicable to the nature of
the project and determining how to satisfy them. The contents of the quality plan
are specific to the nature of the project. For example a project that develops
custom software must address design issues and define the software
development life cycles used whereas purchasing a Common off the Shelf System
(COTS) does not. A project installing hardware in a nuclear plant must be aware
of the International Electro-technical Commission standards for safety and
dependability.
5.3
Risks in the Preparation of the Quality Management Plan
While preparing the plan, the following Risks must be avoided:



Preparing a plan that is too general
Preparing a plan that is too detailed and hence too rigid to be implemented with
success.
If the QA Plan and activities are not consistent with the effort and scope of the
project it is supporting, there is a risk of effort being spent on the wrong issues.
Quality Management
Page 1





Preparing a plan that is based on the generic plan and does not include specific
issues related to the project at hand
Not assigning a person responsible for the Quality Assurance
Not ensuring that management is aware of the plan and what it entails
Not budgeting for sufficient time and resources when planning to manage Quality
for the entire project or process life cycles
Preparing a plan and not following it, or updating it as changes to the project
occur
It remains the responsibility of the QA Unit to ensure that such events do not arise.
5.4
SOP 1: Preparing the Quality Plan
The QA activities begin when the initial project planning activities begin.
Objective: To prepare a plan which identifies how all the activities of Quality Assurance
and Control are to be executed in a project or in ongoing operations in the Ministry or
Agency.
Standard Operating Procedure:
The QM Project Plan template is found in Document 3 in this segment and is available on
the website of OMSAR. Use it to define the following:
1.
Project Scope or the Scope of the Operations: This step ensures that Quality
Management is to be carried out over a valid range of activities. It should not
ignore crucial activities and must not extend beyond the required scope.
2.
Methodology: The plan should refer to the Standard Operating Procedures to be
presented in this segment. Should there be any deviations from these SOPs, the
Methodology section should identify and justify such differences.
3.
Roles and Responsibilities: All activities in the plan must be associated with a
specific person or unit to carry them out.
4.
Other Required Resources: As part of the plan, you need to identify the other
resources, such as the following:




Software tools
Printing of forms
Outsourced activities
Training
5.
Budgeting: Determine the budget needed to carry out the Quality Assurance
Plan. This is arrived at by estimating the costs of all resources needed for the
Quality Management project.
6.
Timing of Quality Management Activities: This section should define at what
stage each of the other SOPs should take place.
7.
Documents and Deliverables: It is critical to attach to the plan all
documentation required in the QA activities such as test scripts, forms and report
Quality Management
Page 2
structures including reporting formats, templates and tools. Identify the forms to
be used, databases or other documents such as spreadsheets, software tools, etc.
8.
Repository: Indicate where all Quality Management documents will be kept, who
is responsible for them and what the access privileges are. This is critical if such
documentation is to be maintained in electronic format.
9.
Acceptance Criteria: Acceptance criteria must be clearly defined for the project.
The QM Planning template in its electronic form can be downloaded from OMSAR's
website for ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG/QM.
5.5
Good Practices: Defining Acceptance Criteria for SOPs
Most of the Standard Operating Procedures (SOPs) presented in the coming sections
involve direct objectives. They are designed so they can be verified and accepted as they
are completed.
An acceptance procedure is part of the Quality Management culture. It involves the final
test that establishes whether the SOP has passed or failed. Acceptance criteria for each
SOP can be identified beforehand and included in the Quality Plan.
Furthermore, the results of the acceptance criteria are useful. They can be used to
investigate the causes of any variance in required results of the procedures and the
actual results. Improvements can then be proposed and implemented to the SOP in
question.
The QA officer could compile a list of lessons learned that can be shared by the Project
Manager and introduced into the next project Quality Plan. Because quality implies that
here is always room for improvement, this should be a continuing process.
5.6
Good Practices: Project Documentation
At the heart of any quality system lies the documentation. Records are the backbone of
“Say as you do and do as you say”.
Proper documentation ensures the following:





Requirements are to be clearly stated. In this manner, conformance testing can
be carried out objectively.
All stakeholders should be in agreement as to what is being carried out
The history of all works completed is tracked and monitored
The various processes carried out can then be examined when needing
improvement
The stakeholders are held accountable and responsible for all their tasks
Following the principles and methods of Configuration Management (these are covered in
a separate segment which can be downloaded from OMSAR's website for ICT Standards
and Guidelines at www.omsar.gov.lb/ICTSG/CM), the QA Unit must implement a proper
control of all documentation with the following purposes:
Quality Management
Page 3



Document control is used to ensure that the version of the document in use at a
given time (past or present) is known, and that it can be reproduced when
needed.
Changes to the documentation are controlled to ensure that they are well
recorded and tracked
Documents are available for sharing in a secure repository where check out
control is implemented
The type of documents used in the Quality Management project will differ in specific
terms. Generally, these could be any of the following:
Means of Communication with the client
Minutes of meetings
Progress reports
Correspondence using Electronic mail
Timing plans (Software based or hard copies)
Delivery and acceptance certificates
Project Deliverables
Requirements documents
Design documents
Design drawings
User guides
Training Plans
Training sessions
Installation guides
Test scripts and plans
Quality Process documentation
Project plans
Risk plans
Subcontractor management plan
Configuration Management plans
Performance measurements and metrics plan
Communications plan
Training plans
Documentation plans
Deployment plans
Audit reports
Review reports
Documented procedures for verification and validation
Documented procedures for corrective actions and issue escalation
Maintenance, support or warranty logs and registers
Trouble reports
Change request logs
5.7
Good Practices: The Communications Plan
One of the most important issues in Quality Management is the broad communications
that should be followed. Every stakeholder must be aware of his or her responsibilities.
This is best carried out through the documentation defined in the previous section.
Communications Planning covers all channels and media used to share and manage
information in a project. It would be a bit too early to define and plan Communications
Quality Management
Page 4
at this stage. Whatever is already known about Communications Planning can be
entered.
Communications planning develops a set of items to be communicated for which the
following six questions need to be answered:
1.
2.
3.
4.
5.
6.
What information is needed?
What Level of urgency?
Who needs it?
When or how frequently is it to be produced?
In what Form or way will it be sent?
Where is it to be kept?
Use the Communications Planning form which is found with the Software Applications
document. It can be downloaded from OMSAR's website for ICT Standards and
Guidelines at www.omsar.gov.lb/ICTSG/SW.
5.8
Good Practices: Use Performance Indicators
This involves establishing as part of the Quality Management plan various measurements
needed to assure the management that the process is within control.
Here are a few good practices to follow in this area:



A set of minimum essential project measurements and metrics would be required
as it is often difficult to start with a large number of metrics. At later stages,
additional measurements can be introduced.
Benchmark the works so that all metrics can be compared with other procedures.
Use the Ishikawa Cause and Effect diagramming tool to analyze the reasons why
a procedure is failing or not being followed. (A detailed discussion of the tool is
presented in the Risk Management segment. It is discussed towards the end of
the document “Additional Tools and Techniques for Risk Management”. The
document is part of the additional documents of the Risk Management segment
which can be downloaded from OMSAR's website for ICT Standards and
Guidelines at www.omsar.gov.lb/ICTSG/RM.
Performance Indicators are discussed in most of the segments where managerial
standards and guidelines are presented. Such segments as Risk Management,
Configuration Management, Software Applications, etc, have special sections on
Performance Indicators which should be setup in coordination with the QA Unit as some
of these measurements have a direct effect on QA. All of these can be viewed at
OMSAR's website for ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG.
Quality Management
Page 5
6.0
Quality Assurance
Quality Assurance is a meta-activity. It is the quality control of quality control. This
means that when the stakeholders wish to get “assured” that the Quality Control
procedures are valid, ie, can actually trap or identify variance from stated requirements,
Quality Assurance is brought in.
As a process, it covers the following aims:
1.
It examines Quality Control procedures are examined for their validity. Will the
software test scripts actually ascertain whether all functions have been
implemented without error?
2.
Quality Assurance is a managerial tool designed at reviewing Quality Control
activities to ensure that they are actually taking place.
3.
Quality Assurance is resorted to while planning Quality Management activities.
4.
Quality Assurance is concerned with the improvement of the “system” when
imperfections or inadequacies are found. Such inadequacies may reside in the
actual production process itself (Software development, delivery, installation,
etc). They may reside in the Quality Control procedures or even in the QA
procedures themselves.
The following SOPs cover some QA procedures.
6.1
SOP 2: Quality Assurance Audits and Reviews
Objectives: the main purpose of QA audits and reviews is to ensure that the processes
being used to insure quality will actually measure conformance between what was
promised and what was delivered.
A second objective is to ensure that the SOPs are being carried out properly. The QA
review does not concern itself with the quality of the product as much as that of the
process used to produce the product.
Defining Audits: Audits are the process of comparing the project documents against
the Quality Plan. The idea is to look for compliance with the Quality Plan and to find
evidence for all that is stated:



By the Quality Plan
By the contract
By other self imposed standards (such as company policy or company standard)
The QA officer formally documents the result of the audit in a Compliance Report. The
QA officer calls on either upper management or the Project Manager or both to draw
their attention to the non-compliances committed on the project and develop a plan to
fix them.
Defining Reviews: Reviews are the process of reviewing either the project itself, a
document, or a particular procedure. QA reviews are usually informal procedures and do
not result in a Compliance Report.
Quality Management
Page 6
Usually a full review is performed before installation to ensure the fulfillments of all the
deliverables, and the high quality of the deliverables.
Scope of usage: Audits act on the whole project while reviews can be used on any
process such as the various SOPs included in this segment.
Risks:



If QA reviews are not executed on a regular basis, there will be a risk that the
Audit activities described in the next SOP may be found too late not to be
measuring conformance. The reasons for such a deviation need not be related to
the processes themselves as much as to changing circumstances and technology.
For example, suppose a specific process was used to qualify installation and that
after the introduction of new technologies, the tests used to establish whether
installation was completed properly or not were not applicable anymore, the Audit
activity would fail.
If QA reviews are entrusted to persons who are not competent or who do not
understand the processes being reviewed, there will be a risk that poor processes
may be passed when they should not.
Lack of regularity in applying reviews can be damaging
Standard Operating Procedure:
1.
Identify the Quality Assurance procedure to be reviewed.
2.
Identify the persons who may need to assist the QA Unit in the review process.
3.
Collect all documentation being used in the Audit procedure: test plans,
procedures, test results forms, etc.
4.
Walkthrough the procedures and verify whether the activities to be executed will
actually lead to an assurance that such a procedure will test conformance. For
example, if performance qualification is being reviewed for the speed of a network
connection, the various steps in the tests must be studied to verify that they do
lead to the required test results.
5.
Analyze the results of the review and either pass the review or fail it.
6.
If the review passes the procedure, end the procedure.
7.
If the review fails the procedure, analyze the procedure in question and
recommend ways of improving the Quality Control SOP in question.
Note that in this regard, the improvement may be in the actual ICT process itself
(Example: programming in an environment without programming standards leads
to increased bugs). It may also be due to a poor Quality Assurance Audit or
Review process itself. (Example, the test plans are not rigorous and are letting
many bugs slip through them).
8.
Communicate the results to the persons concerned and document all suggestions.
9.
Implement the approved suggestions for improvement.
Quality Management
Page 7
Good Practices for Reviews
1.
Methods to perform reviews can be a walkthrough and an inspection. In the
walkthrough approach, the project team members and the QA Unit together walk
through the ready-to-install product and find out any deviation or error and
correct them immediately.
2.
During review process, the review team should look for:





3.
Necessity of the procedure
Feasibility of the procedure
Correctness of the tests
Clarity of the instructions
Testability of the procedure
Whenever possible, experts can be brought on from outside the environment to
provide council on specific technical issues. For example, the Ministry or Agency
may not have deep knowledge in the procedures needed for testing
telecommunications performance. A telecommunications expert can be brought in
to assist in reviewing the Performance Qualification procedures.
Acceptance Criteria:
The review is considered accepted when the final result is either a pass or a fail that is
approved by the QA Unit and the Management. Furthermore, an additional condition is
required in case the result is a fail and that is the approval of all improvements to the
process along with their documentation.
6.2
SOP 3: Project Management: Planning, Tracking, and Oversight
Objectives: Project Management has become a major managerial area of concern.
Project Management processes are many and are out of the scope of this segment. The
Quality Management of a Project consists of ensuring that the Project Scope is met and
that the scope of the products and services being produced by the Project are being met.
This means that the project triangle consisting of functions (Scope), schedule and costs
are to be determined and planned for.
Quality Management becomes a case of ensuring that the Project Team meets the
promises of the three sides of the triangle as stated in the project plans.
Quality Management
Page 8
Sc
op
le
eo
du
rF
un
he
Sc
ctio
n
s
Quality
Budget
Figure 3 : Control the Project Triangle through Quality Management
Risks:





Confusing the product with the project is a major risk
Not having a project manager for the project
Not consulting with all stakeholders to ensure that their requirements are met
and clearly stated
Not having a fully detailed project plan that consists of a detailed definition of the
Project triangle: functions, schedules and costing.
Not monitoring and controlling all the activities of a project may result in the
triangle creeping out of its stated promises.
Scope of usage: all ICT projects must be subjected to Quality Management.
Standard Operating Procedures
It is not possible within the scope of this segment to present all the SOPs required in a
project. For a start, the SOPs related to Project Management differ from those related to
Product Management. In simplified terms project management handles the politics while
product management handles the engineering or technical aspect of the works (i.e.,
design, programming standards, software tools, etc.)
The SOPs related to Project Management cover such diverse areas as:
Scheduling
Costing
Scoping
Risk Management
Integration
Human Resources Management
Procurement
Communications
Quality Management
These are the 9 areas identified by the Project Management Institute as being the main
areas of knowledge to be addressed for proper Project Management.
The SOPs related to Product Management or engineering strongly depend on the
Quality Management
Page 9
products being produced. For ICT, many of these SOPs are presented in sections 7.0 to
9.0.
Good Practices
General questions can be asked such as:











Did the project start off with a suitable project definition document?
Is there a project plan and is it written according to standard procedures?
Are the objectives and goals of the projects clearly defined?
Is the product scope clear? Are the deliverables clearly identified and specified?
Is the project scope (As opposed to product scope check) well defined and are all
project activities properly defined?
Is the schedule clear and has it been communicated to all concerned parties?
Is the budget accurate and has all the funding been secured?
Are all relationships with vendors clear?
Is there a communications plan for the project? Is it being followed? That is, is
there an agreement on the level and frequency of formal communication that is
expected during the project?
Is there a risk management plan for the project?
Is there a configuration management plan for the project?
At the beginning of the project, it is a good practice to respond to these questions:







Have the project sponsor (System Owner or beneficiary) and key stakeholders
been identified?
Did the key stakeholders participate in the planning?
Are all their requirements addressed and clearly stated?
Have the sponsor(s) and major stakeholders formally approved the Project
Definition?
Have a number of milestones been established to review progress so far and
validate the project is on track for completion?
Are the resource requirements adequate and have they been well estimated?
Does the estimated effort, cost and duration appear reasonable?
At every major milestone, the following questions can be asked:















Is the Project Plan being used to manage the work performed by the team?
Does the Project Plan accurately reflect the remaining work effort?
Can the Project Manager monitor the project and can he or she control the
various activities in the project?
Has the product scope up to this point been completed?
Has the project scope up to this point been completed?
Are the project finances being actively managed to complete within the budget?
Is the project on track in terms of function, schedule and costs?
Is Risk Management being properly applied and are all risks being responded to?
Are issues being addressed and resolved in a timely manner?
Are scope change requests being properly identified and managed?
Were any major changes encountered that require that the Project Definition be
updated?
Are status reports and status meetings being utilized?
Are major stakeholders being communicated to effectively?
Are the business customers happy with the project progress so far?
Are customer expectations being properly managed?
Quality Management
Page 10

Are proper processes being utilized to ensure completeness, correctness and
overall quality?
At the End of the Project, the following questions can be asked:







Have the deliverables specified in the project definition and the contract been
completed as specified in the Requirements Document?
Have all agreed-upon deliverables been formally approved and accepted?
Have all contractual obligations been completed?
Has turnover occurred as expected based on prior agreements?
Has training occurred as expected based on prior agreements?
Has documentation been completed and turned over as expected based on prior
agreements?
Have payments been made based on the completion of agreed-upon interim
deliverables?
Project Management principles are wide in scope and should be learnt and adhered to for
the success of ICT projects.
Quality Management
Page 11
6.3
SOP 4: Configuration Management
Configuration Management is the basic control mechanism that establishes and
maintains the integrity of the physical and functional aspects of a configuration of ICT
resources throughout life cycles of projects or operations.
Configuration Management relates to Quality Management in that it is the mechanism
that ensures that a configuration of software or hardware components remains the same
unless changed through a disciplined change control procedure. Documented control of
all changes ensures that the user gets a configuration that is fit for its purpose.
Configuration Management is a separate segment and can be downloaded from OMSAR's
website for ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG/CM.
Configuration Management is made up of four processes. The first is executed on a one
time basis while the last three are ongoing processes:




Process
Process
Process
Process
1:
2:
3:
4:
Plan the Configuration Management Project
Identifying the Items in a Configuration
Implementing the Change Control Procedure
Monitoring and Tracking the Configuration
These processes achieve integrity by essentially controlling the changes made on the
initial configuration and tracking such changes.
These processes are presented in detail in the Configuration Management Segment.
Kindly review that segment for a full discussion of the various SOPs.
Quality Management
Page 12
7.0
Quality Control of Software Development
Software Applications are discussed in detail in a separate segment. The SOPs covered in
that segment are the following:
SOP
SOP
SOP
SOP
5:
6:
7:
8:
Requirements Management Quality Control
Qualification of Functional Design
Qualification of Technical Design
Software Quality Control (Testing)
Section
Section
Section
Section
7.2
7.3
7.4
7.5
The above SOPs are therefore not discussed in this segment.
7.1
What About Commercial Off the Shelf Software Applications?
Commercial Off the Shelf Software or COTS are applications that have been successfully
developed, i.e., have passed all the SOPs in this section and are therefore ready for
Installation, Operation and Performance Qualification.
Section 8.0 follows and discusses the QC SOPs for the Deployment and Operation of
Systems and Products. These SOPs would also apply to such Software Applications.
7.2
SOP 5: Qualification of Requirements
The Purpose of the Analysis of Requirements is to establish a common understanding
between the internal customer or user and the software developer. The understanding is
that of the internal customer's requirements to be addressed by the software
development project. The result of Analysis of Requirements is a clear list of all
stakeholders’ requirements. Usually, for software applications, this may consist of UML
Use Cases and related Business Objects. For other systems or for non-functional
requirements, these are detailed lists consisting of various specific requirements, needs
or expectations. (Refer to the Software Applications Segment for a full procedure on how
to Analyze Requirements).
Quality Control in this area examines compliance and ensures that the Requirements
have been developed in detail and that all stakeholders’ needs have been addressed. It
also ensures that the requirements are reviewed by all, resolving conflicting
requirements and ensuring that each stakeholder approves the final list.
Kindly refer to the related segment on Software Applications for a detailed presentation
of this SOP.
7.3
SOP 6: Qualification of Functional Design
The purpose of Functional Design is to prepare a document that specifies very clearly
what functions to include in an application, how these functions interact and what
processes they result in. The document will be essentially addressed to the user or the
internal customer in the Ministry or Agency. However, it will be used by designers to
generate a Technical Design (Discussed next).
Quality Management
Page 13
Functional Design is usually developed using suitable modeling languages: charts,
diagrams, specific notations supported by text. Different techniques and methods are
used.
The purpose of this Quality Control activity is to ensure two major requirements:


That a process exists for Functional Design and that it is being adhered to
That the specific Functional Design being audited does conform with the user’s
requirements and will lead to a working system.
Kindly refer to the related segment on Software Applications for a detailed presentation
of this SOP.
7.4
SOP 7: Qualification of Technical Design
The purpose of Technical Design is to convert the Functional Design document into
detailed instructions to the development team. It specifies all the technical issues needed
by such a team such as ERDs, screen layouts, programming standards, class definitions,
report layouts, interfaces with other systems, component deployment, etc.
The Technical Design document will be essentially addressed to the developers in the
Ministry or Agency or in a Vendor’s software unit.
Technical Design is usually developed using suitable modeling languages: charts,
diagrams, specific notations supported by text. Different techniques and methods are
used.
The purpose of this Quality Control activity is to ensure two major requirements:


That a process exists for Technical Design and that it is being adhered to
That the specific Technical Design being audited does conform with the Functional
Design and will lead to a working system.
Kindly refer to the related segment on Software Applications for a detailed presentation
of this SOP.
7.5
SOP 8: Software Application Testing
Successful development of software applications relies on efficient testing. A variety of
tests should be applied. Software Quality Control covers the conducting of such tests
while Quality Assurance covers the auditing and review of tests and their procedures.
There is a variety of tests that can be applied such as unit tests, integration tests,
interface testing, etc. All of these tests are defined in the related segment on Software
Applications for a detailed presentation of this SOP.
Quality Management
Page 14
8.0
Quality Control of the Deployment of Products and Systems
This section covers the Quality Assurance activities put in the form of Standard
Operating Procedures (SOPs) for the deployment of products and systems. The SOPs
presented are:
SOP
SOP
SOP
SOP
9: Qualification of Technical Specifications
10: Qualification of Installation
11: Qualification of Operation
12: Qualification of Performance
Section
Section
Section
Section
8.2
8.3
8.4
8.4
These SOPs apply to the following:













8.1
Hardware components
Operating software
Software tools
Commercial off the shelf Software (COTS)
Software that has been developed and released for installation
Network components
Networks
Communications items
Room and environment components
Consultancy services: design, supervision, project management, etc
Training
Maintenance
Support
SOP 9: Qualification of Technical Specifications
This SOP is a technical audit of any product or service that the Ministry or Agency is
interested in acquiring externally or developing internally. Once developed, the Technical
Specifications become the basis of Delivery and Acceptance of the product or service or
system and hence the heart of any Commercial or Internal Agreement.
Example: a UPS has Specifications such as power capacity, battery capacity in amperehours, number of output connectors, whether there is an RS-232 connection and related
software to use on the connected PC or not, whether there is an alarm when power is
down or unstable, etc. These are technical Specifications and can be verified by a
Specifications Qualification. Other issues to be raised in the Specification relate to how
the UPS operates:





What happens when the power goes down?
What happens when the power resumes?
How do we charge the unit on first use or after stoppages?
How do we turn off the alarms?
How do we run the software on the PC so that it can shut down the operating
system with impending power drops?
Objectives: This SOP has as its main purpose the verification that the specifications as
stated in the Technical Specifications document correctly reflect the requirements of the
internal customer and are sufficient to act as a means of accepting the delivery of the
item.
Quality Management
Page 15
Risks:





Technical Specifications not covering all aspects of the item
Specifications prepared by persons without the proper and related experience
Specifications based on proposals or verbal definitions
Specifications that are over-stated and lead to higher priced goods
Specifications that are too wide and lead to a corresponding variety in the
proposed goods
This Activity covers a variety of situations:




when the Ministry or Agency is issuing an RFP that contains a design
When the design team of the Ministry or Agency is preparing designs for its
internal use while developing or building systems
When the supplier is proposing a system and must include a design with the
proposal
When one of the products or services a supplier is delivering includes a systems
design.
Standard Operating Procedure:
The following procedure applies to all types of items listed earlier:
1.
Identify the party that will approve the design specifications. The party should
be given the authority to approve such a design. This may be a single user or a
Technical Evaluation Committee. Such a party should have the proper experience
in the related area.
2.
Establish Standards: define ahead of time what standards are required to
ensure that a specific function or feature is properly specified.
Example: when acquiring Monitors, there may be a standard that defines the
minimum to be 17” screens.
Example: while defining a network, the Ministry or Agency may have a standard
for drawing the network topography may be used.
This is a critical step as it is such standards that will be used when evaluating the
design in this SOP.
3.
Collect Required Documents: identify and collect the documentation that the
Design would be based on. Such documents as the following may be needed:









User requirements specification
Functional specification (if available at this time)
Supplier qualification or Audit Report
Any related agreements
Any purchasing standards
Technical drawings
Data sheets for equipment
Catalogs
Etc
Quality Management
Page 16
In the case of including the design in an RFP, such documents must be included in
the RFP as an appendix.
4.
Involve Users: ensure that the staff who will be the direct beneficiaries and
users of the deliverables are part of the planning process that defines what is to
be designed. The users must then analyze all documentation and approve it.
5.
Review Specifications: the team setup to evaluate the specifications will now
review the specifications and establish whether or not they consist of a complete
system design that can be used for later construction, development, delivery,
installation and use.
6.
Approve the Specifications: once the Specifications are seen as fit, they can be
approved and hence, released as the Technical Terms of Reference.
Documentation or Deliverables:




The design documents
All supporting documents
Standards for accepting Specifications
Specification Qualification Review report
Good Practices and Recommendations:
1.
In the case of complex designs, it may be useful to submit the design to experts
outside the Ministry or Agency. An independent objective review may pinpoint
design weaknesses that are hard to see by the team developing them. Such
parties can be other ICT Units in the Government or independent third parties
such as Consultants.
2.
The presentation of the design in the RFP will be read by private sector parties. A
good practice is to ensure that the design, is very clearly presented. Some
designs that are included in RFPs are only clear to the parties that developed
them.
3.
What to do with Demonstrations? Many disputes arise from Agencies
acquiring software only presented before agreement through live demonstrations.
Such demonstrations are risky because:



Demonstrations are almost never complete
Demonstrations may hide a lot of problems with the system
Systems being demonstrated may not be the same as those being
delivered and it is difficult to compare the two.
To avoid this problem, consider demonstrations only as a means of clarifying
various functionalities. They would be used as a review of the “qualitative
aspects” of the systems. They should not replace Technical Specifications.
Acceptance Criteria:
Once the party authorized to approve the design has issued its approval, the
Qualification of Technical Specification is considered as accepted.
Quality Management
Page 17
8.2
SOP 10: Qualification of Installation
Installation is a process that should be separated from delivery and operation. It is usual
to sign off delivery by a simple counting exercise. Such deliveries are not part of this
segment. However, installation follows strict technical rules such as in the case of
servers, software applications, operating systems, room items, etc. There needs to be a
procedure to qualify that a proper installation has taken place.
The next step after installation is the Qualification of Operation. This is discussed in the
next section.
Objectives: before an ICT system or item is brought into use, it should be properly
installed and confirmed as being capable of operation. The objective of this Activity is to
provide an SOP for the Qualification of installation of a wide variety of systems. This is
essentially a delivery process and should not be confused with operational qualification
to be discussed in the next SOP.
Definition: Installation qualification (IQ) is the documented verification that all key
aspects of the installation adhere to approved intentions. These intentions could have
been expressed through:






Technical specifications
Design specifications
System specifications
Manufacturers' recommendations
Developers’ recommendations
Data sheets
Scope of usage: hardware, software or systems related to networks and
telecommunications.
Risks:





Confusing installation with operation
Improper or lacking installation procedures
Installation by persons not qualified to do the job
Installation without qualification through tests and clear results
Lack of a clear definition as to who can authorize the proper installation of the
system or item.
Standard Operating Procedure:
1.
Identify the party that will approve the installation. The party should be given
the authority to approve the IQ. This may be a single user or a Technical
Evaluation Committee.
2.
Standard Installation Procedures: these are SOP’s that define Standard
Installation procedure for each type of system or deliverable being installed.
Installation procedures should be defined ahead of time by various parties
depending on the nature of the system being delivered.
The SOP will ensure that all components of the ICT System are installed as per
the specifications of the supplier or developer. The SOP should contain the
following:
Quality Management
Page 18





The step by step installation procedure
The tests needed to verify that the installation has been carried out
successfully.
The installation acceptance criteria without which the Ministry or Agency
will not be able to establish whether a system has been properly installed
or not.
Site or environmental conditions necessary for the proper installation of
the system.
The party authorized to carry out the installation.
Should such a procedure not be available, then the ICT Unit should prepare one
to ensure that the system is being installed according to a standard process. The
procedure may also include guidelines and related options.
3.
Install the system as per the standard instructions and confirm each step.
4.
Record all the test results.
5.
Pass or fail the installed products.
Documentation and Deliverables:





Standards Installation Procedures for the type of system in question
The test results produced by the Installation Procedure
A document confirming the proper installation of the system in question.3.
A list of all documentation submitted with the system
A list of all components in the system to be used for Configuration Management
Good Practices and Recommendations:
1.
In all agreements, insist that Suppliers provide Installation procedures. Without
these, proper IQ cannot take place.
2.
Document all installation problems for later use in Risk Analysis.
3.
Update the Configuration Management database to reflect the installation.
4.
Environment: make sure that physical location or site meets the installation
conditions specified in the system documentation for such factors as air
conditioning, low moisture, power, and room conditions.
5.
Ensure that all related software is the proper version for the installation.
6.
System Documentation: a list of all system documentation promised to be
delivered with the system should be compiled and tested against delivery of such
documents. The documents will be subjected to Configuration Control and must
be registered in the Configuration.
Acceptance Criteria: the system is fully installed when the installation process is
fulfilled as per the supplied Installation Procedure. It would then be ready for Operational
Qualification.
Quality Management
Page 19
8.3
SOP 11: Qualification of Operation
Operation is the process of using the system’s or the item’s functions.
Difference between installation and operation: there is often a confusion between
installation and operation. Many ICT systems can be installed and confirmed to be ready
to operate without being fully operational.
Example: a software developer may install a full software application and confirm that it
can be launched, some menu items can be accessed and it can be shut down without
error. However, this does not mean that it is fully operational.
Difference between Operation and Performance: OQ implies that all functions and
features promised with the system must be operational. Performance implies that other
factors such as loads, volumes and other capacity and power related issues are not
problematic. Performance Qualification (PQ) is dealt with in the next Section.
Objectives: Operational qualification (OQ) is the documented verification that each unit
or subsystem operates as intended throughout its anticipated operating range.
A system that has been properly installed is not often tested for proper use and
operation. Before bringing such systems into full operation, they should be properly
tested and confirmed as operating properly. The objective of this qualification is to
provide an SOP for the Qualification of Operation of a wide variety of systems.
OQ tests the built-in capabilities of the new system. In OQ there is a focus on specific
functions and how these functions can be tested. OQ consists of such assurances and
questions as the following:




Do key functions of the system operate correctly as specified in the design?
Does the system have the ability to operate all the software that either came with
it or was installed on it as part of the installation procedure?
Can we demonstrate that the functions are working properly without errors and
missing functionalities?
Can the system connect to all its units, be networked or be directly connected
and use the connectivity for the various functions involved?
Scope of usage: hardware, software or systems related to networks and
telecommunications.
Risks:





Confusing installation with operation
Improper or lacking operations procedures. This often results from improper
Technical Specifications. (See the earlier SOP for this qualification).
Operation by persons not qualified to do the job
Operation without qualification through tests and clear results
Lack of a clear definition as to who can authorize the proper operation of the
system or item.
Standard Operating Procedure:
1.
Identify the party that will approve the Operational Qualification. The party
should be given the authority to approve the OQ. This may be a single user or a
Technical Evaluation Committee.
Quality Management
Page 20
2.
Installation: ensure that the system has been installed properly through a
review of the Installation Qualifications results.
3.
Operating Instructions: Ensure that the system has the proper operating
instructions. These could be SOP’s that define operation or use procedures for
each aspect of the system being tested. Operating Instructions should be defined
ahead of time by various parties depending on the nature of the system being
tested.
4.
Test Procedures: ensure that the system has proper test procedures. It is these
tests that will decide whether the system is “Qualified” as operational or not.
5.
Operate each function of the system to ensure that they are in working order.
This can be carried out by the supplier of the system or by persons identified in
Step 1 above.
6.
Record all the test results.
7.
Pass or fail the products or services.
Documentation and Deliverables:
1.
2.
3.
4.
Standards Operating Procedures for the type of system in question
A document confirming the proper operation of the system in question.
A list of all test results
A list of possible causes and remedies in case the errors are diagnosed while
operating.
Good Practices and Recommendations:
1.
Include users in the test process
2.
Cover all “testing zones”. For example, there are many systems that get passed
for regular operation leaving out of zone tests such as closing of periods or the
use of large values during data entry or the testing of record sharing and locking
in multi-user environments. Usually, the reason is due to lack of data, knowledge
or time.
Acceptance Criteria: the system is fully operational when the test process is completed
and no errors or missing functions are noted. Note that operation may be completed and
yet the system may not be performing properly. For this, the Ministry or Agency may
need to consult the Performance Qualification Activity.
Quality Management
Page 21
8.4
SOP 12: Qualification of Performance
Some systems will perform differently under different loads and conditions. For example,
a network may have a specific performance requirement. A software application may be
designed properly but may fail to meet the required loads because of poor programming
or too many database calls.
It is therefore important to resort to a Performance Qualification for those systems
where performance is important.
Objectives: Performance is a measure of various parameters in a system such as
speed, response, capacity, power, etc. Performance Qualification (PQ) ensures that the
total system performs as intended in the specified operating range. The aim of this
Activity is to develop a Standard Operating Procedure that allows the Ministry or Agency
to verify that its systems are performing the way they should.
Scope of usage: the system includes all hardware and software components, associated
equipment, people and procedures that make up the system.
Which systems should be performance tested?










New systems that are being delivered and used for the first time
Existing systems to be qualified on a regular basis
Systems that have undergone modifications
Systems that have had additional usage
Systems that have seen sudden deterioration of performance should be tested
and qualified
Systems expanded to operate at a higher capacity
Lack of knowledge as to what constitutes a performance measurement.
Lack of knowledge of the indicators that define whether the measurements are
within or outside the proper ranges.
Wrong measures are used.
Measurements taken in wrong environments or using wrong data.
Risks:





Acquiring systems without proper Performance Qualification
Improper or lacking performance procedures and tests
Performance testing by persons not qualified to do the job
Use of tests that do not cover all real life situations. For example, tests on small
databases or tests on loads that are not realistic because no study was made of
load distribution.
Lack of a clear definition as to who can authorize the performance testing of the
system or item.
Standard Operating Procedure:
The following procedure is used to qualify if a system is running within its proper “Zone”
of performance:
1.
Defining the operating environment: a description of the use of the system in
the context of the work environment should be developed. For example, it should
be specified that this system is to be used by the employees in the Ministry or
Agency during their office hours from 8:00 till 14:00.
Quality Management
Page 22
2.
Setting operational limits: the SOP shall include testing the system against
(but not exceeding) its operational capacity.
The performance levels should be set by the user but should not exceed the rated
capacity provided by the supplier.
A set of measurement indicators should be developed and approved before
running the Performance Qualification (PQ) Scripts. Here are some examples of
indicators:



When 20 users are using the system, the response time for accessing a
Citizen’s record should not be longer than 2 second.
Opening a web page and viewing all images on the Intranet should take no
more than 1 second.
Preparing a full statement of all certificates should not take longer than 15
minutes.
These indicators define the expected results of the test or the qualification.
3.
Test Data: disputes will arise when different parties do not agree as to what data
is to be used. Agreements do not often specify that and Suppliers are tempted to
use test data rather than actual data. Users will often refuse to use actual data
stating that the data is not representative.
Therefore, the SOP should define what the Qualification Data is: actual data for a
full month? A full year? Random data generated specifically for the Qualification?
Abnormal data or data which is outside the operating ranges should also be
tested to ensure that it is handled correctly in the system.
Whatever data is to be used, the justification for using it should be documented.
4.
Requirements for setting up the data: should it be required to setup special
data for the tests, the procedure and requirements for such setup should be
documented in the SOP.
5.
Test Procedures or scripts: the SOP must contain one or more Qualification
Scripts that describe the procedures needed to verify the performance of the
system against the User Requirements or the Indicators established earlier.
They should simulate the operation of the system in a live situation, using all
system components and operating procedures.
Here is an example of a typical database application test procedure or script. The
purpose is to test the response of the system using a Search program when 20
users are busy posting transaction vouchers:
1. Initialize the database to purge all transaction records
2. Generate test data as per part 3 of the SOP. (This is not included in this
example but can briefly be introduced as: create 200 citizen records, create
50,000 transaction records, etc).
Quality Management
Page 23
3. Have 20 users log onto the system. They should be ready to use the
Transactions entry program when instructed to do so. For the moment, there
should be no entry.
4. Make ready a test station and log onto the system. Launch the program that
searches for citizens using the location parameter. The program will display all
citizens that meet the entered location.
5. Do not search yet.
6. Prepare the users to start entry of vouchers without interruption. A supervisor
will time the start of entry so that users start one after the other with a 5
minutes interval.
7. As each user starts, the tester at the test station will search for the citizens
and note down the time it took the system to respond.
8. Once the last of the 20 users is left working for 10 minutes, the test is over.
9. Plot the response time as the users come on to the system and analyze the
results.
Since test procedures are often carried out by persons not located in the same
room, it is necessary to have a good way of communicating the procedure and
ensuring that all users know how to execute it.
6.
Test Results: enter the test results in a form reserved for such results in the
SOP. The executor should record, sign and date the results. Screen prints or
electronic logs should be retained to verify the results.
7.
Unexpected Results: systems may crash or show unexpected behavior. This
should be documented for later analysis.
8.
Should the SOP result in disqualifying the system, two steps are required:


Diagnose the problem causing the lack of performance
Initiate remedial action
The system should then be resubmitted for Performance Qualification.
9.
Qualification Report: if the PQ generates extensive documentation, then a
summary report should be written. It should include such information as:




Whether or not the qualification scripts were followed
Whether or not the expected results were attained
Description of any deviation from expected results
Any follow-up activities to correct any deviations
Documentation and Deliverables:




The definition of the environment
The operating limits of the system
The Performance test scripts
The Qualification Report to include all test results, unexpected behavior and any
remedial action that is required for correct performance.
Good Practices and Recommendations:
Quality Management
Page 24
1.
When systems are being designed or acquired from Suppliers, it would be the
right time to establish broad lines for Performance Qualification. At a later date,
more specific SOPs can refine the qualification criteria.
2.
Where appropriate, automated testing tools may be used to record results. These
are available commercially.
3.
Data may need to be charted or analyzed for averages, standard deviations,
trends, etc.
4.
In some situations where performance suddenly deteriorates, there may not be
qualifications scripts. In this case, historical information may be used. However,
the actions taken, the data used and the results obtained when the historical data
was generated must be clear.
5.
While carrying out Performance Qualification tests, the Change Control history for
that system must be analyzed to ensure that no changes have been introduced
that would affect test results. (Example: some systems have their memory
reduced to be installed on other systems. This would affect test results).
6.
The PQ should always be performed at the user site and will include testing
specific to the user environment and defined ways of working.
Acceptance Criteria:
Upon the successful completion of a Performance Qualification SOP for each system,
application, etc, that item is considered as passed. When remedial action is to be taken,
then the subsequent PQ will be subjected to the same conditions.
Quality Management
Page 25
9.0
Quality Control of ICT Operations
ICT Units have various critical operations or processes that need to be audited on a
regular basis. These are usually ongoing operations and are probably not parts of specific
projects.
The following SOPs are be presented in this section:
SOP 13: Maintenance Control
SOP 14: Data Entry Control
SOP 15: Training Quality Control
9.1
Section 9.1
Section 9.2
Section 9.3
SOP 13: Maintenance Control
Objectives: Maintenance is one of the ongoing operations of ICT. It is often ignored
until problems arise. The objective of this activity is to provide an SOP for controlling the
Quality of maintenance activities of ICT products and systems. This will ensure that their
purpose is met.
Definition: Maintenance is the process of attending to the product or system after its
operation and usage. Maintenance activities usually cover the following:




Preventive Maintenance
Intervention due to failures
Installation of additional functions
Upgrades and updates
Notice that even if maintenance contracts cover installation of upgrades or items, the
various SOPs discussed in Section 8.0 should be carried out.
Scope of usage: all types of ICT processes.
Risks:






Installation of new items without following Installation Qualification
Operating new items without following Operation Qualification
Not resorting to Performance testing on changing the system in a major way. (For
example, a fault resulted in the need to change the CPU of a particular PC.
Performance may be impaired if the parameters of the CPU differ slightly from the
original item).
Not clarifying the period during which maintenance requests can be made
Not conforming with Configuration Management procedures should maintenance
require a change in the configuration (Updates, replacements, relocation, etc).
Not having a valid and clear maintenance agreement
Standard Operating Procedure:
1.
Help Desks: Identify a person or unit within the Ministry or Agency responsible
for receiving all complaints from users. This person or function shall be referred
to as the Help Desk and shall be trained to log and dispatch and follow through
the resolution of the complaint. The Help Desk belongs to the IT Unit and it is
Quality Management
Page 26
preferable that the person has an administrative profile (technical people tend to
just fix the problem without logging it because it is not a big deal..
2.
Configuration Management: ensure that all items are part of the Configuration
so as to prevent ambiguity when attempting to service them.
3.
Define the Maintenance Procedures: The maintenance procedures can be
customized to best serve the Ministry or Agency. However, at a minimum they
shall include:

Report the problem, complaint, or request for assistance using a Change
Request Form (CRF). Even when the user calls the Help Desk directly on the
telephone, the Help Desk should fill for the user a Change Request Form.
Errors that were not submitted in writing do not exist. Note that electronic
systems are acceptable, i.e., users can submit their CRF using the Web in
technically advanced agencies.

The Change Request Form shall include the date of the occurrence of the
problem, the date of the request, the name and location of the user, the
identification of the equipment in question, and a thorough description of the
problem. Sentences like “the ups beeps” or “the report does not print” or “the
bill does not compute” are not acceptable. The help desk shall be trained to
ask the users to identify patterns, to identify frequencies and repeatable
actions that lead to the problem.

In case of software system the description shall always include the name of
the screen, report, or program, and a verbatim transcript of any error
messages on the screen. The help desk shall be trained to know where to
expect screen and report names, as well as the location of the error messages
on the screen. Finally, the help desk will try to identify with the user
repeatable sequence of actions that cause the problem to occur.

The Help Desk shall be trained to dispatch the call to the service people.
These are the software supplier, the hardware vendor, the IT department, the
technician, etc.. The Help Desk shall be trained on the means of
communicating with the service people (e-mail, phone, fax, website, etc..)
Finally, the Help Desk shall be trained to look up the maintenance agreement
with the service people. If any costs are to be incurred then the approval of
the department head must first be secured before dispatching the problem.

Service people shall be formally notified that they must stop by the Help Desk
on the day of the maintenance visit in order to update the Maintenance Log.
The Maintenance Log is a document including one entry per CRF in
chronological order and containing the CRF number (if any - preventive
maintenance actions have no CRF numbers), a summary of the CRF or
maintenance purpose, the equipment number, the module name if any, the
classification of the maintenance into the four categories listed above (more
categories can be added), the action performed as part of the maintenance,
and the date of the action. If more than one visit or action has been
performed before solving the problem, the Maintenance Log shall include one
entry for each intervention, visit, action on the same CRF.

The user who requested the CRF is the person who signs the Maintenance Log
entry to close the request.
Quality Management
Page 27
3.
Define the Change Control Procedures. An added twist for software systems
is the need to differentiate between bugs and new functionality. The System
Owner, (usually the head of the department who commissioned the system),
must approve the new functionality before the Help Desk dispatches the request,
be it to the maintenance provider or to the IT department. There might be some
discussion on what the maintenance provider will and will not do as part of the
maintenance contract. In such cases, the System Owner may abort and close the
request.
There is yet another twist if the ICT system is under configuration management. CRF
must be bundled together (fixes of bugs or additions of new functionality), and applied at
once to the system through releasing a new version of the system. When releasing a
new version, all version installation procedures are documented, existing documentation
is updated, the reasons for updating the version are clearly stated, and integration
testing is performed on the system after the upgrade. It is recommended that a backup
be taken before upgrading the version.
4.
Update existing documentation. The Help Desk shall be aware of the
documentation existing for the ICT system at hand. The IT department shall
update the documentation by reviewing the Maintenance Log periodically. (every
six months is recommended).
5.
Schedule preventive maintenance for equipments, databases, and systems on
a regular basis. Don’t wait for degradation in performance. Preventive
maintenance must also be logged in the Maintenance Log.
Documentation and Deliverables:


Change Request Forms
Maintenance Log: an integrated one, or one per equipment/system.
Good Practices and Recommendations:
Automate the Maintenance Log. Even when using a spreadsheet, it is possible to sort
entries by type of maintenance, date, supplier, equipment, etc.. Hence be able to make
intelligent recommendations about maintenance procedures changes.
Acceptance Criteria: The maintenance is done when the user requesting the
maintenance or the System Owner sign the Maintenance Log entry.
Quality Management
Page 28
9.2
SOP 14: Data Entry Control
Objectives: This SOP has two aims:


To ensure that all data entry procedures are documented
To ensure that they are being used properly
Meeting the above objectives will reduce errors and improve the speed of entry so that
the overall throughput of a data entry unit also improves.
Scope of usage: generally, this would apply to data entry for operational,
administrative and financial application software. However, in some cases, there are data
entry procedures that are covered by Office Technology products.
Risks:



Improper entry of data causes errors, rework and possibly damages with citizens
and other users of the data.
Erroneous or slow entry of data leading to loss of time and re-entry. This reduces
the throughput of the system.
Lack of standardized data entry will reduce the flexibility of data entry by making
staff specialized in their work.
Standard Operating Procedure:
The SOP is essentially made up of two parts. One part ensures that the Data Entry
Procedures are properly documented. The other part ensures that they are being used
on a regular basis.
1.
Documenting Data Entry Procedure: For each data entry form in each
software application, a detailed Data Entry Procedure is to be prepared. If the
software is well documented, it should have such procedures. If not, such
procedures should be prepared and clearly documented.
The Data Entry Procedure would be used in the following cases:
Training
Day to day entry tasks
Validation and consistency checking
Auditing purposes
Refreshing the user
A Sample Data Entry Procedure is presented in the Appendix.
2.
Testing Data Entry Procedures: On preparing a Data Entry Procedure, it
should be tested to ensure that all data elements are clearly explained and that
the procedure is easy to understand and use. The results of the test are to be
documented.
On a regular basis, the QA Unit should review the data entry activities to ensure that the
procedures are being used.
Documentation and Deliverables: all Data Entry procedures
Quality Management
Page 29
Good Practices
1.
Error analysis should be carried out. When errors are reported, they should be
registered and totaled. If specific forms are found to generate more errors than
others, then there is the possibility that the form is not being properly understood
or that it may have programming problems.
2.
Data Entry Procedure scripts: Include all data entry procedures in the
Configuration so that their versions can be controlled.
3.
Audit Trails: Use these when possible. Audit Trails are rigorous ways of verifying
proper data entry. Throughout the years, the term got distorted to mean the
following: a list of transactions as entered. This is not correct. Audit trails
technically mean the following. A batch of vouchers is entered. One of its fields is
accumulated manually, say the amount on the voucher. The batch is controlled as
one single group of vouchers by the system. The batch or Audit Trail is entered
into the system. The batch is not updated or posted to the database unless the
computed total equals the entered total. Along with this checking, the system
must be able to produce a validation list of all vouchers before committing the
entry.
4.
Whenever the Ministry or Agency is developing its own systems or requesting
such systems to be developed by third parties, it is essential to follow some Data
Entry standards
5.
Data Integrity: various systems are designed without proper data integrity. For
example, there are many Inventory Management systems that allow the quantity
to go negative. There are cases where the accounting vouchers posted from an
operational module do not correspond to the totals recognized by the accounting
system. For example, purchases in the Inventory Management system do not
correspond to the total inventory in the Accounting System. Such areas should be
investigated and checked on a regular basis to ensure that there is Data Integrity.
6.
Checks and Control: The following types are important to implement in all data
entry screens in software applications:








Validate all field that have ranges such as dates or amounts
Try to increase the number of lookup tables so that users do not enter country
codes or currencies whichever way they wish.
Allow the user, under privilege control, to add a parameter that is not in a
lookup table on the spot without having to go to another screen.
Allow the user to search for major tables such as citizens, projects,
contractors, etc. This should be available during deletions, modifications,
printing and other system functions.
Design screen layouts to be similar to actual vouchers. This eases data entry
and requires less training for the user
Use clear color coding as per Windows standards: Black labels, White for
enterable fields and Grayed fields for non-enterable or for system responses
Differentiate between Info, Error and Warning messages through the proper
use of buttons: Info (OK), Error (OK), Warning (Yes, No), Choices (Yes, No,
Cancel).
Use clear and unambiguous messages
Quality Management
Page 30


Avoid cluttering the screen with a large number of fields. It becomes difficult
to visually scan the screen and validate the data. In the case of large number
of fields, it is best to use TABs or even multiple screens.
Do not allow the system to accept to create or modify a record unless all data
is validated. Many systems suffer from temporary entries that are never
completed.
The above good practices should be standardized across various applications to ensure
that users get familiar with the look and feel of applications and hence require less
training.
Acceptance Criteria: Data Entry Procedures are critical and must be validated by the
developers of the software and the users in the related Departments. Upon full validation
that such procedures are corrected, it is recommended that they be tested. They can
then be accepted.
9.3
SOP 15: Training Control
Objectives: The objective of this Standard Operating Procedure is to ensure that
Training is applied as required. Since ongoing training is such an integral part of ICT
operations, there are various aspects of Training that need to be controlled for Quality.
Training can be provided by the supplier, arranged for in a formal training center, or
informal through demonstrations and workshops. The goal is the transfer of knowledge
from the people who have produced the systems to the people who will use it.
Scope of usage: Training covers all aspects of ICT operations:







Software applications
Operating software
Office technology products
Hardware
Software development processes
Networks and communications
Etc.
Risks: the lack of proper training will result in the following risks:



Loss of interest on the part of the user hence loss of interest in the system
Misuse of the system hence potentially down time
Failure of the intended ICT project because turn over to the client has been
unsuccessful due to not knowing how to operate the system due to poor training
Standard Operating Procedure:
1.
Pre-requisites of the Training SOP: in order to be able to control the Quality
of Training, several requirements have to be clarified in detail:


A proper ICT Unit organizational structure needs to be defined. This leads
to the definition of the terms of reference and classification of all jobs.
From the above, it will be known what competencies are required for each
job and at what stage in time.
Quality Management
Page 31



Competency testing must be applied to assess the level of each person in
the ICT Unit. The shortages are then identified and converted into training
requirements.
Knowledge must be acquired about the various training resources
available: institutes, courses, certifications, internal experience, etc.
A mapping is then prepared to chart the training needed per person, and
where it should be carried out and when.
2.
Scheduling: a register is to be setup that defines all training to take place. This
would include names, dates, type of training, source of training, etc.
3.
Training planning is documented so that all trainees can confirm their ability to
attend the required courses.
4.
Perform the training taking care of having prepared adequate training
literature. The user guide, other guides, the scenario and testing plan, and
especially conceived training material are all very good training materials as long
as the teacher has adequately prepared himself to the messages he/she wants to
transfer to the end users.
5.
Evaluate the trainees: This is completed through internal tests or through
results provided by the instructors. Such evaluations must be recorded in the
Training register defined earlier. It is Good Practice to evaluate the courses
themselves to be able to improve the Quality of such training. A third kind of
evaluation is ongoing refresher testing to ensure that staff are retaining what
they have been taught.
6.
Control the Variations: as part of Quality Control, it is essential to control the
following variations:




Have all the trainees reached the level expected from them?
Have all the courses delivered the level expected from them?
Has the training improved the operations?
Have all training materials been supplied and are they registered through
Configuration Management and made available for sharing by the rest of
the team?
In the case of variations, the ICT Unit must be able to analyze the situation and
plan for improvements.
Documentation and Deliverables:









List of required competencies
List of training resources: institutes, courses, etc.
Training material
Register of all training material
Other technical reference material
Test plans for evaluation of trainees
Test plans for evaluation of instructors and courses
Training transactions: notifications, confirmations, registers, etc.
Full training records per person, per subject, per course, per unit, etc.
Good Practices and Recommendations:
Quality Management
Page 32
1.
Expand training to include eLearning
2.
Expand training to include visits to vendor sites, exhibitions, lectures and
conferences
3.
Apply “Train the trainers” policies to share the knowledge and lower the costs
4.
Create internal training mechanisms so that staff with specialized interests can
transfer it to others. This can be done through weekly or monthly talks.
Quality Management
Page 33