A Comparative Evaluation of Maintainability:
A Study of Engineering Department’s Website Maintainability
Nary Subramanian*
Richard Puerzer
Lawrence Chung
Dept. of Computer Science
University of Texas at Tyler
Tyler, TX
koolnary@yahoo.com
Dept. of Engineering
Hofstra University
Hempstead, N.Y.
eggrjp@hofstra.edu
Dept. of Computer Science
University of Texas at Dallas
Richardson, TX
chung@utdallas.edu
taken for the approval process through the pipeline, and so
on. Therefore, while it usually takes a few days to make
even minor modifications to the website while on the other,
the chain of checks and balances ensures that the updates
work and the accuracy of the new information is high, and
since the maintenance is performed by faculty during their
spare time, the cost to the department is kept to a minimum.
In this paper we present the qualitative evaluation of the
maintainability of HED website using the NFR Framework
[1, 2], where NFR stands for Non-Functional Requirement –
the maintainability of the current process in HED is
systematically evaluated and suggestions provided on how
the maintainability may be improved. The maintainability
process at HED is contrasted with other website processes
used in the academia and this helps us to compare and
contrast the HED process. While comparing maintenance
models used in the academia we used the closely related
NFRs of maintainability and cost – a complex maintenance
process involves higher cost both in terms of time and
money, while a less costly process may negatively affect
maintainability of the website.
One approach to maintainability evaluation in the
literature is based on metrics: thus [7] develops a set of
metrics for estimating web engineering effort – these metrics
are mostly based on time taken for a particular activity;
mean-time-to-change (MTTC) is also used as a measure of
maintainability of software in general [9] (see page 630 of
[9]); and the IEEE Standard 982.1-1988 [10] suggests the
usage of software maturity index to evaluate the
maintainability (more appropriately, stability) of software. In
[11] an accessibility framework is proposed that evaluates
web accessibility using a set of accessibility factors and it
has been suggested that maintainability may be similarly
evaluated. The Software Engineering Institute has proposed
a maintainability index (MI) that is based on a polynomial
that includes Halstead volume, cyclomatic complexity, and
lines of code [12]. Our work differs from the above in the
following ways: we have made a comparative evaluation and
analysis of the website maintenance models used in the
academia using the NFR Framework.
Section 2 discusses some of the website maintenance
models used in academia including the process at HED,
Section 3 applies the NFR Framework for analyzing and
evaluating the HED model, Section 4 presents the
observations from this analysis, while Section 5 presents the
conclusions and directions for future research. In this paper
We have been maintaining the Engineering
Department’s website at Hofstra University for the past five
years and our experience has been a mixed one: on the one
hand it usually takes a few days to make even minor
modifications to the website while on the other, the chain of
checks and balances ensures that the updates work and the
accuracy of the new information is high. We thought it will
be an interesting and useful exercise to analyze and
qualitatively evaluate the website maintenance policy at
Hofstra compared to other universities so that this
information will help all concerned stakeholders to make
modifications to the policy, if needed. We used the NFR
Framework for the analysis and evaluation, and compared
the closely related NFRs (or Non-Functional Requirements)
of maintainability and cost (both, time and money costs), for
the three different website maintenance policies used in the
academia including that of Hofstra Engineering
Department, and concluded that while there are tradeoffs
involved, small changes in Hofstra’s policies could
significantly increase the maintainability of the Engineering
Department website.
Keywords: Website, maintainability, NFR Framework
1. Introduction
Website maintenance could involve a high investment
of time and money for the institution that owns the website.
Moreover, in order to have some degree of control on the
maintenance activity most institutions have some form of
policy for maintaining websites. While different policies
have their plus and minus points, we feel that an evaluation
of the maintainability afforded by a policy will be of great
help to the policy makers by providing a qualitative measure
of their policies. We have been maintaining the website for
Hofstra Engineering Department (HED) at www.hofsta.edu/
engineering for the past five years and our experience with
the procedure for updating the HED website has been mixed:
we have been alternately frustrated and enjoyed the
experience. This has been due to various factors: availability
of correct data on time, ease with which we have been able
to update the pages using the Hofstra technology, the time
______________________________________
* This work was done when this author was a member of the
faculty of Department of Engineering, Hofstra University.
1
we have used the phrases maintenance process, maintenance
policy, and maintenance model synonymously.
seen that the task approval process is a (maximum) four step
process that ensures that all errors are removed, and that all
departmental and university regulations are met. While this
process is systematic, this four-step process could take
anywhere between a couple of days to a few months since all
the work in the department is done by full time faculty
members during their spare time, and sometimes not all data
is available to begin with. Moreover, this four-step process
will have to be followed for even minor modifications such
as spelling/grammar errors, incorrect links, or wrong data.
Therefore, maintainability does not seem high.
In order to compare and contrast HED process, we
considered two other processes followed in the academia –
one in which department has a full time staff to maintain
web pages (which we call Full Time Staff Model or FTS
Model) and the other in which a centralized University web
team (which we call Centralized University Web Team
Model or CUWT Model) does all the maintenance. Both
these models are given in Figure 2 and Figure 3. In the FTS
model each department has a team of full time staff to
maintain departmental websites (for example, the University
of Texas at Dallas, Computer Science department). In the
CUWT model there exists a team of web maintenance
professionals who help out all the departments in the
academic institution (for example, [15], [16], [17]).
2. Website Maintenance Processes
The HED website maintenance process [3] is given in
Figure 1. An engineering department faculty member serves
Requests from faculty, students, department
Engineering Department Web-Site
Developer/Maintainer - A Faculty Member
Tasks of Maintainer:
1. Data Collection/Verification
2. New Page Development
3. Existing Page Modification
4. Link Creation
Engineering Department Web-Site
Reviewer - Another Faculty Member
Engineering Department Chairman
University Web Editor - Full time
University Employee
Tasks of Reviewer:
1. Check Format
2. Check Conformance to
Department Regulations
Approved web page becomes live
Tasks of Web Editor:
1. Check Technical Feasibility
2. Check Conformance to
University Regulations
Tasks of Chairman:
1. Confirm Data
2. Provide Clarification
Legend:
Normal
Workflow
Corrections/
Feedback
Figure 1. The Hofstra Engineering Department Website
Maintenance Model
as the website developer and maintainer. His/her job is to
create new web pages or modify existing pages – the
requests come from faculty, from students to update a
student organization webpage on the HED, and from the
department itself for creating (e.g., a new lab page) or
modifying existing pages. The developer/maintainer
creates/updates the webpage by starting a task flow in the
Hofstra web maintenance system (called WMX). Once the
developer/maintainer creates/updates the webpage(s) using
WMX, the task flow is modified to that of the review and
assigned to the department reviewer, an activity performed
by another department faculty member. The reviewer either
approves the task in which case the reviewer assigns the task
to the next step, or the reviewer may find errors in the task
(such as non-conformance to department standards and
regulations) and assigns it back to the developer with
comments. The reviewer may assign the task (in case of
acceptance) either to the University Web Editor (the usual
case) or to the Engineering Department Chairman (in rare
cases). In case the task is assigned to the University web
editor, that person does a feasibility and conformance check
on the page(s) – the page(s) should not have too much data
such as video (so that bandwidth is not consumed) and the
page should conform to University regulations. In case of
non-acceptance by the University web editor the task is
reassigned to the HED developer/maintainer; else the page is
made live. In the case when the HED reviewer assigns the
task to the HED Chairman the latter either confirms existing
data or provides clarifications and reassigns the task to the
reviewer in both acceptance and rejection cases. As can be
Requests from faculty, students, department Requests from faculty, students, department
University Web Staff - Full time staff
members
Department Web Staff - Full time
staff members
Updated department website
Updated department website
Figure 2. The FTS and CUTW Maintenance Models
3. Evaluating Website Maintenance Model
The various models of web maintenance were compared
for their maintainability using the NFR Framework [1, 2, 4,
5]. The Softgoal Interdependency Graph that results from the
application of the NFR Framework is shown in Figure 3
(details given in the original version of this paper). Using the
label propagation rules of the NFR Framework it can be
determined that while the root NFR softgoal Maintainability
[Engineering Website] is denied, the root NFR softgoal Cost
[Maintaining, Engineering Website] is satisficed. Therefore,
the HED website maintenance model denies maintainability,
that is, it is not highly maintainable, but it is cost effective.
By observing Figure 3 we can determine the reasons for
poor maintainability of the HED model: several leaf NFR
softgoals (including the critical softgoals) are denied. The
maintainability of the HED model can be improved by any
one (or more) of the following means:
2
Cost
[Maintaining, Engineering Website]
Maintainability
[Engineering Website]
Modifiability
[Existing Page,
Engineering Website]
Creatability
[New Page,
Engineering Website]
Availability
[Data, New
Page]
Efficiency
[WMX, Modified Page]
Approvability
[New Page]
Efficiency
[WMX, New Page]
++
Developability
[New Page]
++
Claim1
Time
[Approval,
New Page]
Creatability
[New Links ]
+
+
++
-++
-
-
Claim3
Claim7
--
+
--
---
++
Claim11
++
++
++
Claim1: Data for new page is not always fully available;
s ometimes partial data is available and it takes s everal
iterations to complete the new page.
Claim2: Takes a long time to create a new page - about 30 minutes
on an average per page with at leas t 10 different page vis its
Claim3: It is not eas y at all to create a new page: s everal different
objects need to be created in different areas (text, images ,
etc.) and it takes at leas t 10 different pages vis its to create
the web page. In fact, a tas k flow to create the new
page needs to be s tarted which is as s igned to the developer.
Claim4: It takes a long time to make even a s imple modification at leas t 30 minutes on an average.
Claim5: It is difficult to make even a s imple page modification it s tarts with a tas k flow creation that is as s igned to the
maintainer and only then can the modifications be made.
Claim6: New links , es pecially links from menu items ,
can only be done by the web editor by reques t;
this takes time.
Claim7: Approval for a new page may be needed from
at mos t three different entities and therefore
could take a long time (running into days /weeks ).
Claim8: New links , es pecially links from menu items ,
can only be done by the web editor by reques t;
this takes time.
Claim9: Approval for a modification may be needed from
at mos t three different entities and therefore
could take a long time (running into days /weeks ).
Claim10: Department inves ts time in new page development
and exis ting page modification: developer/maintainer,
reviewer and chairman all inves t their time. And this is
done during their s pare time as this is not their full-time
job.
Claim11: Money cos t is not involved at all s ince all work is done by full-time faculty members .
Claim12: Data for page modification is s ometimes not
completely available or is in error - and it takes s ome
effort to get the correct data.
Creatability
Creatability
[Modified Page] [New Links ]
--
--
+
++
++
Claim10
+
Claim9
Availability
[Data,
Modification]
Claim2
Money
[Department,
Webs ite Maintenance]
++
Time
[Approval, Modification]
!
!
Cos t
[Money]
Time
[Department,
Webs ite Maintenance]
Approvability
[Modified Page]
Us ability
[WMX,
Modifying Page]
Time
[Modifying Page]
Us ability
[WMX, New Page]
Time
[Creating New Page]
Cos t
[Time]
---
Claim8
--
--
++
--
--
--
Claim4
++
Claim5
++
Claim6
--
++
Claim12
Dat a
Collect ion/
Verificat ion
Exist ing P age Link Creat ion
New P age
Development Modificat ion
Engineering
Department
Website
Developer/
Maintainer
Check Format
Check
Conformance
t o Depart ment
Regulat ions
Engineering
Department
Website
Reviewer
P rovide
Clarificat ion
Confirm
Dat a
Check
Conformance
t o Universit y
Regulat ions
Check
T echnical
Feasibilit y
University
Web Editor
Department
Chairman
Hofstra Engineering
Website Maintenance
Model
Figure 3. Complete SIG for HED Model
1.
2.
The NFR softgoals Time[Creating New Page] and
Time[Approval, New Page] are both satisficed – then
using the propagation rules, we can determine that the
root
NFR
softgoal
Maintainability[Engineering
Website] will be satisficed. This means that the time to
create a new page should be made favorable to the
maintainers (e.g., by automation) and the time for
approving a new page should be reduced (e.g., by
reducing approval stages) – both of these measures will
help improve HED website maintainability.
The NFR softgoals Time[Modifying Page] and
Time[Approval, Modification] are both satisficed and
this will satisfice the root NFR softgoal
Maintainability[Engineering Website]. This means that
if the time to modify a page (e.g., by automation) and
the time taken to approve modifications (e.g., by
reducing approval stages) are both reduced then the
maintainability of HED website will improve.
maintainability and cost effectiveness? This can be answered
from the SIG of Figure 4 (this is a partial SIG – for
succinctness sake we have shown only the leaf NFR
softgoals; the other NFR softgoals can be seen from Figure
3). By applying the propagation rules, we will be able to
determine that the FTS model is highly maintainable but
expensive to the department, while for the CUTW model the
availability of data is a serious concern – if data is
incomplete it usually takes a long time to get the data from
the concerned entity (faculty, student or the department) –
hence, the CUTW scores poorly on maintainability but is
very good in terms of cost effectiveness. Based on our
analysis of the maintainability of the HED website we
suggest the modified process for HED in Figure 5 that will
improve the maintainability without much effect on the cost
and elements of this modified process used in Figure 4.
Availability
[Data, New
Page]
We can see the potential of the NFR Framework in
evaluating the maintainability of the HED model as well as
in analyzing the reasons for the poor (or good)
maintainability.
Time
Developability Creatability
[Creating New Page]
[New Page]
[New Links]
!
Time
Time
Money
Time
Availability
Creatability [Approval,
[Department,
[Department,
[Approval, [Data,
Time
Creatability
New Page] Modification] [Modifying Page] [Modified Page] [New Links] Modification] Website Maintenance] Website Maintenance]
!
++
++
++
++
++
++
++
+
--
++
++
++
++
++
++
++
++
++
--
++
++
++
++
-++
++
++
Using
FrontPage
Using
T emplates
++
+
++
5. Observations
Data
Collection/
Verification
From the application of the NFR Framework, we
determined that the HED website maintenance model is not
easy to maintain but cost effective; and we also determined
what changes to the policy need to be implemented to make
the process more maintainable. How do the other
maintenance models (Figure 2) fare with respect to
Existing Page Modification Link Creation
New Page
Development Modification Approval
Engineering
Department
Website
Developer/
Maintainer
FTS
Model
CUTW
Model
...
Modified Hofstra
Engineering
Website Maintenance
Model
Figure 4. Partial SIGs for the FTS, CUTW, and
Modified HED Models
3
anonymous reviewers of the original version of this paper
for providing excellent comments and suggestions.
5. Conclusions
Our experience with maintaining the Hofstra
Engineering Department (HED) website convinced us that
the maintainability of the process used to maintain web
pages was low. However, we wished to analyze and evaluate
the process and compare it with other maintenance processes
used in the academia – the full-time staff model (FTS
model) that uses a dedicated departmental staff to maintain
the departmental website, and the centralized university web
team model (CUWT model) that uses a central web
maintenance team to maintain websites of all departments.
References
1.
2.
3.
WebTeam Monthly, February 2005, Volume 1, Issue 3,
The Monthly Journal of Web Development at Hofstra
University.
4.
L. Chung and N. Subramanian, “Architecture-Based Semantic
Evolution: A Study of Remotely Controlled Embedded
Systems”, Proc. Int. Conf. on Software Maintenance, IEEE
Computer Press, Florence, Italy, Nov., 2001, pp. 663-666.
L. Chung and N. Subramanian, “Architecture-based Semantic
Evolution for Embedded Systems: A Study of Remotely
Controlled Systems”, Journal of Software Maintenance and
Evolution: Research and Practice, 15(3), May/June 2003, pp.
145 - 190.
E. Mendes, N. Mosley, and S. Counsell, “Estimating Design
and Authoring Effort”, IEEE Multimedia, January-March
2001, pp. 50-57.
W. W. Vasconcelos and J. Cavalcanti, “Agent-Based Web Site
Maintenance”, Technical Report AUCS/TR0401, Department
of Computing Science, Univ. of Aberdeen, UK, Nov. 2003.
R. S. Pressman, Software Engineering: A Practitioner’s
Approach, McGraw Hill, Sixth Edition, 2005.
Software Engineering Standards, 1994 Edition, IEEE, 1994.
C. Boldyreff, “Determination and Evaluation of Web
Accessibility”, Proc. 11th Int. Workshops on Enabling
Technologies: Infrastructure for Collaborative Enterprises,
IEEE Computer Press, Pittsburg, USA, June 2002, pp. 35 –
42.
“Maintainability Index Technique for Measuring Program
Maintainability”, Software Engineering Institute, available at
http://www.sei.cmu.edu/str/descriptions/mitmpm_body.html,
March 2002.
G. Brajnik, “Automatic Web Usability Evaluation: What
needs to be done?”, Sixth Conference on Human Factors and
the Web, June 2000, available at http://www.tri.sbc.com
/hfweb/brajnik/hfweb-brajnik.html.
www.utdallas.edu
www.swarthmore.edu
www.richlandcollege.edu
www.cshl.org
M. Kajko-Mattsson, “Towards a Business Maintenance
Model”, Proc. Int. Conf. on Software Maintenance, IEEE
Computer Press, Florence, Italy, Nov., 2001, pp. 500-509.
L. Chung and N. Subramanian, “Process-Oriented Metrics for
Software Architecture Adaptability”, Proc. Int. Symp. on
Requirements Engineering, IEEE Computer Press, Aug-Sep.
2001, pp. 310-311.
N. Subramanian, Adaptable Software Architecture Generation
Using The NFR Approach, Ph.D. Thesis, The University of
Texas at Dallas, 2003.
Requests from faculty, students, department
Tasks of Maintainer:
1. Data Collection/Verification
2. New Page Development
3. Existing Page Modification
4. Modification Approval
5. Link Creation
Engineering Department Web-Site
Developer/Maintainer - A Faculty Member
Modifications
Engineering Department
Web-Site Reviewer Another Faculty Member
5.
Engineering Department Chairman
Legend:
University Web Editor - Full time
University Employee
6.
Normal Workflow
Correction/Feedback
7.
Approved web page becomes live
Figure 5. Modified HED Maintenance Model
8.
Using the NFR Framework [1, 2] we determined that the
maintainability of the HED model was not good but cost
effective. The FTS model has better maintainability but not
as cost effective as the HED model, while the CUWT is cost
effective but not as maintainable as the FTS process. We
analyzed the reasons for the poor maintainability of the HED
process and suggested modifications to the process to
improve the maintainability – it turns out that our
suggestions coincide with some of the improvements made
to the HED model recently.
However, there are several areas for further study – for
example, we have not compared the professionalism of web
updates, the accuracy of the models, and the satisfaction of
the users; we would like to extend the qualitative analysis to
quantitative analysis on the lines of [18]; and we could
extend this approach to evaluate the maintainability of the
processes used in the industry [7, 17]. However, we believe
that the process of maintainability analysis and evaluation
adopted in this paper is practical and useful to stakeholders
involved in the maintenance of web-based systems both in
the industry and academia.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
Acknowledgements
We would like to thank Ms. Lalitha Krishnan, Web Site
Manager, Cold Spring Harbor Labs, New York, for
reviewing the original version of this paper and providing
valuable feedback. We would also like to thank the
19.
4
L. Chung, B. A. Nixon, E. Yu, and J. Mylopoulos, NonFunctional Requirements in Software Engineering, Kluwer
Academic Publishers, Boston, 2000.
J. Mylopoulos, L. Chung, and B. Nixon, “Representing and
Using Nonfunctional Requirements: A Process-Oriented
Approach”, IEEE Transactions on Software Engineering,
Vol. 18, No. 6, June 1992, pp. 483-497.